Block AI crawlers in robots.txt

To keep your pages out of AI model training, put the training crawlers in one group with Disallow: /: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl) and Meta-ExternalAgent (Meta). Search engines and AI search crawlers keep their access through the * group.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Disallow:

In the tester below, GPTBot is disallowed by line 8, Disallow: /, and so is every other token in that group. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are allowed: no group names them, so the empty Disallow: under * applies.

What each token controls

From each vendor’s documentation, fetched 2026-10-15, with the date the page itself carries:

token               vendor        controls                                   page dated
GPTBot              OpenAI        crawling for model training                undated
OAI-SearchBot       OpenAI        showing up in ChatGPT search               undated
ChatGPT-User        OpenAI        pages a ChatGPT user asks it to visit      undated
ClaudeBot           Anthropic     collecting content for model training      2026-04-07
Claude-SearchBot    Anthropic     indexing for Claude's search results       2026-04-07
Claude-User         Anthropic     pages a Claude user asks it to visit       2026-04-07
Google-Extended     Google        Gemini training and grounding, not Search  2026-07-14
Applebot-Extended   Apple         training on what Applebot crawls           2026-09-04
CCBot               Common Crawl  Common Crawl's public web archive          undated
PerplexityBot       Perplexity    Perplexity search results, not training    2026-01-29
Meta-ExternalAgent  Meta          AI training and indexing for its products  2026-05-21

Common Crawl’s archive is open for anyone to download, so blocking CCBot keeps future pages out of a dataset model builders can use too.

What it doesn’t do

• It’s a request, not a lock. RFC 9309 (section 3) says the protocol is no substitute for security measures, and the file is public. To refuse requests, check the user agent against the IP ranges most of these vendors publish, at your server or CDN.
• User-triggered fetchers may skip it. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores” them, and Meta says Meta-ExternalFetcher may bypass them. Anthropic says its bots, Claude-User included, honor robots.txt.
• Google-Extended and Applebot-Extended don’t crawl. They only decide what Google’s and Applebot’s fetches may be used for, so blocking them leaves you in Google Search and in Siri, Spotlight and Safari. Google says Googlebot is the control for Search, AI features included; nosnippet and noindex limit what those show.
• It works from now on. Anthropic’s page says a block keeps the site’s future materials out of training. OpenAI and Perplexity quote up to about 24 hours to pick up a change.
• Each host needs its own file. blog.example.com doesn’t read example.com/robots.txt.

Bytespider, which block lists attribute to ByteDance, is left out of the table: no ByteDance page documenting the token or its robots.txt handling could be found. Adding a group for it costs nothing, but nothing documents that it’s obeyed.