AI crawlers and robots.txt: every user agent and what it controls
AI companies crawl with named user agents you can allow or block in robots.txt. The ones that matter most are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity). Blocking a training crawler keeps your pages out of future models; blocking a search crawler keeps them out of that assistant's answers today.
The user agents
| Company | User agent | What it does |
|---|---|---|
| OpenAI | GPTBot | Collects pages that may be used to train OpenAI's models. |
| OpenAI | OAI-SearchBot | Indexes pages so ChatGPT search can show and link them. |
| OpenAI | ChatGPT-User | Fetches a page when a ChatGPT user asks about it. |
| Anthropic | ClaudeBot | Collects pages that may be used to train Claude. |
| Anthropic | Claude-SearchBot | Indexes pages for Claude's search results. |
| Anthropic | Claude-User | Fetches a page when a Claude user asks about it. |
| Perplexity | PerplexityBot | Indexes pages so Perplexity can cite them. |
| Perplexity | Perplexity-User | Fetches a page when a Perplexity user asks about it. |
| Google-Extended | Not a separate crawler: a robots.txt token that controls whether Googlebot's pages are used for Gemini. | |
| Apple | Applebot-Extended | A token that controls whether Applebot's pages are used to train Apple's models. |
| Common Crawl | CCBot | Builds the open web archive many AI models are trained on. |
Each company documents its crawlers. OpenAI's list is at platform.openai.com/docs/bots, Google's at developers.google.com/search/docs/crawling-indexing/overview-google-crawlers. Check them before relying on any rule, because names change.
Allow every AI crawler
If you want to be found and cited, the simplest robots.txt says nothing about AI bots at all, because no rule means allowed. Make sure no group blocks them by accident:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlA common mistake is a security plugin or CDN setting that adds Disallow: / for GPTBot or ClaudeBot. CrawlReadable checks each one by name.
Stay in AI search but out of training
Block the training crawlers and allow the search and user agents. Answers can still cite you; future models won't train on your pages.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /This is a trade-off. Models trained without your pages know less about you, so assistants answering from memory may describe you less accurately.
What robots.txt can't do
robots.txt is a request, not a lock. Well-behaved crawlers follow it. To enforce a block you need firewall rules, and those are invisible to outside checkers like ours: we can read your robots.txt, but we can't see whether your CDN blocks a bot's IP addresses.
Questions
Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. Search uses OAI-SearchBot, and live lookups use ChatGPT-User. Blocking GPTBot only opts your pages out of training data.
Do AI crawlers obey robots.txt?
The major ones say they do, and publish their user agents so you can control them. Robots.txt has no enforcement, so use firewall rules if you need a hard block.