The AI bots we track
Each AI company sends out one or more bots, and each bot has a name that sites use in robots.txt to let it in or keep it out. We track 25 names: the 24 we score, plus Anthropic's old name anthropic-ai. We checked each against the company's own documentation on 19 Oct 2026. The 12 "major" bots are the ones a site must block to count as "Blocks AI".
AI training
Collects pages to train AI models like ChatGPT and Claude. Blocking these keeps a site out of future training. It does not stop AI from reading the site to answer questions.
| Bot name | Company | Follows robots.txt? | Blocked by | Named by | What the company says |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Says yes | 32 / 159 | 36 | "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Source |
| ClaudeBot | Anthropic | Says yes | 41 / 159 | 41 | "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Source |
| Google-Extended | Says yes | 31 / 159 | 32 | A control token with no crawler of its own. Covers Gemini training and grounding. "Google-Extended does not impact a site's inclusion in Google Search." It does not control AI Overviews or AI Mode. Source | |
| Applebot-Extended | Apple | Says yes | 38 / 159 | 30 | "Applebot-Extended does not crawl webpages." It only decides whether pages Applebot crawled may train Apple's models. Source |
| CCBot | Common Crawl | Says yes | 39 / 159 | 31 | Builds an open web archive that is widely used downstream to train language models. Its FAQ says it checks robots.txt first. Source |
| Meta-ExternalAgent | Meta | Suggests yes | 33 / 159 | 28 | Crawls "for use cases such as training foundation AI models". Meta says to block it in robots.txt; it has no explicit compliance sentence. Source |
| Bytespider minor | ByteDance | No company page | 40 / 159 | 30 | No operator documentation found. Widely seen in server logs; independent reports dispute that it follows robots.txt. (low confidence) |
| MistralAI-Training minor | Mistral AI | Says yes | 14 / 159 | 1 | Newly documented token. "Webmasters can disallow this user agent in their robots.txt file." Source |
| AI2Bot minor | Allen Institute for AI | Does not say | 21 / 159 | 9 | "This web content is used to train open language models." No robots.txt statement. Source |
| anthropic-ai minor old name | Anthropic (legacy) | Unknown | 31 / 159 | 25 | Legacy token no longer on Anthropic's crawler page. Tracked because many robots.txt files still name it. (low confidence) |
AI search
Fetches pages so AI tools can quote and link to them in answers. Blocking these usually means the site stops showing up as a source.
| Bot name | Company | Follows robots.txt? | Blocked by | Named by | What the company says |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Says yes | 19 / 159 | 27 | Controls inclusion in ChatGPT search results; changes take about 24 hours. Source |
| Claude-SearchBot | Anthropic | Says yes | 27 / 159 | 24 | "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization." Source |
| PerplexityBot | Perplexity | Suggests yes | 33 / 159 | 35 | "It is not used to crawl content for AI foundation models." Source |
| Amazonbot minor | Amazon | Says yes | 32 / 159 | 31 | Mixed use: improves Amazon products, and content "may be used to train Amazon AI models." Source |
| Amzn-SearchBot minor | Amazon | Says yes | 16 / 159 | 6 | "Amzn-SearchBot does not crawl content for generative AI model training." Source |
| DuckAssistBot minor | DuckDuckGo | Says yes | 22 / 159 | 16 | Fetches pages for AI-assisted answers; "not used in any way to train AI models." Source |
| MistralAI-Index minor | Mistral AI | Suggests yes | 13 / 159 | 0 | Newly documented token for Mistral's search index. Source |
| Meta-WebIndexer minor | Meta | Suggests yes | 17 / 159 | 9 | "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." Source |
AI assistants
Visits a page the moment a person asks an AI to, for example to check a price or book a table. Several AI companies say these visits may ignore robots.txt.
| Bot name | Company | Follows robots.txt? | Blocked by | Named by | What the company says |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | Not always | 21 / 159 | 29 | "Because these actions are initiated by a user, robots.txt rules may not apply." Source |
| Claude-User | Anthropic | Says yes | 28 / 159 | 24 | "Claude-User allows site owners to control which sites can be accessed through these user-initiated requests." Source |
| Perplexity-User | Perplexity | Mostly ignores it | 26 / 159 | 20 | "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Source |
| Google-Agent minor | Mostly ignores it | 14 / 159 | 2 | "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." Google is testing Web Bot Auth signatures for it. Source | |
| Meta-ExternalFetcher minor | Meta | Not always | 25 / 159 | 16 | "This crawler may bypass robots.txt rules." Source |
| Amzn-User minor | Amazon | Not always | 14 / 159 | 6 | "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." Source |
| MistralAI-User minor | Mistral AI | Suggests yes | 22 / 159 | 9 | "MistralAI-User governs which sites these user requests can be made to." Source |
AI tools robots.txt can't name
Grok (xAI)
xAI publishes no crawler or agent token. Its hosted browser uses egress IP ranges that xAI discloses only through account teams.
ChatGPT agent
Uses a generic Chrome user agent; requests are signed with HTTP Message Signatures (Web Bot Auth) under signature agent https://chatgpt.com. No robots.txt token.
Browser-based agents (Claude in Chrome, Comet, others)
Run in the user's own browser, so sites see the user's network identity. robots.txt cannot address them.