X
x.comBlocks AI
Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names Google-Extended and Meta-ExternalAgent directly. A catch-all rule (User-agent: *) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Blocked
9 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Blocked
8 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Blocked
7 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Blocked | |||
| GPTBotOpenAI | Blocked | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 74 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Blocked | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 88 | Suggests yes |
| BytespiderByteDance | Blocked | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Blocked | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Blocked | |||
| OAI-SearchBotOpenAI | Blocked | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Blocked | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Blocked | Named, line 85 | Suggests yes |
| AI assistants · Blocked | |||
| ChatGPT-UserOpenAI | Blocked | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Blocked | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Blocked | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 94 | Says not always |
| Amzn-UserAmazon | Blocked | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Named | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| FacebookBot | Has rules | line 77 | Not in our verified list |
| meta-externalads | Has rules | line 91 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# Google / Bing Search Engine Robots# ==================================# Shared group so Googlebot and Bingbot always get the same rules.User-agent: GooglebotUser-agent: Bingbot Allow: /*?s=Allow: /*?t=Allow: /*?ref_src=Allow: /hashtag/*?src=Allow: /search?q=%23Allow: /i/api/# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in# favor of Allow, so /i/api/ fetches stay crawlable.Allow: /i/api/*Disallow: /*?lang=en-ssAllow: /*?lang=Disallow: /search/realtimeDisallow: /search/usersDisallow: /search/*/grid Disallow: /*/analyticsDisallow: /*/followersDisallow: /*/followingDisallow: /*/verified_followers Disallow: /account/deactivatedDisallow: /settings/deactivated # Only `*` and `$` are metacharacters here (RFC 9309); regex classes match# literally. /photo is anchored so /status/*/photo/1 media pages stay open.Disallow: /*/likesDisallow: /*/likes?Disallow: /*/retweetsDisallow: /*/retweets?Disallow: /*/mediaDisallow: /*/media?Disallow: /*/photo$Disallow: /*/photo?Allow: /*? User-agent: facebookexternalhit Allow: /*?lang=Allow: /*?s=Allow: /*?t=Allow: /*?ref_src=Allow: /hashtag/*?src=Allow: /search?*cashtagRestId=Allow: /i/api/Allow: /i/api/*Disallow: /search?q=Disallow: /search/realtimeDisallow: /search/usersDisallow: /search/*/grid Disallow: /*?Disallow: /*/followersDisallow: /*/followingDisallow: /*/verified_followers Disallow: /account/deactivatedDisallow: /settings/deactivated Disallow: /*/likesDisallow: /*/likes?Disallow: /*/retweetsDisallow: /*/retweets?Disallow: /*/mediaDisallow: /*/media?Disallow: /*/photo$Disallow: /*/photo?… 72 lines not about AI bots User-Agent: Google-ExtendedDisallow: * User-Agent: FacebookBotDisallow: * User-agent: DiscordbotDisallow: * # Meta AI / product crawlers (product tokens; versions like /1.1 are ignored… 4 lines not about AI bots# under RFC 9309 token matching).User-agent: meta-webindexerDisallow: * User-agent: meta-externalagentDisallow: * User-agent: meta-externaladsDisallow: * User-agent: meta-externalfetcherDisallow: * # Every bot that might possibly read and respect this file# ========================================================User-agent: *Disallow: / # WHAT-4882 - Keep notification-email links (/i/u) out of search results.# Named crawlers stay un-blocked on purpose: they must crawl /i/u to see its# X-Robots-Tag noindex (the robots.txt Noindex directive died in 2019).Disallow: /i/u # Wait 1 second between successive requests. See ONBOARD-2698 for details.Crawl-delay: 1 # Independent of user agent. Links in the sitemap are full URLs using https://# and need to match the protocol of the sitemap.Sitemap: https://x.com/sitemap.xml … 18 lines not about AI bots
Notes about AI written in the file: "Meta AI / product crawlers (product tokens; versions like /1.1 are ignored" · "Independent of user agent. Links in the sitemap are full URLs using https://"
Everything else we checked
| Front door | We did not visit: the rules ask our bot to stay out. |
|---|---|
| llms.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| ai.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| TDMRep | Disallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
36 versions on record since 6 Dec 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Sep 2026Signals Named Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher explicitly, without changing what they may reach.Found in the Internet Archive's copies: changed between 6 Aug 2026 and 3 Sep 2026. Evidence: before · after
Show exactly what changed
- # Google Search Engine Robot- # ==========================+ # Google / Bing Search Engine Robots+ # ==================================+ # Shared group so Googlebot and Bingbot always get the same rules.User-agent: Googlebot+ User-agent: Bingbot- Allow: /*?lang=+ Allow: /*?s=+ Allow: /*?t=+ Allow: /*?ref_src=Allow: /hashtag/*?src=…Allow: /i/api/+ # Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in+ # favor of Allow, so /i/api/ fetches stay crawlable.+ Allow: /i/api/*+ Disallow: /*?lang=en-ss+ Allow: /*?lang=Disallow: /search/realtime…+ Disallow: /*/analytics+ Disallow: /*/followers+ Disallow: /*/following+ Disallow: /*/verified_followers++ Disallow: /account/deactivated+ Disallow: /settings/deactivated++ # Only `*` and `$` are metacharacters here (RFC 9309); regex classes match+ # literally. /photo is anchored so /status/*/photo/1 media pages stay open.+ Disallow: /*/likes+ Disallow: /*/likes?+ Disallow: /*/retweets+ Disallow: /*/retweets?+ Disallow: /*/media+ Disallow: /*/media?+ Disallow: /*/photo$+ Disallow: /*/photo?+ Allow: /*?++ User-agent: facebookexternalhit++ Allow: /*?lang=+ Allow: /*?s=+ Allow: /*?t=+ Allow: /*?ref_src=+ Allow: /hashtag/*?src=+ Allow: /search?*cashtagRestId=+ Allow: /i/api/+ Allow: /i/api/*+ Disallow: /search?q=+ Disallow: /search/realtime+ Disallow: /search/users+ Disallow: /search/*/grid+Disallow: /*?…… 61 more lines - Apr 2024
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.