All sites / News and publishing
The Economist
economist.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 12 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
6 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
3 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
3 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 69 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 93 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 74 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 84 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 90 | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 87 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 81 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 57 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 20 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 41 | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 13 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 27 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 48 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# robots.txt # Sitemaps Sitemap: https://www.economist.com/sitemap.xmlSitemap: https://www.economist.com/googlenews.xml # ==========================================# USER-FACING LLM BOTS# Allowed to fetch on behalf of a user# ==========================================… 11 lines not about AI bots User-agent: ChatGPT-UserAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: Claude-SearchBotAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: Claude-UserAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: Claude-WebAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: PerplexityBotAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: Perplexity-UserAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / # I have added OAI-SearchBot (OpenAI's live search agent) … 1 line not about AI bots# as it behaves the same way as ChatGPT-User.User-agent: OAI-SearchBotAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / # ==========================================# TRAINING BOTS & MASS SCRAPERS# Completely blocked from the entire site# ==========================================… 4 lines not about AI bots User-agent: GPTBotDisallow: / # Google-Extended is used by the Gemini app for searches on… 1 line not about AI bots# behalf of the user but it also used for trainingUser-agent: Google-ExtendedAllow: /pro$Allow: /pro/Allow: /syndication$Allow: /syndication/Disallow: / User-agent: anthropic-aiDisallow: / User-agent: Applebot-ExtendedDisallow: / User-Agent: BytespiderDisallow: / User-agent: CCBotDisallow: / User-agent: ClaudeBotDisallow: / # ==========================================# OTHER MISCELLANEOUS BOTS# Completely blocked# ========================================== User-agent: PiplBotDisallow: / User-agent: TurnitinBotDisallow: /… 10 lines not about AI bots User-agent: PetalBotDisallow: / User-agent: MoodleBotDisallow: / User-agent: magpie-crawlerDisallow: / User-agent: ia_archiverDisallow: / User-Agent: Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)Disallow: /checkout # ==========================================# GENERAL CRAWLERS (e.g., Googlebot, Bingbot)# Allowed by default, but blocked from select paths# ==========================================User-agent: *Disallow: /5605/Disallow: /pubads.g.doubleclick.net/Disallow: /assets/infographic/Disallow: /search/Disallow: /search?q=Disallow: /graphqlDisallow: /apiDisallow: /subscribe/getstarted/Disallow: /checkoutDisallow: /switch*Disallow: /audio-edition-podcast/*/index.xml… 28 lines not about AI bots
Notes about AI written in the file: "USER-FACING LLM BOTS" · "I have added OAI-SearchBot (OpenAI's live search agent)" · "TRAINING BOTS & MASS SCRAPERS" · "behalf of the user but it also used for training"
Everything else we checked
| Front door | Human check The homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site (HTTP 403).GET https://www.economist.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| llms.txt | Refused (HTTP 403)https://www.economist.com/llms.txt |
| ai.txt | Refused (HTTP 403) |
| TDMRep | Refused (HTTP 403)/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
46 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Aug 2026More blocking Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.Found in the Internet Archive's copies: changed between 12 Jul 2026 and 7 Aug 2026. Evidence: before · after
Show exactly what changed
- # Bot Directives+ # ==========================================+ # USER-FACING LLM BOTS+ # Allowed to fetch on behalf of a user+ # ==========================================- User-agent: GPTBot+ User-agent: ChatGPT-User+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: Google-Extended+ User-agent: Claude-SearchBot+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: ChatGPT-User+ User-agent: Claude-User+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: CCBot+ User-agent: Claude-Web+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: PiplBot+ User-agent: PerplexityBot+ Allow: /pro$+ Allow: /pro/Disallow: /+ User-agent: Perplexity-User+ Allow: /pro$+ Allow: /pro/+ Disallow: /++ # I have added OAI-SearchBot (OpenAI's live search agent)+ # as it behaves the same way as ChatGPT-User.+ User-agent: OAI-SearchBot+ Allow: /pro$+ Allow: /pro/+ Disallow: /++ # ==========================================+ # TRAINING BOTS & MASS SCRAPERS+ # Completely blocked from the entire site+ # ==========================================++ User-agent: GPTBot+ Disallow: /++ # Google-Extended is used by the Gemini app for searches on+ # behalf of the user but it also used for training+ User-agent: Google-Extended… 59 more lines - Sep 2025More blocking Now blocked: Perplexity-User.Found in the Internet Archive's copies: changed between 1 Aug 2025 and 1 Sep 2025. Evidence: before · after
Show exactly what changed
- # Specific robot directives:+ # Sitemaps- # Description : Google AdSense delivers advertisements to a broad network of affiliated sites.- # A robot analyses the pages that display the ads in order to target the ads to the page content.- User-agent: Mediapartners-Google*- Allow: /+ Sitemap: https://www.economist.com/sitemap.xml+ Sitemap: https://www.economist.com/googlenews.xml- # Description : The Grapeshot crawler is an automated robot that visits pages to examine and analyse the content.- # This adds an exception to crawl delay while preserving disallows.- User-agent: grapeshot- Allow: /+ # Bot Directives- # GPTBot is OpenAI’s web crawlerUser-agent: GPTBot…- # Allows us to block Google's bot BardUser-agent: Google-Extended…- # ChatGPT-User is OpenAI’s web crawlerUser-agent: ChatGPT-User…- # Common Crawl botUser-agent: CCBot…- # PiplBot is PiplBot's web crawlerUser-agent: PiplBot…- # anthropic-ai is Anthropic's web crawlerUser-agent: anthropic-ai…- # Claude-Web is Claude’s web crawlerUser-agent: Claude-Web…- # TurnitinBot is Turnitin’s web crawlerUser-agent: TurnitinBot…- # PetalBot is Petal’s web crawlerUser-agent: PetalBot…- # MoodleBot is Moodle’s web crawlerUser-agent: MoodleBot…- # magpie-crawler is Brandwatch.com’s web crawlerUser-agent: magpie-crawler…… 131 more lines - Nov 2024More blocking Now blocked: Bytespider.Found in the Internet Archive's copies: changed between 1 Oct 2024 and 1 Nov 2024. Evidence: before · after
Show exactly what changed
+ # Bytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs including those powering ChatGPT competitor Doubao.+ User-Agent: Bytespider+ Disallow: /+# No robots are allowed to index private paths: - Jul 2024More blocking Now blocked: Applebot-Extended and PerplexityBot.Found in the Internet Archive's copies: changed between 1 Jun 2024 and 1 Jul 2024. Evidence: before · after
Show exactly what changed
- # omgili is Omgili’s web crawler- User-Agent: omgili+ # Applebot-Extended is Apple's secondary user agent+ User-agent: Applebot-ExtendedDisallow: /- # omgilibot is Omgili’s web crawler- User-Agent: omgilibot+ # PerplexityBot is the crawler for perplexity AI+ User-Agent: PerplexityBotDisallow: /…Disallow: /whichmba/webinars?+ Disallow: /checkout# Paths (no clean URLs) - Nov 2023More blocking Now blocked: Google-Extended, CCBot, anthropic-ai and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after
Show exactly what changed
- # Description : GPTBot is OpenAI’s web crawler.+ # GPTBot is OpenAI’s web crawlerUser-agent: GPTBot…+ # Allows us to block Google's bot Bard+ User-agent: Google-Extended+ Disallow: /++ # ChatGPT-User is OpenAI’s web crawler+ User-agent: ChatGPT-User+ Disallow: /++ # Common Crawl bot+ User-agent: CCBot+ Disallow: /++ # PiplBot is PiplBot's web crawler+ User-agent: PiplBot+ Disallow: /++ # anthropic-ai is Anthropic's web crawler+ User-agent: anthropic-ai+ Disallow: /++ # Claude-Web is Claude’s web crawler+ User-agent: Claude-Web+ Disallow: /++ # TurnitinBot is Turnitin’s web crawler+ User-agent: TurnitinBot+ Disallow: /++ # PetalBot is Petal’s web crawler+ User-agent: PetalBot+ Disallow: /++ # MoodleBot is Moodle’s web crawler+ User-agent: MoodleBot+ Disallow: /++ # magpie-crawler is Brandwatch.com’s web crawler+ User-agent: magpie-crawler+ Disallow: /++ # ia_archiver is Wayback Machine’s web crawler+ User-agent: ia_archiver+ Disallow: /++ # omgili is Omgili’s web crawler+ User-Agent: omgili+ Disallow: /++ # omgilibot is Omgili’s web crawler+ User-Agent: omgilibot+ Disallow: /+# No robots are allowed to index private paths: - Sep 2023More blocking Now blocked: GPTBot.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
Show exactly what changed
# robots.txt- #- # Sitemap- Sitemap: https://www.economist.com/sitemap.xml- Sitemap: https://www.economist.com/googlenews.xml-# Specific robot directives:…User-agent: Mediapartners-Google*- Disallow:+ Allow: /…User-agent: grapeshot- Disallow:+ Allow: /+ # Description : GPTBot is OpenAI’s web crawler.+ User-agent: GPTBot+ Disallow: /+# No robots are allowed to index private paths:…+ # Sitemap+ Sitemap: https://www.economist.com/sitemap.xml+ Sitemap: https://www.economist.com/googlenews.xml# Directories…Disallow: /report-abuse/+ Disallow: /user/Disallow: /users/…Disallow: /vote/+ Disallow: /taxonomy/term/# Paths (no trailing /, beware this will stop file like /admin.html being…Disallow: /contact+ Disallow: /labDisallow: /logout…Disallow: /user+ Disallow: /uspodDisallow: /which-mba…Disallow: /bookmarks-
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.