{"domain":"economist.com","name":"The Economist","category":"news","subcategory":"magazine","country":"GB","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 12 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.313Z","last_checked":"2026-10-19T13:36:07.042Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.economist.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.economist.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.economist.com/robots.txt","hops":[{"from":"https://economist.com/robots.txt","to":"https://www.economist.com/robots.txt","status":301}],"blob":"5b0aecabb2641f39b0f0f6f3f4ab695575f1454b051cf4c9ea43b038f40130ce","text":"# robots.txt\n\n# Sitemaps\n\nSitemap: https://www.economist.com/sitemap.xml\nSitemap: https://www.economist.com/googlenews.xml\n\n# ==========================================\n# USER-FACING LLM BOTS\n# Allowed to fetch on behalf of a user\n# ==========================================\n\nUser-agent: ChatGPT-User\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: Claude-SearchBot\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: Claude-User\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: Claude-Web\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: PerplexityBot\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: Perplexity-User\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\n# I have added OAI-SearchBot (OpenAI's live search agent) \n# as it behaves the same way as ChatGPT-User.\nUser-agent: OAI-SearchBot\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\n# ==========================================\n# TRAINING BOTS & MASS SCRAPERS\n# Completely blocked from the entire site\n# ==========================================\n\nUser-agent: GPTBot\nDisallow: /\n\n# Google-Extended is used by the Gemini app for searches on\n# behalf of the user but it also used for training\nUser-agent: Google-Extended\nAllow: /pro$\nAllow: /pro/\nAllow: /syndication$\nAllow: /syndication/\nDisallow: /\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-Agent: Bytespider\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\n# ==========================================\n# OTHER MISCELLANEOUS BOTS\n# Completely blocked\n# ==========================================\n\nUser-agent: PiplBot\nDisallow: /\n\nUser-agent: TurnitinBot\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: MoodleBot\nDisallow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: ia_archiver\nDisallow: /\n\nUser-Agent: Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)\nDisallow: /checkout\n\n# ==========================================\n# GENERAL CRAWLERS (e.g., Googlebot, Bingbot)\n# Allowed by default, but blocked from select paths\n# ==========================================\nUser-agent: *\nDisallow: /5605/\nDisallow: /pubads.g.doubleclick.net/\nDisallow: /assets/infographic/\nDisallow: /search/\nDisallow: /search?q=\nDisallow: /graphql\nDisallow: /api\nDisallow: /subscribe/getstarted/\nDisallow: /checkout\nDisallow: /switch*\nDisallow: /audio-edition-podcast/*/index.xml","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":9,"text":"USER-FACING LLM BOTS"},{"line":55,"text":"I have added OAI-SearchBot (OpenAI's live search agent)"},{"line":65,"text":"TRAINING BOTS & MASS SCRAPERS"},{"line":73,"text":"behalf of the user but it also used for training"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["ChatGPT-User","Claude-SearchBot","Claude-User","Claude-Web","PerplexityBot","Perplexity-User","OAI-SearchBot","GPTBot","Google-Extended","anthropic-ai","Applebot-Extended","Bytespider","CCBot","ClaudeBot","PiplBot","TurnitinBot","PetalBot","MoodleBot","magpie-crawler","ia_archiver","Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)","*"],"lines":137},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":70},"any":{"allowed":false,"line":70},"search":{"allowed":false,"line":70},"cart":{"allowed":false,"line":70},"checkout":{"allowed":false,"line":70},"account":{"allowed":false,"line":70}},"disallows":1,"lines":[69,70],"groupRanges":[[69,70]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":94},"any":{"allowed":false,"line":94},"search":{"allowed":false,"line":94},"cart":{"allowed":false,"line":94},"checkout":{"allowed":false,"line":94},"account":{"allowed":false,"line":94}},"disallows":1,"lines":[93,94],"groupRanges":[[93,94]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":79},"any":{"allowed":false,"line":79},"search":{"allowed":false,"line":79},"cart":{"allowed":false,"line":79},"checkout":{"allowed":false,"line":79},"account":{"allowed":false,"line":79}},"disallows":1,"lines":[74,79],"groupRanges":[[74,79]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":85},"any":{"allowed":false,"line":85},"search":{"allowed":false,"line":85},"cart":{"allowed":false,"line":85},"checkout":{"allowed":false,"line":85},"account":{"allowed":false,"line":85}},"disallows":1,"lines":[84,85],"groupRanges":[[84,85]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":91},"any":{"allowed":false,"line":91},"search":{"allowed":false,"line":91},"cart":{"allowed":false,"line":91},"checkout":{"allowed":false,"line":91},"account":{"allowed":false,"line":91}},"disallows":1,"lines":[90,91],"groupRanges":[[90,91]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":88},"any":{"allowed":false,"line":88},"search":{"allowed":false,"line":88},"cart":{"allowed":false,"line":88},"checkout":{"allowed":false,"line":88},"account":{"allowed":false,"line":88}},"disallows":1,"lines":[87,88],"groupRanges":[[87,88]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":82},"any":{"allowed":false,"line":82},"search":{"allowed":false,"line":82},"cart":{"allowed":false,"line":82},"checkout":{"allowed":false,"line":82},"account":{"allowed":false,"line":82}},"disallows":1,"lines":[81,82],"groupRanges":[[81,82]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":62},"any":{"allowed":false,"line":62},"search":{"allowed":false,"line":62},"cart":{"allowed":false,"line":62},"checkout":{"allowed":false,"line":62},"account":{"allowed":false,"line":62}},"disallows":1,"lines":[57,62],"groupRanges":[[57,62]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":25},"any":{"allowed":false,"line":25},"search":{"allowed":false,"line":25},"cart":{"allowed":false,"line":25},"checkout":{"allowed":false,"line":25},"account":{"allowed":false,"line":25}},"disallows":1,"lines":[20,25],"groupRanges":[[20,25]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":46},"any":{"allowed":false,"line":46},"search":{"allowed":false,"line":46},"cart":{"allowed":false,"line":46},"checkout":{"allowed":false,"line":46},"account":{"allowed":false,"line":46}},"disallows":1,"lines":[41,46],"groupRanges":[[41,46]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":18},"any":{"allowed":false,"line":18},"search":{"allowed":false,"line":18},"cart":{"allowed":false,"line":18},"checkout":{"allowed":false,"line":18},"account":{"allowed":false,"line":18}},"disallows":1,"lines":[13,18],"groupRanges":[[13,18]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":32},"any":{"allowed":false,"line":32},"search":{"allowed":false,"line":32},"cart":{"allowed":false,"line":32},"checkout":{"allowed":false,"line":32},"account":{"allowed":false,"line":32}},"disallows":1,"lines":[27,32],"groupRanges":[[27,32]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":53},"any":{"allowed":false,"line":53},"search":{"allowed":false,"line":53},"cart":{"allowed":false,"line":53},"checkout":{"allowed":false,"line":53},"account":{"allowed":false,"line":53}},"disallows":1,"lines":[48,53],"groupRanges":[[48,53]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":135},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[126,137],"groupRanges":[[126,137]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":5,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":3,"full":4,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"refused","http":403,"url":"https://www.economist.com/llms.txt"},"ai_txt":{"state":"refused","http":403,"url":"https://www.economist.com/ai.txt"},"tdmrep":{"state":"refused","http":403,"url":"https://www.economist.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":46,"first":"2023-01-01","archive_points":45,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12}]},"events":[{"domain":"economist.com","name":"The Economist","category":"news","date":"2026-08","precision":"month","source":"archive","provisional":false,"between":["2026-07-12","2026-08-07"],"summary":"Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.","dir":"up","up":["ClaudeBot","OAI-SearchBot","Claude-SearchBot","Claude-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260712121601/https://www.economist.com/robots.txt","https://web.archive.org/web/20260807091347/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Bot Directives"},{"t":"add","s":"# =========================================="},{"t":"add","s":"# USER-FACING LLM BOTS"},{"t":"add","s":"# Allowed to fetch on behalf of a user"},{"t":"add","s":"# =========================================="},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: CCBot"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: PiplBot"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# I have added OAI-SearchBot (OpenAI's live search agent) "},{"t":"add","s":"# as it behaves the same way as ChatGPT-User."},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"Allow: /pro$"},{"t":"add","s":"Allow: /pro/"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# =========================================="},{"t":"add","s":"# TRAINING BOTS & MASS SCRAPERS"},{"t":"add","s":"# Completely blocked from the entire site"},{"t":"add","s":"# =========================================="},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Google-Extended is used by the Gemini app for searches on"},{"t":"add","s":"# behalf of the user but it also used for training"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"gap","s":"… 59 more lines"}]},{"domain":"economist.com","name":"The Economist","category":"news","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-09-01"],"summary":"Now blocked: Perplexity-User.","dir":"up","up":["Perplexity-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250801023103/https://www.economist.com/robots.txt","https://web.archive.org/web/20250901053150/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Specific robot directives:"},{"t":"add","s":"# Sitemaps"},{"t":"ctx","s":""},{"t":"del","s":"# Description : Google AdSense delivers advertisements to a broad network of affiliated sites."},{"t":"del","s":"# A robot analyses the pages that display the ads in order to target the ads to the page content."},{"t":"del","s":"User-agent: Mediapartners-Google*"},{"t":"del","s":"Allow: /"},{"t":"add","s":"Sitemap: https://www.economist.com/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.economist.com/googlenews.xml"},{"t":"ctx","s":""},{"t":"del","s":"# Description : The Grapeshot crawler is an automated robot that visits pages to examine and analyse the content."},{"t":"del","s":"# This adds an exception to crawl delay while preserving disallows."},{"t":"del","s":"User-agent: grapeshot"},{"t":"del","s":"Allow: /"},{"t":"add","s":"# Bot Directives"},{"t":"ctx","s":""},{"t":"del","s":"# GPTBot is OpenAI’s web crawler"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Allows us to block Google's bot Bard"},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# ChatGPT-User is OpenAI’s web crawler"},{"t":"ctx","s":"User-agent: ChatGPT-User"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Common Crawl bot"},{"t":"ctx","s":"User-agent: CCBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# PiplBot is PiplBot's web crawler"},{"t":"ctx","s":"User-agent: PiplBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# anthropic-ai is Anthropic's web crawler"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Claude-Web is Claude’s web crawler"},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# TurnitinBot is Turnitin’s web crawler"},{"t":"ctx","s":"User-agent: TurnitinBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# PetalBot is Petal’s web crawler"},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# MoodleBot is Moodle’s web crawler"},{"t":"ctx","s":"User-agent: MoodleBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# magpie-crawler is Brandwatch.com’s web crawler"},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"gap","s":"…"},{"t":"gap","s":"… 131 more lines"}]},{"domain":"economist.com","name":"The Economist","category":"news","date":"2024-11","precision":"month","source":"archive","provisional":false,"between":["2024-10-01","2024-11-01"],"summary":"Now blocked: Bytespider.","dir":"up","up":["Bytespider"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241001013710/https://www.economist.com/robots.txt","https://web.archive.org/web/20241101020741/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# Bytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs including those powering ChatGPT competitor Doubao."},{"t":"add","s":"User-Agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# No robots are allowed to index private paths:"}]},{"domain":"economist.com","name":"The Economist","category":"news","date":"2024-07","precision":"month","source":"archive","provisional":false,"between":["2024-06-01","2024-07-01"],"summary":"Now blocked: Applebot-Extended and PerplexityBot.","dir":"up","up":["Applebot-Extended","PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240601011915/https://www.economist.com/robots.txt","https://web.archive.org/web/20240701014007/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# omgili is Omgili’s web crawler"},{"t":"del","s":"User-Agent: omgili"},{"t":"add","s":"# Applebot-Extended is Apple's secondary user agent "},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"# omgilibot is Omgili’s web crawler"},{"t":"del","s":"User-Agent: omgilibot"},{"t":"add","s":"# PerplexityBot is the crawler for perplexity AI"},{"t":"add","s":"User-Agent: PerplexityBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /whichmba/webinars?"},{"t":"add","s":"Disallow: /checkout"},{"t":"ctx","s":"# Paths (no clean URLs)"}]},{"domain":"economist.com","name":"The Economist","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended, CCBot, anthropic-ai and ChatGPT-User.","dir":"up","up":["Google-Extended","CCBot","anthropic-ai","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001002307/https://www.economist.com/robots.txt","https://web.archive.org/web/20231101010450/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Description : GPTBot is OpenAI’s web crawler."},{"t":"add","s":"# GPTBot is OpenAI’s web crawler"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Allows us to block Google's bot Bard"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# ChatGPT-User is OpenAI’s web crawler"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Common Crawl bot"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# PiplBot is PiplBot's web crawler"},{"t":"add","s":"User-agent: PiplBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# anthropic-ai is Anthropic's web crawler"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Claude-Web is Claude’s web crawler"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# TurnitinBot is Turnitin’s web crawler"},{"t":"add","s":"User-agent: TurnitinBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# PetalBot is Petal’s web crawler"},{"t":"add","s":"User-agent: PetalBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# MoodleBot is Moodle’s web crawler"},{"t":"add","s":"User-agent: MoodleBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# magpie-crawler is Brandwatch.com’s web crawler"},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# ia_archiver is Wayback Machine’s web crawler"},{"t":"add","s":"User-agent: ia_archiver"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# omgili is Omgili’s web crawler"},{"t":"add","s":"User-Agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# omgilibot is Omgili’s web crawler"},{"t":"add","s":"User-Agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# No robots are allowed to index private paths:"}]},{"domain":"economist.com","name":"The Economist","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801020419/https://www.economist.com/robots.txt","https://web.archive.org/web/20230901013757/https://www.economist.com/robots.txt"],"diff":[{"t":"ctx","s":"# robots.txt"},{"t":"del","s":"#"},{"t":"ctx","s":""},{"t":"del","s":"# Sitemap"},{"t":"del","s":"Sitemap: https://www.economist.com/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.economist.com/googlenews.xml"},{"t":"del","s":""},{"t":"ctx","s":"# Specific robot directives:"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Mediapartners-Google*"},{"t":"del","s":"Disallow:"},{"t":"add","s":"Allow: /"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: grapeshot"},{"t":"del","s":"Disallow:"},{"t":"add","s":"Allow: /"},{"t":"ctx","s":""},{"t":"add","s":"# Description : GPTBot is OpenAI’s web crawler."},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# No robots are allowed to index private paths:"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Sitemap"},{"t":"add","s":"Sitemap: https://www.economist.com/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.economist.com/googlenews.xml"},{"t":"ctx","s":"# Directories"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /report-abuse/"},{"t":"add","s":"Disallow: /user/"},{"t":"ctx","s":"Disallow: /users/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /vote/"},{"t":"add","s":"Disallow: /taxonomy/term/"},{"t":"ctx","s":"# Paths (no trailing /,  beware this will stop file like /admin.html being"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /contact"},{"t":"add","s":"Disallow: /lab"},{"t":"ctx","s":"Disallow: /logout"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /user"},{"t":"add","s":"Disallow: /uspod"},{"t":"ctx","s":"Disallow: /which-mba"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /bookmarks"},{"t":"del","s":""}]}]}