All sites / News and publishing
Politico
politico.comBlocks AI
Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 18 of the 24 bots in total. It names every major AI bot one by one. (We used the Internet Archive's copy from 3 Oct 2026, because the site would not give our bot its robots.txt: a Cloudflare challenge page.) The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.
Last checked 19 Oct 2026. Rules file from the Internet Archive copy of 3 Oct 2026. Everything we checked · History
Bots that train AI Some blocked
8 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
5 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
5 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 192 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 120 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 177 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 65 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 83 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 216 | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 80 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Named, line 47 | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 62 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 243 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 102 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 277 | Suggests yes |
| AmazonbotAmazon | Blocked | Named, line 56 | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Named, line 147 | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 94 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 110 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 269 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 222 | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Blocked | Named, line 228 | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Ai2Bot-Dolma | Blocked | line 50 | Not in our verified list |
| bigsur.ai | Blocked | line 74 | Not in our verified list |
| ChatGPT Agent | Blocked | line 86 | Not in our verified list |
| Claude-Web | Blocked | line 117 | Not in our verified list |
| CloudVertexBot | Blocked | line 123 | Not in our verified list |
| cohere-ai | Blocked | line 126 | Not in our verified list |
| cohere-training-data-crawler | Blocked | line 129 | Not in our verified list |
| Diffbot | Blocked | line 144 | Documented by its operator |
| FacebookBot | Blocked | line 156 | Not in our verified list |
| FirecrawlAgent | Blocked | line 165 | Not in our verified list |
| Gemini-Deep-Research | Blocked | line 171 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 174 | Documented by its operator |
| GoogleAgent-Mariner | Blocked | line 180 | Not in our verified list |
| GoogleOther | Blocked | line 183 | Documented by its operator |
| GoogleOther-Image | Blocked | line 186 | Documented by its operator |
| GoogleOther-Video | Blocked | line 189 | Documented by its operator |
| ImagesiftBot | Blocked | line 201 | Not in our verified list |
| omgili | Blocked | line 251 | Not in our verified list |
| omgilibot | Blocked | line 254 | Not in our verified list |
| PetalBot | Blocked | line 285 | Not in our verified list |
| Timpibot | Blocked | line 324 | Not in our verified list |
| Webzio-Extended | Blocked | line 333 | Not in our verified list |
| YouBot | Blocked | line 348 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
Sitemap: https://www.politico.com/sitemap.xmlSitemap: https://www.politico.com/news-sitemap.xmlSitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xmlSitemap: https://www.politico.com/liveupdates-sitemap-latest.xmlSitemap: https://www.politico.com/liveupdates-sitemap-content.xmlSitemap: https://www.politico.com/sanity-content/sitemap.xml User-agent: *Disallow: /2stepDisallow: /campaigntrailnotusedDisallow: /CandidateRankingDisallow: /candidatesDisallow: /CustomCompanionTestDisallow: /debateDisallow: /ryanDisallow: /promotionsDisallow: /2014-election/results/mobile/iphoneDisallow: /2014-election/results/mobile/ipadDisallow: /_previewDisallow: /326homeDisallow: /327homeDisallow: /_styleguideDisallow: /newsletters/politico-toplinesDisallow : /*?tab=most-read*Disallow : /*?ml=*Disallow : /*?hp=*Disallow : /*?video=*Disallow : /*?nid=*Disallow : /*?cid=*Disallow : /*?source=*Disallow : /*?slide=*Disallow : /*?category=*Disallow : /*?p=*Disallow : /*?print=* User-agent: Googlebot-NewsDisallow: /newsletters/politico-toplinesAllow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: AddSearchBotDisallow: /… 45 lines not about AI bots User-agent: AI2BotDisallow: / User-agent: Ai2Bot-DolmaDisallow: / User-agent: aiHitBotDisallow: /… 2 lines not about AI bots User-agent: AmazonbotDisallow: / User-agent: AndibotDisallow: /… 2 lines not about AI bots User-agent: anthropic-aiDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: AwarioDisallow: / User-agent: bedrockbotDisallow: /… 5 lines not about AI bots User-agent: bigsur.aiDisallow: / User-agent: Brightbot 1.0Disallow: /… 2 lines not about AI bots User-agent: BytespiderDisallow: / User-agent: CCBotDisallow: / User-agent: ChatGPT AgentDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: ChatGPT-UserDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: Claude-SearchBotDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: Claude-UserDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: Claude-WebDisallow: / User-agent: ClaudeBotDisallow: / User-agent: CloudVertexBotDisallow: / User-agent: cohere-aiDisallow: / User-agent: cohere-training-data-crawlerDisallow: / User-agent: CotoyogiDisallow: / User-agent: CrawlspaceDisallow: / User-agent: Datenbank CrawlerDisallow: / User-agent: DevinDisallow: /… 11 lines not about AI bots User-agent: DiffbotDisallow: / User-agent: DuckAssistBotDisallow: / User-agent: Echobot BotDisallow: / User-agent: EchoboxBotDisallow: /… 5 lines not about AI bots User-agent: FacebookBotDisallow: / User-agent: facebookexternalhitDisallow: / User-agent: Factset_spyderbotDisallow: /… 5 lines not about AI bots User-agent: FirecrawlAgentDisallow: / User-agent: FriendlyCrawlerDisallow: /… 2 lines not about AI bots User-agent: Gemini-Deep-ResearchDisallow: / User-agent: Google-CloudVertexBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: GoogleAgent-MarinerDisallow: / User-agent: GoogleOtherDisallow: / User-agent: GoogleOther-ImageDisallow: / User-agent: GoogleOther-VideoDisallow: / User-agent: GPTBotDisallow: / User-agent: iaskspider/2.0Disallow: / User-agent: ICC-CrawlerDisallow: /… 5 lines not about AI bots User-agent: ImagesiftBotDisallow: / User-agent: img2datasetDisallow: / User-agent: ISSCyberRiskCrawlerDisallow: / User-agent: Kangaroo BotDisallow: / User-agent: LinerBotDisallow: /… 11 lines not about AI bots User-agent: meta-externalagentDisallow: / User-agent: Meta-ExternalAgentDisallow: / User-agent: meta-externalfetcherDisallow: / User-agent: Meta-ExternalFetcherDisallow: / User-agent: MistralAI-UserDisallow: / User-agent: MistralAI-User/1.0Disallow: / User-agent: MyCentralAIScraperBotDisallow: / User-agent: netEstate Imprint CrawlerDisallow: / User-agent: NovaActDisallow: /… 8 lines not about AI bots User-agent: OAI-SearchBotDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: omgiliDisallow: / User-agent: omgilibotDisallow: / User-agent: OperatorDisallow: / User-agent: PanguBotDisallow: / User-agent: PanscientDisallow: / User-agent: panscient.comDisallow: /… 11 lines not about AI bots User-agent: Perplexity-UserDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: PerplexityBotDisallow: /Allow: /sponsored/Allow: /sponsor-contentAllow: /sponsored-content/Allow: /magazine/sponsor-content User-agent: PetalBotDisallow: / User-agent: PhindBotDisallow: / User-agent: Poseidon Research CrawlerDisallow: / User-agent: QualifiedBotDisallow: / User-agent: QuillBotDisallow: / User-agent: quillbot.comDisallow: / User-agent: SBIntuitionsBotDisallow: / User-agent: ScrapyDisallow: / User-agent: SemrushBot-OCOBDisallow: / User-agent: SemrushBot-SWADisallow: / User-agent: Sidetrade indexer botDisallow: / User-agent: ThinkbotDisallow: / User-agent: TikTokSpiderDisallow: /… 35 lines not about AI bots User-agent: TimpibotDisallow: / User-agent: VelenPublicWebCrawlerDisallow: / User-agent: WARDBotDisallow: /… 5 lines not about AI bots User-agent: Webzio-ExtendedDisallow: / User-agent: wpbotDisallow: / User-agent: YaKDisallow: / User-agent: YandexAdditionalDisallow: / User-agent: YandexAdditionalBotDisallow: /… 11 lines not about AI bots User-agent: YouBotDisallow: /
Everything else we checked
| Front door | Human check The homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site (HTTP 403).GET https://politico.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| llms.txt | Skipped: robots.txt request was refused |
| ai.txt | Skipped: robots.txt request was refused |
| TDMRep | Skipped: robots.txt request was refused/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
44 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Sep 2025More blocking Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.Found in the Internet Archive's copies: changed between 1 Aug 2025 and 2 Sep 2025. Evidence: before · after
Show exactly what changed
Disallow: /newsletters/politico-toplines+ Disallow : /*?tab=most-read*+ Disallow : /*?ml=*+ Disallow : /*?hp=*+ Disallow : /*?video=*+ Disallow : /*?nid=*+ Disallow : /*?cid=*+ Disallow : /*?source=*+ Disallow : /*?slide=*+ Disallow : /*?category=*+ Disallow : /*?p=*+ Disallow : /*?print=*+User-agent: Googlebot-News…+ User-agent: AddSearchBot+ Disallow: /++ User-agent: AI2Bot+ Disallow: /++ User-agent: Ai2Bot-Dolma+ Disallow: /++ User-agent: aiHitBot+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: Andibot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Applebot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: Awario+ Disallow: /++ User-agent: bedrockbot+ Disallow: /++ User-agent: bigsur.ai+ Disallow: /++ User-agent: Brightbot 1.0+ Disallow: /++ User-agent: Bytespider+ Disallow: /+User-agent: CCBot… 233 more lines - Jun 2024More blocking Now blocked: ClaudeBot.Found in the Internet Archive's copies: changed between 1 May 2024 and 1 Jun 2024. Evidence: before · after
Show exactly what changed
Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml+ Sitemap: https://www.politico.com/liveupdates-sitemap-latest.xml+ Sitemap: https://www.politico.com/liveupdates-sitemap-content.xml…Disallow: /_styleguide+ Disallow: /newsletters/politico-toplines…Disallow: /magazine/sponsor-content+ Disallow: /newsletters/politico-toplines…Disallow: /++ User-agent: ClaudeBot+ Disallow: / - Jan 2024More blocking Now blocked: CCBot.Found in the Internet Archive's copies: changed between 1 Dec 2023 and 1 Jan 2024. Evidence: before · after
Show exactly what changed
Sitemap: https://www.politico.com/news-sitemap.xml- Sitemap: https://www.politico.com/pro/sitemap.xml+ Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml…Disallow: /2014-election/results/mobile/ipad- Disallow: /staffDisallow: /_preview…Disallow: /magazine/sponsor-content++ User-agent: CCBot+ Disallow: /
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.