{"domain":"politico.com","name":"Politico","category":"news","subcategory":"digital","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 18 of the 24 bots in total. It names every major AI bot one by one. (We used the Internet Archive's copy from 3 Oct 2026, because the site would not give our bot its robots.txt: a Cloudflare challenge page.) The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.278Z","last_checked":"2026-10-19T13:35:14.736Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://politico.com"},"robots":{"via":{"type":"archive","date":"2026-10-03","url":"https://web.archive.org/web/20261003091219/https://www.politico.com/robots.txt","blob":"60d23f61bee9b1e6c11b21b17265e3fdc667e06a14e4e2907a222a4c02d323b6"},"kind":"refused","http":403,"error":null,"url":"https://www.politico.com/robots.txt","hops":[{"from":"https://politico.com/robots.txt","to":"https://www.politico.com/robots.txt","status":301}],"blob":"60d23f61bee9b1e6c11b21b17265e3fdc667e06a14e4e2907a222a4c02d323b6","text":"Sitemap: https://www.politico.com/sitemap.xml\r\nSitemap: https://www.politico.com/news-sitemap.xml\r\nSitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml\r\nSitemap: https://www.politico.com/liveupdates-sitemap-latest.xml\r\nSitemap: https://www.politico.com/liveupdates-sitemap-content.xml\r\nSitemap: https://www.politico.com/sanity-content/sitemap.xml\r\n\r\nUser-agent: *\r\nDisallow: /2step\r\nDisallow: /campaigntrailnotused\r\nDisallow: /CandidateRanking\r\nDisallow: /candidates\r\nDisallow: /CustomCompanionTest\r\nDisallow: /debate\r\nDisallow: /ryan\r\nDisallow: /promotions\r\nDisallow: /2014-election/results/mobile/iphone\r\nDisallow: /2014-election/results/mobile/ipad\r\nDisallow: /_preview\r\nDisallow: /326home\r\nDisallow: /327home\r\nDisallow: /_styleguide\r\nDisallow: /newsletters/politico-toplines\r\nDisallow : /*?tab=most-read*\r\nDisallow : /*?ml=*\r\nDisallow : /*?hp=*\r\nDisallow : /*?video=*\r\nDisallow : /*?nid=*\r\nDisallow : /*?cid=*\r\nDisallow : /*?source=*\r\nDisallow : /*?slide=*\r\nDisallow : /*?category=*\r\nDisallow : /*?p=*\r\nDisallow : /*?print=*\r\n\r\n\r\nUser-agent: Googlebot-News\r\nDisallow: /newsletters/politico-toplines\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\nUser-agent: AddSearchBot\r\nDisallow: /\r\n\r\nUser-agent: AI2Bot\r\nDisallow: /\r\n\r\nUser-agent: Ai2Bot-Dolma\r\nDisallow: /\r\n\r\nUser-agent: aiHitBot\r\nDisallow: /\r\n\r\nUser-agent: Amazonbot\r\nDisallow: /\r\n\r\nUser-agent: Andibot\r\nDisallow: /\r\n\r\nUser-agent: anthropic-ai\r\nDisallow: /\r\n\r\nUser-agent: Applebot-Extended\r\nDisallow: /\r\n\r\nUser-agent: Awario\r\nDisallow: /\r\n\r\nUser-agent: bedrockbot\r\nDisallow: /\r\n\r\nUser-agent: bigsur.ai\r\nDisallow: /\r\n\r\nUser-agent: Brightbot 1.0\r\nDisallow: /\r\n\r\nUser-agent: Bytespider\r\nDisallow: /\r\n\r\nUser-agent: CCBot\r\nDisallow: /\r\n\r\nUser-agent: ChatGPT Agent\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: ChatGPT-User\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: Claude-SearchBot\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: Claude-User\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\nUser-agent: Claude-Web\r\nDisallow: /\r\n\r\nUser-agent: ClaudeBot\r\nDisallow: /\r\n\r\nUser-agent: CloudVertexBot\r\nDisallow: /\r\n\r\nUser-agent: cohere-ai\r\nDisallow: /\r\n\r\nUser-agent: cohere-training-data-crawler\r\nDisallow: /\r\n\r\nUser-agent: Cotoyogi\r\nDisallow: /\r\n\r\nUser-agent: Crawlspace\r\nDisallow: /\r\n\r\nUser-agent: Datenbank Crawler\r\nDisallow: /\r\n\r\nUser-agent: Devin\r\nDisallow: /\r\n\r\nUser-agent: Diffbot\r\nDisallow: /\r\n\r\nUser-agent: DuckAssistBot\r\nDisallow: /\r\n\r\nUser-agent: Echobot Bot\r\nDisallow: /\r\n\r\nUser-agent: EchoboxBot\r\nDisallow: /\r\n\r\nUser-agent: FacebookBot\r\nDisallow: /\r\n\r\nUser-agent: facebookexternalhit\r\nDisallow: /\r\n\r\nUser-agent: Factset_spyderbot\r\nDisallow: /\r\n\r\nUser-agent: FirecrawlAgent\r\nDisallow: /\r\n\r\nUser-agent: FriendlyCrawler\r\nDisallow: /\r\n\r\nUser-agent: Gemini-Deep-Research\r\nDisallow: /\r\n\r\nUser-agent: Google-CloudVertexBot\r\nDisallow: /\r\n\r\nUser-agent: Google-Extended\r\nDisallow: /\r\n\r\nUser-agent: GoogleAgent-Mariner\r\nDisallow: /\r\n\r\nUser-agent: GoogleOther\r\nDisallow: /\r\n\r\nUser-agent: GoogleOther-Image\r\nDisallow: /\r\n\r\nUser-agent: GoogleOther-Video\r\nDisallow: /\r\n\r\nUser-agent: GPTBot\r\nDisallow: /\r\n\r\nUser-agent: iaskspider/2.0\r\nDisallow: /\r\n\r\nUser-agent: ICC-Crawler\r\nDisallow: /\r\n\r\nUser-agent: ImagesiftBot\r\nDisallow: /\r\n\r\nUser-agent: img2dataset\r\nDisallow: /\r\n\r\nUser-agent: ISSCyberRiskCrawler\r\nDisallow: /\r\n\r\nUser-agent: Kangaroo Bot\r\nDisallow: /\r\n\r\nUser-agent: LinerBot\r\nDisallow: /\r\n\r\nUser-agent: meta-externalagent\r\nDisallow: /\r\n\r\nUser-agent: Meta-ExternalAgent\r\nDisallow: /\r\n\r\nUser-agent: meta-externalfetcher\r\nDisallow: /\r\n\r\nUser-agent: Meta-ExternalFetcher\r\nDisallow: /\r\n\r\nUser-agent: MistralAI-User\r\nDisallow: /\r\n\r\nUser-agent: MistralAI-User/1.0\r\nDisallow: /\r\n\r\nUser-agent: MyCentralAIScraperBot\r\nDisallow: /\r\n\r\nUser-agent: netEstate Imprint Crawler\r\nDisallow: /\r\n\r\nUser-agent: NovaAct\r\nDisallow: /\r\n\r\nUser-agent: OAI-SearchBot\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: omgili\r\nDisallow: /\r\n\r\nUser-agent: omgilibot\r\nDisallow: /\r\n\r\nUser-agent: Operator\r\nDisallow: /\r\n\r\nUser-agent: PanguBot\r\nDisallow: /\r\n\r\nUser-agent: Panscient\r\nDisallow: /\r\n\r\nUser-agent: panscient.com\r\nDisallow: /\r\n\r\nUser-agent: Perplexity-User\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: PerplexityBot\r\nDisallow: /\r\nAllow: /sponsored/\r\nAllow: /sponsor-content\r\nAllow: /sponsored-content/\r\nAllow: /magazine/sponsor-content\r\n\r\n\r\nUser-agent: PetalBot\r\nDisallow: /\r\n\r\nUser-agent: PhindBot\r\nDisallow: /\r\n\r\nUser-agent: Poseidon Research Crawler\r\nDisallow: /\r\n\r\nUser-agent: QualifiedBot\r\nDisallow: /\r\n\r\nUser-agent: QuillBot\r\nDisallow: /\r\n\r\nUser-agent: quillbot.com\r\nDisallow: /\r\n\r\nUser-agent: SBIntuitionsBot\r\nDisallow: /\r\n\r\nUser-agent: Scrapy\r\nDisallow: /\r\n\r\nUser-agent: SemrushBot-OCOB\r\nDisallow: /\r\n\r\nUser-agent: SemrushBot-SWA\r\nDisallow: /\r\n\r\nUser-agent: Sidetrade indexer bot\r\nDisallow: /\r\n\r\nUser-agent: Thinkbot\r\nDisallow: /\r\n\r\nUser-agent: TikTokSpider\r\nDisallow: /\r\n\r\nUser-agent: Timpibot\r\nDisallow: /\r\n\r\nUser-agent: VelenPublicWebCrawler\r\nDisallow: /\r\n\r\nUser-agent: WARDBot\r\nDisallow: /\r\n\r\nUser-agent: Webzio-Extended\r\nDisallow: /\r\n\r\nUser-agent: wpbot\r\nDisallow: /\r\n\r\nUser-agent: YaK\r\nDisallow: /\r\n\r\nUser-agent: YandexAdditional\r\nDisallow: /\r\n\r\nUser-agent: YandexAdditionalBot\r\nDisallow: /\r\n\r\nUser-agent: YouBot\r\nDisallow: /","refused":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Googlebot-News","AddSearchBot","AI2Bot","Ai2Bot-Dolma","aiHitBot","Amazonbot","Andibot","anthropic-ai","Applebot-Extended","Awario","bedrockbot","bigsur.ai","Brightbot 1.0","Bytespider","CCBot","ChatGPT Agent","ChatGPT-User","Claude-SearchBot","Claude-User","Claude-Web","ClaudeBot","CloudVertexBot","cohere-ai","cohere-training-data-crawler","Cotoyogi","Crawlspace","Datenbank Crawler","Devin","Diffbot","DuckAssistBot","Echobot Bot","EchoboxBot","FacebookBot","facebookexternalhit","Factset_spyderbot","FirecrawlAgent","FriendlyCrawler","Gemini-Deep-Research","Google-CloudVertexBot","Google-Extended","GoogleAgent-Mariner","GoogleOther","GoogleOther-Image","GoogleOther-Video","GPTBot","iaskspider/2.0","ICC-Crawler","ImagesiftBot","img2dataset","ISSCyberRiskCrawler","Kangaroo Bot","LinerBot","meta-externalagent","Meta-ExternalAgent","meta-externalfetcher","Meta-ExternalFetcher","MistralAI-User","MistralAI-User/1.0","MyCentralAIScraperBot","netEstate Imprint Crawler","NovaAct","OAI-SearchBot","omgili","omgilibot","Operator","PanguBot","Panscient","panscient.com","Perplexity-User","PerplexityBot","PetalBot","PhindBot","Poseidon Research Crawler","QualifiedBot","QuillBot","quillbot.com","SBIntuitionsBot","Scrapy","SemrushBot-OCOB","SemrushBot-SWA","Sidetrade indexer bot","Thinkbot","TikTokSpider","Timpibot","VelenPublicWebCrawler","WARDBot","Webzio-Extended","wpbot","YaK","YandexAdditional","YandexAdditionalBot","YouBot"],"lines":349},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":193},"any":{"allowed":false,"line":193},"search":{"allowed":false,"line":193},"cart":{"allowed":false,"line":193},"checkout":{"allowed":false,"line":193},"account":{"allowed":false,"line":193}},"disallows":1,"lines":[192,193],"groupRanges":[[192,193]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":121},"any":{"allowed":false,"line":121},"search":{"allowed":false,"line":121},"cart":{"allowed":false,"line":121},"checkout":{"allowed":false,"line":121},"account":{"allowed":false,"line":121}},"disallows":1,"lines":[120,121],"groupRanges":[[120,121]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":178},"any":{"allowed":false,"line":178},"search":{"allowed":false,"line":178},"cart":{"allowed":false,"line":178},"checkout":{"allowed":false,"line":178},"account":{"allowed":false,"line":178}},"disallows":1,"lines":[177,178],"groupRanges":[[177,178]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":66},"any":{"allowed":false,"line":66},"search":{"allowed":false,"line":66},"cart":{"allowed":false,"line":66},"checkout":{"allowed":false,"line":66},"account":{"allowed":false,"line":66}},"disallows":1,"lines":[65,66],"groupRanges":[[65,66]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":84},"any":{"allowed":false,"line":84},"search":{"allowed":false,"line":84},"cart":{"allowed":false,"line":84},"checkout":{"allowed":false,"line":84},"account":{"allowed":false,"line":84}},"disallows":1,"lines":[83,84],"groupRanges":[[83,84]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":217},"any":{"allowed":false,"line":217},"search":{"allowed":false,"line":217},"cart":{"allowed":false,"line":217},"checkout":{"allowed":false,"line":217},"account":{"allowed":false,"line":217}},"disallows":2,"lines":[216,217,219,220],"groupRanges":[[216,217],[219,220]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":81},"any":{"allowed":false,"line":81},"search":{"allowed":false,"line":81},"cart":{"allowed":false,"line":81},"checkout":{"allowed":false,"line":81},"account":{"allowed":false,"line":81}},"disallows":1,"lines":[80,81],"groupRanges":[[80,81]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":48},"any":{"allowed":false,"line":48},"search":{"allowed":false,"line":48},"cart":{"allowed":false,"line":48},"checkout":{"allowed":false,"line":48},"account":{"allowed":false,"line":48}},"disallows":1,"lines":[47,48],"groupRanges":[[47,48]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":63},"any":{"allowed":false,"line":63},"search":{"allowed":false,"line":63},"cart":{"allowed":false,"line":63},"checkout":{"allowed":false,"line":63},"account":{"allowed":false,"line":63}},"disallows":1,"lines":[62,63],"groupRanges":[[62,63]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":244},"any":{"allowed":false,"line":244},"search":{"allowed":false,"line":244},"cart":{"allowed":false,"line":244},"checkout":{"allowed":false,"line":244},"account":{"allowed":false,"line":244}},"disallows":1,"lines":[243,248],"groupRanges":[[243,248]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":103},"any":{"allowed":false,"line":103},"search":{"allowed":false,"line":103},"cart":{"allowed":false,"line":103},"checkout":{"allowed":false,"line":103},"account":{"allowed":false,"line":103}},"disallows":1,"lines":[102,107],"groupRanges":[[102,107]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":278},"any":{"allowed":false,"line":278},"search":{"allowed":false,"line":278},"cart":{"allowed":false,"line":278},"checkout":{"allowed":false,"line":278},"account":{"allowed":false,"line":278}},"disallows":1,"lines":[277,282],"groupRanges":[[277,282]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":57},"any":{"allowed":false,"line":57},"search":{"allowed":false,"line":57},"cart":{"allowed":false,"line":57},"checkout":{"allowed":false,"line":57},"account":{"allowed":false,"line":57}},"disallows":1,"lines":[56,57],"groupRanges":[[56,57]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":148},"any":{"allowed":false,"line":148},"search":{"allowed":false,"line":148},"cart":{"allowed":false,"line":148},"checkout":{"allowed":false,"line":148},"account":{"allowed":false,"line":148}},"disallows":1,"lines":[147,148],"groupRanges":[[147,148]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":95},"any":{"allowed":false,"line":95},"search":{"allowed":false,"line":95},"cart":{"allowed":false,"line":95},"checkout":{"allowed":false,"line":95},"account":{"allowed":false,"line":95}},"disallows":1,"lines":[94,99],"groupRanges":[[94,99]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":111},"any":{"allowed":false,"line":111},"search":{"allowed":false,"line":111},"cart":{"allowed":false,"line":111},"checkout":{"allowed":false,"line":111},"account":{"allowed":false,"line":111}},"disallows":1,"lines":[110,115],"groupRanges":[[110,115]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":270},"any":{"allowed":false,"line":270},"search":{"allowed":false,"line":270},"cart":{"allowed":false,"line":270},"checkout":{"allowed":false,"line":270},"account":{"allowed":false,"line":270}},"disallows":1,"lines":[269,274],"groupRanges":[[269,274]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":223},"any":{"allowed":false,"line":223},"search":{"allowed":false,"line":223},"cart":{"allowed":false,"line":223},"checkout":{"allowed":false,"line":223},"account":{"allowed":false,"line":223}},"disallows":2,"lines":[222,223,225,226],"groupRanges":[[222,223],[225,226]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":229},"any":{"allowed":false,"line":229},"search":{"allowed":false,"line":229},"cart":{"allowed":false,"line":229},"checkout":{"allowed":false,"line":229},"account":{"allowed":false,"line":229}},"disallows":2,"lines":[228,229,231,232],"groupRanges":[[228,229],[231,232]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[8,34],"groupRanges":[[8,34]]}],"lanes":{"training":{"level":"mixed","none":8,"full":1,"restricted":0,"n":9},"search":{"level":"mixed","none":5,"full":3,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":5,"full":2,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"skipped","reason":"Skipped: robots.txt request was refused"},"ai_txt":{"state":"skipped","reason":"Skipped: robots.txt request was refused"},"tdmrep":{"state":"skipped","reason":"Skipped: robots.txt request was refused"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":44,"first":"2023-01-01","archive_points":44,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2025-09","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2025-10","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2025-11","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2025-12","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-01","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-02","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-03","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-04","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-05","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-06","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-07","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-08","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-09","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-10","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18}]},"events":[{"domain":"politico.com","name":"Politico","category":"news","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-09-02"],"summary":"Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.","dir":"up","up":["GPTBot","Google-Extended","Applebot-Extended","Meta-ExternalAgent","Bytespider","AI2Bot","anthropic-ai","OAI-SearchBot","Claude-SearchBot","PerplexityBot","Amazonbot","DuckAssistBot","ChatGPT-User","Claude-User","Perplexity-User","Meta-ExternalFetcher","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"blocked","evidence":["https://web.archive.org/web/20250801015710/https://www.politico.com/robots.txt","https://web.archive.org/web/20250902050836/https://www.politico.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /newsletters/politico-toplines"},{"t":"add","s":"Disallow : /*?tab=most-read*"},{"t":"add","s":"Disallow : /*?ml=*"},{"t":"add","s":"Disallow : /*?hp=*"},{"t":"add","s":"Disallow : /*?video=*"},{"t":"add","s":"Disallow : /*?nid=*"},{"t":"add","s":"Disallow : /*?cid=*"},{"t":"add","s":"Disallow : /*?source=*"},{"t":"add","s":"Disallow : /*?slide=*"},{"t":"add","s":"Disallow : /*?category=*"},{"t":"add","s":"Disallow : /*?p=*"},{"t":"add","s":"Disallow : /*?print=*"},{"t":"ctx","s":""},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Googlebot-News"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AddSearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: aiHitBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Andibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Awario"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: bedrockbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: bigsur.ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Brightbot 1.0"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: CCBot"},{"t":"gap","s":"… 233 more lines"}]},{"domain":"politico.com","name":"Politico","category":"news","date":"2024-06","precision":"month","source":"archive","provisional":false,"between":["2024-05-01","2024-06-01"],"summary":"Now blocked: ClaudeBot.","dir":"up","up":["ClaudeBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240501024436/https://www.politico.com/robots.txt","https://web.archive.org/web/20240601005552/https://www.politico.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml"},{"t":"add","s":"Sitemap: https://www.politico.com/liveupdates-sitemap-latest.xml"},{"t":"add","s":"Sitemap: https://www.politico.com/liveupdates-sitemap-content.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /_styleguide"},{"t":"add","s":"Disallow: /newsletters/politico-toplines"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /magazine/sponsor-content"},{"t":"add","s":"Disallow: /newsletters/politico-toplines"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"}]},{"domain":"politico.com","name":"Politico","category":"news","date":"2024-01","precision":"month","source":"archive","provisional":false,"between":["2023-12-01","2024-01-01"],"summary":"Now blocked: CCBot.","dir":"up","up":["CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20231201000722/https://www.politico.com/robots.txt","https://web.archive.org/web/20240101004219/https://www.politico.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.politico.com/news-sitemap.xml"},{"t":"del","s":"Sitemap: https://www.politico.com/pro/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /2014-election/results/mobile/ipad"},{"t":"del","s":"Disallow: /staff"},{"t":"ctx","s":"Disallow: /_preview"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /magazine/sponsor-content"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"}]}]}