{"domain":"cnn.com","name":"CNN","category":"news","subcategory":"broadcaster","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 17 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Applebot-Extended, CCBot and OAI-SearchBot. Google-Extended and Meta-ExternalAgent can reach most of it.","computed_at":"2026-10-20T03:43:15.189Z","last_checked":"2026-10-19T13:34:16.305Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.cnn.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.cnn.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.cnn.com/robots.txt","hops":[{"from":"https://cnn.com/robots.txt","to":"https://www.cnn.com/robots.txt","status":301}],"blob":"832cbd6cc1e8f1586a6e23fdd3f076fadebb8e06304d57f08e4b3a73d0635671","text":"Sitemap: https://www.cnn.com/sitemap/news.xml\nSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml\nSitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml\nSitemap: https://www.cnn.com/sitemap/section/politics.xml\nSitemap: https://www.cnn.com/sitemap/article/opinions.xml\nSitemap: https://www.cnn.com/sitemap/article.xml\nSitemap: https://www.cnn.com/sitemap/section.xml\nSitemap: https://www.cnn.com/sitemap/video.xml\nSitemap: https://www.cnn.com/sitemap/gallery.xml\nSitemap: https://www.cnn.com/sitemap/markets/stocks.xml\nSitemap: https://www.cnn.com/sitemap/live-story.xml\nSitemap: https://www.cnn.com/sitemap/election-center/politics.xml\nSitemap: https://www.cnn.com/sitemap/tve.xml\nUser-agent: AI2Bot\nUser-agent: Ai2Bot-Dolma\nUser-agent: AliyunSecBot\nUser-agent: Amazonbot\nUser-agent: amzn-searchbot\nUser-agent: amzn-user\nUser-agent: anthropic-ai\nUser-agent: Applebot-Extended\nUser-agent: Archive.org_bot\nUser-agent: AwarioRssBot\nUser-agent: AwarioSmartBot\nUser-agent: Brightbot 1.0\nUser-agent: Bytespider\nUser-agent: CCBot\nUser-agent: ChatGPT-User\nUser-agent: ClaudeBot\nUser-agent: Claude-SearchBot\nUser-agent: Claude-User\nUser-agent: Claude-Web\nUser-agent: cohere-ai\nUser-agent: cohere-training-data-crawler\nUser-agent: Crawlspace\nUser-agent: DataForSeoBot\nUser-agent: Diffbot\nUser-agent: DuckAssistBot\nUser-agent: EchoboxBot\nUser-agent: Exabot\nUser-agent: ExaSearchBot\nUser-agent: ExaUser\nUser-agent: FacebookBot\nUser-agent: FriendlyCrawler\nUser-agent: GPTBot\nUser-agent: GoogleOther\nUser-agent: GoogleOther-Image\nUser-agent: GoogleOther-Video\nUser-Agent: IAB-Tech-Lab\nUser-agent: iaskspider/2.0\nUser-agent: ICC-Crawler\nUser-agent: img2dataset\nUser-agent: ISSCyberRiskCrawler\nUser-agent: ImagesiftBot\nUser-agent: Kangaroo Bot\nUser-agent: magpie-crawler\nUser-agent: MistralAI-user\nUser-agent: MyCentralAIScraperBot\nUser-agent: NewsNow\nUser-agent: news-please\nUser-agent: OAI-SearchBot\nUser-agent: omgili\nUser-agent: omgilibot\nUser-agent: PanguBot\nUser-agent: Panscient\nUser-agent: PiplBot\nUser-agent: PerplexityBot\nUser-agent: Perplexity-User\nUser-agent: PetalBot\nUser-agent: Poseidon Research Crawler\nUser-agent: QuillBot\nUser-agent: quillbot.com\nUser-agent: Quora-Bot\nUser-agent: SBIntuitionsBot\nUser-agent: Scrapy\nUser-agent: SeekrBot\nUser-agent: SemrushBot-OCOB\nUser-agent: SemrushBot-SWA\nUser-agent: SeznamHomepageCrawler\nUser-agent: ShapBot\nUser-agent: Shap-User\nUser-agent: Sidetrade indexer bot\nUser-agent: TaraGroup Intelligence Bot\nUser-agent: Timpibot\nUser-agent: TurnitinBot\nUser-agent: VelenPublicWebCrawler\nUser-agent: ViennaTinyBot\nUser-agent: Webzio-Extended\nUser-agent: YandexAdditional\nUser-agent: YandexAdditionalBot\nUser-agent: YouBot\nDisallow: /\nUser-agent: *\nAllow: /partners/ipad/live-video.json\nDisallow: /*.jsx$\nDisallow: *.jsx$\nDisallow: /*.jsx/\nDisallow: *.jsx?\nDisallow: /ads/\nDisallow: /aol/\nDisallow: /api/\nDisallow: /beta/\nDisallow: /browsers/\nDisallow: /cl/\nDisallow: /cnews/\nDisallow: /cnn_adspaces\nDisallow: /cnnbeta/\nDisallow: /cnnintl_adspaces\nDisallow: /development\nDisallow: /editionssi\nDisallow: /help/cnnx.html\nDisallow: /NewsPass\nDisallow: /NOKIA\nDisallow: /partners/\nDisallow: /pipeline/\nDisallow: /pointroll/\nDisallow: /POLLSERVER/\nDisallow: /pr/\nDisallow: /PV/\nDisallow: /Quickcast/\nDisallow: /quickcast/\nDisallow: /QUICKNEWS/\nDisallow: /search\nDisallow: /subscriptions/video-docs\nDisallow: /terms\nDisallow: /test/\nDisallow: /virtual/\nDisallow: /WEB-INF/\nDisallow: /web.projects/\nDisallow: /webview/\nUser-agent: Googlebot-News\nDisallow: /sponsor\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["AI2Bot","Ai2Bot-Dolma","AliyunSecBot","Amazonbot","amzn-searchbot","amzn-user","anthropic-ai","Applebot-Extended","Archive.org_bot","AwarioRssBot","AwarioSmartBot","Brightbot 1.0","Bytespider","CCBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","Claude-Web","cohere-ai","cohere-training-data-crawler","Crawlspace","DataForSeoBot","Diffbot","DuckAssistBot","EchoboxBot","Exabot","ExaSearchBot","ExaUser","FacebookBot","FriendlyCrawler","GPTBot","GoogleOther","GoogleOther-Image","GoogleOther-Video","IAB-Tech-Lab","iaskspider/2.0","ICC-Crawler","img2dataset","ISSCyberRiskCrawler","ImagesiftBot","Kangaroo Bot","magpie-crawler","MistralAI-user","MyCentralAIScraperBot","NewsNow","news-please","OAI-SearchBot","omgili","omgilibot","PanguBot","Panscient","PiplBot","PerplexityBot","Perplexity-User","PetalBot","Poseidon Research Crawler","QuillBot","quillbot.com","Quora-Bot","SBIntuitionsBot","Scrapy","SeekrBot","SemrushBot-OCOB","SemrushBot-SWA","SeznamHomepageCrawler","ShapBot","Shap-User","Sidetrade indexer bot","TaraGroup Intelligence Bot","Timpibot","TurnitinBot","VelenPublicWebCrawler","ViennaTinyBot","Webzio-Extended","YandexAdditional","YandexAdditionalBot","YouBot","*","Googlebot-News"],"lines":133},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Amzn-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":92},"any":{"allowed":false,"line":92},"search":{"allowed":false,"line":92},"cart":{"allowed":false,"line":92},"checkout":{"allowed":false,"line":92},"account":{"allowed":false,"line":92}},"disallows":1,"lines":[14,92],"groupRanges":[[14,92]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":123},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":36,"lines":[93,130],"groupRanges":[[93,130]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":6,"full":2,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":5,"full":2,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":["Fastly"],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.cnn.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.cnn.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.cnn.com/.well-known/tdmrep.json"},"meta":{"title":"Breaking News, Latest News and Videos | CNN","robotsMeta":[],"noai":[],"noindex":false,"jsonLdTypes":["WebPage","NewsMediaOrganization","WebSite","SearchAction"],"platform":null,"textRatio":0.003,"llmsLink":false},"x_robots_tag":null,"tos":{"url":"https://www.cnn.com/terms","source":"discovered","skipped":"Disallowed for AgentAccessIndexBot by robots.txt"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":18},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17}]},"events":[{"domain":"cnn.com","name":"CNN","category":"news","date":"2026-10","precision":"month","source":"archive+checker","provisional":true,"between":["2026-10-01","2026-10-19"],"summary":"Now allowed: Google-Extended.","dir":"down","up":[],"down":["Google-Extended"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20261001180945/https://www.cnn.com/robots.txt","https://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"User-agent: Exabot"},{"t":"add","s":"User-agent: ExaSearchBot"},{"t":"add","s":"User-agent: ExaUser"},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"del","s":"User-agent: Google-Extended"},{"t":"ctx","s":"User-agent: GoogleOther"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: SeznamHomepageCrawler"},{"t":"add","s":"User-agent: ShapBot"},{"t":"add","s":"User-agent: Shap-User"},{"t":"ctx","s":"User-agent: Sidetrade indexer bot"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-03-01","2026-06-02"],"summary":"Now blocked: Claude-SearchBot, Claude-User, Perplexity-User and MistralAI-User.","dir":"up","up":["Claude-SearchBot","Claude-User","Perplexity-User","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260301162736/https://www.cnn.com/robots.txt","https://web.archive.org/web/20260602140226/https://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/news.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/sitemaps/sitemap-section.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"User-agent: AliyunSecBot"},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Applebot-Extended"},{"t":"add","s":"User-agent: Archive.org_bot"},{"t":"ctx","s":"User-agent: AwarioRssBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: ClaudeBot"},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"User-agent: Claude-User"},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: EchoboxBot"},{"t":"add","s":"User-agent: Exabot"},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: GoogleOther-Video"},{"t":"add","s":"User-Agent: IAB-Tech-Lab"},{"t":"ctx","s":"User-agent: iaskspider/2.0"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"add","s":"User-agent: MistralAI-user"},{"t":"add","s":"User-agent: MyCentralAIScraperBot"},{"t":"ctx","s":"User-agent: NewsNow"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"add","s":"User-agent: Poseidon Research Crawler"},{"t":"ctx","s":"User-agent: QuillBot"},{"t":"ctx","s":"User-agent: quillbot.com"},{"t":"add","s":"User-agent: Quora-Bot"},{"t":"ctx","s":"User-agent: SBIntuitionsBot"},{"t":"ctx","s":"User-agent: Scrapy"},{"t":"add","s":"User-agent: SeekrBot"},{"t":"ctx","s":"User-agent: SemrushBot-OCOB"},{"t":"ctx","s":"User-agent: SemrushBot-SWA"},{"t":"add","s":"User-agent: SeznamHomepageCrawler"},{"t":"ctx","s":"User-agent: Sidetrade indexer bot"},{"t":"add","s":"User-agent: TaraGroup Intelligence Bot"},{"t":"ctx","s":"User-agent: Timpibot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: VelenPublicWebCrawler"},{"t":"add","s":"User-agent: ViennaTinyBot"},{"t":"ctx","s":"User-agent: Webzio-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /pr/"},{"t":"del","s":"Disallow: /privacy"},{"t":"ctx","s":"Disallow: /PV/"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2026-01","precision":"month","source":"archive","provisional":false,"between":["2025-10-01","2026-01-02"],"summary":"Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.","dir":"mixed","up":["Amzn-SearchBot","Amzn-User"],"down":["Meta-ExternalAgent","Meta-ExternalFetcher"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20251001121744/https://www.cnn.com/robots.txt","https://web.archive.org/web/20260102174350/https://www.cnn.com/robots.txt"],"diff":[{"t":"del","s":"Sitemap: https://www.cnn.com/sitemaps/cnn/index.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/sitemaps/cnn/news.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/news.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/ampstories/sitemap.xml"},{"t":"del","s":"Sitemap: https://edition.cnn.com/sitemaps/news.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/section/politics.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/tve.xml"},{"t":"ctx","s":"User-agent: AI2Bot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"add","s":"User-agent: amzn-searchbot"},{"t":"add","s":"User-agent: amzn-user"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"del","s":"User-agent: Applebot"},{"t":"ctx","s":"User-agent: Applebot-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: DuckAssistBot"},{"t":"add","s":"User-agent: EchoboxBot"},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"del","s":"User-agent: Meta-ExternalAgent"},{"t":"del","s":"User-agent: Meta-ExternalFetcher"},{"t":"ctx","s":"User-agent: NewsNow"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: PanguBot"},{"t":"del","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: Panscient"},{"t":"ctx","s":"User-agent: PiplBot"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"add","s":"User-agent: QuillBot"},{"t":"add","s":"User-agent: quillbot.com"},{"t":"add","s":"User-agent: SBIntuitionsBot"},{"t":"ctx","s":"User-agent: Scrapy"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Webzio-Extended"},{"t":"add","s":"User-agent: YandexAdditional"},{"t":"add","s":"User-agent: YandexAdditionalBot"},{"t":"ctx","s":"User-agent: YouBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /webview/"},{"t":"del","s":"Disallow: /games"},{"t":"ctx","s":"User-agent: Googlebot-News"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2025-04","precision":"month","source":"archive","provisional":false,"between":["2025-03-01","2025-04-01"],"summary":"Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot, OAI-SearchBot, Amazonbot, DuckAssistBot and Meta-ExternalFetcher.","dir":"up","up":["Applebot-Extended","Meta-ExternalAgent","AI2Bot","OAI-SearchBot","Amazonbot","DuckAssistBot","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250301011919/https://www.cnn.com/robots.txt","https://web.archive.org/web/20250401004714/https://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/live-story.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml"},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"ctx","s":"User-agent: AwarioRssBot"},{"t":"ctx","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"User-agent: Brightbot 1.0"},{"t":"ctx","s":"User-agent: Bytespider"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: cohere-ai"},{"t":"add","s":"User-agent: cohere-training-data-crawler"},{"t":"add","s":"User-agent: Crawlspace"},{"t":"ctx","s":"User-agent: DataForSeoBot"},{"t":"ctx","s":"User-agent: Diffbot"},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"add","s":"User-agent: FriendlyCrawler"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: GoogleOther"},{"t":"add","s":"User-agent: GoogleOther-Image"},{"t":"add","s":"User-agent: GoogleOther-Video"},{"t":"add","s":"User-agent: iaskspider/2.0"},{"t":"add","s":"User-agent: ICC-Crawler"},{"t":"add","s":"User-agent: img2dataset"},{"t":"add","s":"User-agent: ISSCyberRiskCrawler"},{"t":"add","s":"User-agent: ImagesiftBot"},{"t":"add","s":"User-agent: Kangaroo Bot"},{"t":"ctx","s":"User-agent: magpie-crawler"},{"t":"add","s":"User-agent: Meta-ExternalAgent"},{"t":"add","s":"User-agent: Meta-ExternalFetcher"},{"t":"ctx","s":"User-agent: NewsNow"},{"t":"ctx","s":"User-agent: news-please"},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"ctx","s":"User-agent: omgili"},{"t":"ctx","s":"User-agent: omgilibot"},{"t":"add","s":"User-agent: PanguBot"},{"t":"ctx","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: PiplBot"},{"t":"add","s":"User-agent: PetalBot"},{"t":"ctx","s":"User-agent: Scrapy"},{"t":"add","s":"User-agent: SemrushBot-OCOB"},{"t":"add","s":"User-agent: SemrushBot-SWA"},{"t":"add","s":"User-agent: Sidetrade indexer bot"},{"t":"add","s":"User-agent: Timpibot"},{"t":"ctx","s":"User-agent: TurnitinBot"},{"t":"add","s":"User-agent: VelenPublicWebCrawler"},{"t":"add","s":"User-agent: Webzio-Extended"},{"t":"add","s":"User-agent: YouBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /search"},{"t":"add","s":"Disallow: /subscriptions/video-docs"},{"t":"ctx","s":"Disallow: /terms"},{"t":"gap","s":"…"},{"t":"gap","s":"… 5 more lines"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2024-05","precision":"month","source":"archive","provisional":false,"between":["2024-04-01","2024-05-01"],"summary":"Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot and ChatGPT-User.","dir":"up","up":["ClaudeBot","CCBot","Bytespider","anthropic-ai","PerplexityBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240401010308/https://www.cnn.com/robots.txt","https://web.archive.org/web/20240501001430/https://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xml"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/section/politics.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/gallery.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"User-agent: AwarioRssBot"},{"t":"add","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"User-agent: FacebookBot"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"User-agent: NewsNow"},{"t":"add","s":"User-agent: news-please"},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: Scrapy"},{"t":"add","s":"User-agent: TurnitinBot"},{"t":"ctx","s":"Disallow: /"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001000140/https://www.cnn.com/robots.txt","https://web.archive.org/web/20231101001304/http://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/video.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/gallery.xml"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: *"}]},{"domain":"cnn.com","name":"CNN","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801014937/http://www.cnn.com/robots.txt","https://web.archive.org/web/20230901001844/https://www.cnn.com/robots.txt"],"diff":[{"t":"ctx","s":"Sitemap: https://www.cnn.com/sitemap/section/politics.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/article/opinions.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/article.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/section.xml"},{"t":"add","s":"Sitemap: https://www.cnn.com/sitemap/video.xml"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /aol/"},{"t":"add","s":"Disallow: /api/"},{"t":"ctx","s":"Disallow: /beta/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /QUICKNEWS/"},{"t":"del","s":"Disallow: /search/"},{"t":"add","s":"Disallow: /search"},{"t":"ctx","s":"Disallow: /terms"}]}]}