All sites / News and publishing
CNN
cnn.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 17 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Applebot-Extended, CCBot and OAI-SearchBot. Google-Extended and Meta-ExternalAgent can reach most of it.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
6 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
6 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
5 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 14 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 14 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 14 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 14 | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 14 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Named, line 14 | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 14 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 14 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 14 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 14 | Suggests yes |
| AmazonbotAmazon | Blocked | Named, line 14 | Says yes |
| Amzn-SearchBotAmazon | Blocked | Named, line 14 | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Named, line 14 | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 14 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 14 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 14 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Blocked | Named, line 14 | Says not always |
| MistralAI-UserMistral AI | Blocked | Named, line 14 | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Ai2Bot-Dolma | Blocked | line 15 | Not in our verified list |
| Claude-Web | Blocked | line 32 | Not in our verified list |
| cohere-ai | Blocked | line 33 | Not in our verified list |
| cohere-training-data-crawler | Blocked | line 34 | Not in our verified list |
| Diffbot | Blocked | line 37 | Documented by its operator |
| FacebookBot | Blocked | line 43 | Not in our verified list |
| GoogleOther | Blocked | line 46 | Documented by its operator |
| GoogleOther-Image | Blocked | line 47 | Documented by its operator |
| GoogleOther-Video | Blocked | line 48 | Documented by its operator |
| ImagesiftBot | Blocked | line 54 | Not in our verified list |
| omgili | Blocked | line 62 | Not in our verified list |
| omgilibot | Blocked | line 63 | Not in our verified list |
| PetalBot | Blocked | line 69 | Not in our verified list |
| Timpibot | Blocked | line 84 | Not in our verified list |
| Webzio-Extended | Blocked | line 88 | Not in our verified list |
| YouBot | Blocked | line 91 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
Sitemap: https://www.cnn.com/sitemap/news.xmlSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xmlSitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xmlSitemap: https://www.cnn.com/sitemap/section/politics.xmlSitemap: https://www.cnn.com/sitemap/article/opinions.xmlSitemap: https://www.cnn.com/sitemap/article.xmlSitemap: https://www.cnn.com/sitemap/section.xmlSitemap: https://www.cnn.com/sitemap/video.xmlSitemap: https://www.cnn.com/sitemap/gallery.xmlSitemap: https://www.cnn.com/sitemap/markets/stocks.xmlSitemap: https://www.cnn.com/sitemap/live-story.xmlSitemap: https://www.cnn.com/sitemap/election-center/politics.xml… 12 lines not about AI botsSitemap: https://www.cnn.com/sitemap/tve.xmlUser-agent: AI2BotUser-agent: Ai2Bot-DolmaUser-agent: AliyunSecBotUser-agent: AmazonbotUser-agent: amzn-searchbotUser-agent: amzn-userUser-agent: anthropic-aiUser-agent: Applebot-ExtendedUser-agent: Archive.org_botUser-agent: AwarioRssBotUser-agent: AwarioSmartBotUser-agent: Brightbot 1.0User-agent: BytespiderUser-agent: CCBotUser-agent: ChatGPT-UserUser-agent: ClaudeBotUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: Claude-WebUser-agent: cohere-aiUser-agent: cohere-training-data-crawlerUser-agent: CrawlspaceUser-agent: DataForSeoBotUser-agent: DiffbotUser-agent: DuckAssistBotUser-agent: EchoboxBotUser-agent: ExabotUser-agent: ExaSearchBotUser-agent: ExaUserUser-agent: FacebookBotUser-agent: FriendlyCrawlerUser-agent: GPTBotUser-agent: GoogleOtherUser-agent: GoogleOther-ImageUser-agent: GoogleOther-VideoUser-Agent: IAB-Tech-LabUser-agent: iaskspider/2.0User-agent: ICC-CrawlerUser-agent: img2datasetUser-agent: ISSCyberRiskCrawlerUser-agent: ImagesiftBotUser-agent: Kangaroo BotUser-agent: magpie-crawlerUser-agent: MistralAI-userUser-agent: MyCentralAIScraperBotUser-agent: NewsNowUser-agent: news-pleaseUser-agent: OAI-SearchBotUser-agent: omgiliUser-agent: omgilibotUser-agent: PanguBotUser-agent: PanscientUser-agent: PiplBotUser-agent: PerplexityBotUser-agent: Perplexity-UserUser-agent: PetalBotUser-agent: Poseidon Research CrawlerUser-agent: QuillBotUser-agent: quillbot.comUser-agent: Quora-BotUser-agent: SBIntuitionsBotUser-agent: ScrapyUser-agent: SeekrBotUser-agent: SemrushBot-OCOBUser-agent: SemrushBot-SWAUser-agent: SeznamHomepageCrawlerUser-agent: ShapBotUser-agent: Shap-UserUser-agent: Sidetrade indexer botUser-agent: TaraGroup Intelligence BotUser-agent: TimpibotUser-agent: TurnitinBotUser-agent: VelenPublicWebCrawlerUser-agent: ViennaTinyBotUser-agent: Webzio-ExtendedUser-agent: YandexAdditionalUser-agent: YandexAdditionalBotUser-agent: YouBotDisallow: /User-agent: *Allow: /partners/ipad/live-video.jsonDisallow: /*.jsx$Disallow: *.jsx$Disallow: /*.jsx/Disallow: *.jsx?Disallow: /ads/Disallow: /aol/Disallow: /api/Disallow: /beta/Disallow: /browsers/Disallow: /cl/Disallow: /cnews/Disallow: /cnn_adspacesDisallow: /cnnbeta/Disallow: /cnnintl_adspacesDisallow: /developmentDisallow: /editionssiDisallow: /help/cnnx.htmlDisallow: /NewsPassDisallow: /NOKIADisallow: /partners/Disallow: /pipeline/Disallow: /pointroll/Disallow: /POLLSERVER/Disallow: /pr/Disallow: /PV/Disallow: /Quickcast/Disallow: /quickcast/Disallow: /QUICKNEWS/Disallow: /searchDisallow: /subscriptions/video-docsDisallow: /termsDisallow: /test/Disallow: /virtual/Disallow: /WEB-INF/Disallow: /web.projects/Disallow: /webview/User-agent: Googlebot-NewsDisallow: /sponsor … 40 lines not about AI bots
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.cnn.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Anti-bot software | Served through Fastly.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: WebPage, NewsMediaOrganization, WebSite, SearchAction. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | https://www.cnn.com/terms (Disallowed for AgentAccessIndexBot by robots.txt) |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags. |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Oct 2026Less blocking Now allowed: Google-Extended.Changed between the Internet Archive's copy from 1 Oct 2026 and our own check on 19 Oct 2026. Not confirmed yet: seen once so far. Evidence: before · after
Show exactly what changed
User-agent: Exabot+ User-agent: ExaSearchBot+ User-agent: ExaUserUser-agent: FacebookBot…User-agent: GPTBot- User-agent: Google-ExtendedUser-agent: GoogleOther…User-agent: SeznamHomepageCrawler+ User-agent: ShapBot+ User-agent: Shap-UserUser-agent: Sidetrade indexer bot - Mar to Jun 2026More blocking Now blocked: Claude-SearchBot, Claude-User, Perplexity-User and MistralAI-User.Found in the Internet Archive's copies: changed between 1 Mar 2026 and 2 Jun 2026. Evidence: before · after
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/news.xml- Sitemap: https://www.cnn.com/sitemaps/sitemap-section.xml- Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xmlSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml…User-agent: Ai2Bot-Dolma+ User-agent: AliyunSecBotUser-agent: Amazonbot…User-agent: Applebot-Extended+ User-agent: Archive.org_botUser-agent: AwarioRssBot…User-agent: ClaudeBot+ User-agent: Claude-SearchBot+ User-agent: Claude-UserUser-agent: Claude-Web…User-agent: EchoboxBot+ User-agent: ExabotUser-agent: FacebookBot…User-agent: GoogleOther-Video+ User-Agent: IAB-Tech-LabUser-agent: iaskspider/2.0…User-agent: magpie-crawler+ User-agent: MistralAI-user+ User-agent: MyCentralAIScraperBotUser-agent: NewsNow…User-agent: PerplexityBot+ User-agent: Perplexity-UserUser-agent: PetalBot+ User-agent: Poseidon Research CrawlerUser-agent: QuillBotUser-agent: quillbot.com+ User-agent: Quora-BotUser-agent: SBIntuitionsBotUser-agent: Scrapy+ User-agent: SeekrBotUser-agent: SemrushBot-OCOBUser-agent: SemrushBot-SWA+ User-agent: SeznamHomepageCrawlerUser-agent: Sidetrade indexer bot+ User-agent: TaraGroup Intelligence BotUser-agent: Timpibot…User-agent: VelenPublicWebCrawler+ User-agent: ViennaTinyBotUser-agent: Webzio-Extended…Disallow: /pr/- Disallow: /privacyDisallow: /PV/ - Oct 2025 to Jan 2026Mixed Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Oct 2025 and 2 Jan 2026. Evidence: before · after
Show exactly what changed
- Sitemap: https://www.cnn.com/sitemaps/cnn/index.xml- Sitemap: https://www.cnn.com/sitemaps/cnn/news.xmlSitemap: https://www.cnn.com/sitemap/news.xml…Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xml- Sitemap: https://www.cnn.com/ampstories/sitemap.xml- Sitemap: https://edition.cnn.com/sitemaps/news.xmlSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xmlSitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml- Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml- Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xmlSitemap: https://www.cnn.com/sitemap/section/politics.xml…Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml+ Sitemap: https://www.cnn.com/sitemap/tve.xmlUser-agent: AI2Bot…User-agent: Amazonbot+ User-agent: amzn-searchbot+ User-agent: amzn-userUser-agent: anthropic-ai- User-agent: ApplebotUser-agent: Applebot-Extended…User-agent: DuckAssistBot+ User-agent: EchoboxBotUser-agent: FacebookBot…User-agent: magpie-crawler- User-agent: Meta-ExternalAgent- User-agent: Meta-ExternalFetcherUser-agent: NewsNow…User-agent: PanguBot- User-agent: PerplexityBot+ User-agent: PanscientUser-agent: PiplBot+ User-agent: PerplexityBotUser-agent: PetalBot+ User-agent: QuillBot+ User-agent: quillbot.com+ User-agent: SBIntuitionsBotUser-agent: Scrapy…User-agent: Webzio-Extended+ User-agent: YandexAdditional+ User-agent: YandexAdditionalBotUser-agent: YouBot…Disallow: /webview/- Disallow: /gamesUser-agent: Googlebot-News - Apr 2025More blocking Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot, OAI-SearchBot, Amazonbot, DuckAssistBot and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Mar 2025 and 1 Apr 2025. Evidence: before · after
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml+ Sitemap: https://www.cnn.com/sitemap/live-story.xml+ Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml+ User-agent: AI2Bot+ User-agent: Ai2Bot-Dolma+ User-agent: AmazonbotUser-agent: anthropic-ai+ User-agent: Applebot+ User-agent: Applebot-ExtendedUser-agent: AwarioRssBotUser-agent: AwarioSmartBot+ User-agent: Brightbot 1.0User-agent: Bytespider…User-agent: cohere-ai+ User-agent: cohere-training-data-crawler+ User-agent: CrawlspaceUser-agent: DataForSeoBotUser-agent: Diffbot+ User-agent: DuckAssistBotUser-agent: FacebookBot+ User-agent: FriendlyCrawlerUser-agent: GPTBotUser-agent: Google-Extended+ User-agent: GoogleOther+ User-agent: GoogleOther-Image+ User-agent: GoogleOther-Video+ User-agent: iaskspider/2.0+ User-agent: ICC-Crawler+ User-agent: img2dataset+ User-agent: ISSCyberRiskCrawler+ User-agent: ImagesiftBot+ User-agent: Kangaroo BotUser-agent: magpie-crawler+ User-agent: Meta-ExternalAgent+ User-agent: Meta-ExternalFetcherUser-agent: NewsNowUser-agent: news-please+ User-agent: OAI-SearchBotUser-agent: omgiliUser-agent: omgilibot+ User-agent: PanguBotUser-agent: PerplexityBot+ User-agent: PiplBot+ User-agent: PetalBotUser-agent: Scrapy+ User-agent: SemrushBot-OCOB+ User-agent: SemrushBot-SWA+ User-agent: Sidetrade indexer bot+ User-agent: TimpibotUser-agent: TurnitinBot+ User-agent: VelenPublicWebCrawler+ User-agent: Webzio-Extended+ User-agent: YouBotDisallow: /…Disallow: /search+ Disallow: /subscriptions/video-docsDisallow: /terms…… 5 more lines - May 2024More blocking Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Apr 2024 and 1 May 2024. Evidence: before · after
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml+ Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml+ Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xmlSitemap: https://www.cnn.com/sitemap/section/politics.xml…Sitemap: https://www.cnn.com/sitemap/gallery.xml+ Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml+ User-agent: anthropic-ai+ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ User-agent: Bytespider+ User-agent: CCBot+ User-agent: ChatGPT-User+ User-agent: ClaudeBot+ User-agent: Claude-Web+ User-agent: cohere-ai+ User-agent: DataForSeoBot+ User-agent: Diffbot+ User-agent: FacebookBotUser-agent: GPTBot- Disallow: /User-agent: Google-Extended+ User-agent: magpie-crawler+ User-agent: NewsNow+ User-agent: news-please+ User-agent: omgili+ User-agent: omgilibot+ User-agent: PerplexityBot+ User-agent: Scrapy+ User-agent: TurnitinBotDisallow: / - Nov 2023More blocking Now blocked: Google-Extended.Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/video.xml+ Sitemap: https://www.cnn.com/sitemap/gallery.xmlUser-agent: GPTBotDisallow: /+ User-agent: Google-Extended+ Disallow: /User-agent: * - Sep 2023More blocking Now blocked: GPTBot.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/section/politics.xml+ Sitemap: https://www.cnn.com/sitemap/article/opinions.xml+ Sitemap: https://www.cnn.com/sitemap/article.xml+ Sitemap: https://www.cnn.com/sitemap/section.xml+ Sitemap: https://www.cnn.com/sitemap/video.xml+ User-agent: GPTBot+ Disallow: /User-agent: *…Disallow: /aol/+ Disallow: /api/Disallow: /beta/…Disallow: /QUICKNEWS/- Disallow: /search/+ Disallow: /searchDisallow: /terms
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.