All sites / News and publishing
NPR
npr.orgBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent and Perplexity-User can reach most of it. (We used the Internet Archive's copy from 1 Oct 2026, because the site would not give our bot its robots.txt: no answer within 15 seconds.)
Last checked 19 Oct 2026. Rules file from the Internet Archive copy of 1 Oct 2026. Everything we checked · History
Bots that train AI Some blocked
6 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
3 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
2 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 42 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 70 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 57 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 64 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 50 | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 67 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 61 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 53 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 73 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 97 | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 46 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 76 | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Claude-Web | Blocked | line 79 | Not in our verified list |
| cohere-ai | Blocked | line 82 | Not in our verified list |
| Diffbot | Blocked | line 85 | Documented by its operator |
| FacebookBot | Blocked | line 88 | Not in our verified list |
| omgili | Blocked | line 91 | Not in our verified list |
| omgilibot | Blocked | line 94 | Not in our verified list |
| PerplexityUser | Blocked | line 100 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# robots.txt for www.npr.org# Changes are tracked in www-render User-agent: *Disallow: /mpx/Disallow: /cgi-binDisallow: /ramfiles/Disallow: /oauth2/Disallow: /account/Disallow: /proxy/Disallow: /*.smilDisallow: /*.asxDisallow: /*.ramDisallow: /*.wavDisallow: /*.rmmDisallow: /*.jsDisallow: /*.auDisallow: /stations/force/force_localization.php?Disallow: /rundowns/segment.php?Disallow: /templates/search/*Disallow: /2013/03/21/174840895/Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.htmlDisallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgradeDisallow: /transcripts/470280334*Disallow: /2015/07/04/419570939/chasing-memories-in-their-refugee-camp-40-years-after-they-fled-vietnamDisallow: /transcripts/419570939*Disallow: /sections/parallels/2016/08/15/480128005/for-french-teens-smoking-still-has-more-allure-than-stigmaDisallow: /transcripts/480128005*Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rentDisallow: /transcripts/830237502*Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-timeDisallow: /transcripts/406744975*Disallow: /surerouteDisallow: /*/partials*Disallow: /*?*Disallow: /proxy/*Disallow: /player/*Disallow: /get/*Disallow: /geolocation … 40 lines not about AI bots# Disallowing the OpenAI web crawlerUser-agent: GPTBotDisallow: / # Disallowing OpenAI pluginsUser-agent: ChatGPT-UserDisallow: / # Disallowing Common CrawlUser-agent: CCBotDisallow: / User-agent: OAI-SearchBotDisallow: / # Disallowing Google Bard and Vertex AI web crawlersUser-agent: Google-ExtendedDisallow: / # Disallowing various botsUser-agent: anthropic-aiDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: BytespiderDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Claude-SearchBotDisallow: / User-agent: Claude-UserDisallow: / User-agent: Claude-WebDisallow: / User-agent: cohere-aiDisallow: / User-agent: DiffbotDisallow: / User-agent: FacebookBotDisallow: / User-agent: omgiliDisallow: / User-agent: omgilibotDisallow: / User-agent: PerplexityBotDisallow: / User-agent: PerplexityUserDisallow: / # Allow Google Search Console for sitemap crawlingUser-agent: Google-InspectionToolAllow: /User-agent: Google-ImageAllow: /User-agent: Google-VideoAllow: /User-agent: GooglebotAllow: / # Allowing the TTD contextual crawler for SponsorshipUser-agent: TTD-ContentAllow: / # Allowing the TTD ads fraud/IVT prevention crawler for SponsorshipUser-agent: Trade Desk ads.txt & sellers.json crawlerAllow: / # Allowing the AdsBot-Google crawler for SponsorshipUser-agent: AdsBot-GoogleAllow: / # Allowing the AdsBot-Google-Mobile crawler for SponsorshipUser-agent: AdsBot-Google-MobileAllow: / # Allowing the Google-Mediapartners crawler for SponsorshipUser-agent: Mediapartners-GoogleAllow: / # Allowing the Google-Display-Ads-Bot crawler for SponsorshipUser-agent: Google-Display-Ads-BotAllow: / # Allowing the IAB tech lab crawler for SponsorshipUser-agent: IAB-Tech-LabAllow: / # Allowing the IAS crawler for SponsorshipUser-agent: ias_crawlerAllow: / # Allowing the IAS Wombles crawler for SponsorshipUser-agent: ias_womblesAllow: / # Allowing the Amazon Standards crawler for SponsorshipUser-agent: Amazon-Advertising-ad-standards-bot/1.0Allow: /# (fetches ads.txt, app-ads.txt, sellers.json)User-agent: APS-ad-standards-bot/1.0Allow: / # New requirement as of 3/2/26 to be eligible for demand from Amazon ads.User-agent: AmazonAdBotAllow: / # Allowing the Audigent crawler for SponsorshipUser-agent: AudigentAdBotAllow: / # Allowing the Concert crawler for SponsorshipUser-agent: Concert/1.0Allow: / # Allowing Centro/BasisUser-agent: CentroAllow: / # Allowing BidSwitch (AdX DSP)User-agent: bidswitchbot/1.0Allow: / # Allowing Roku (AdX DSP)User-agent: DataXu/1.0Allow: / # Allowing PubMaticUser-agent: PubMaticAllow: / # Allowing AdFormUser-agent: IAB ATQ team adform robot site.adform.com/general/url-fetcherAllow: / # Allowing StackAdaptUser-agent: StackAdaptAllow: / # Allowing QuantcastUser-agent: Quantcastbot/1.0Allow: /User-agent: Quantcastbot/2.0Allow: / # Facebook sharingUser-agent: facebookexternalhitAllow: / # Ensures that we're using the correct sitemap. The fact that this is googlecrawl*.npr.org is OK because the crawler will only accept# URLs in this sitemap to match www*.npr.orgSitemap: https://googlecrawl.npr.org/standard/sitemap_index.xmlSitemap: https://googlecrawl.npr.org/news/sitemap_news.xmlSitemap: https://googlecrawl.npr.org/video/sitemap_index.xmlSitemap: https://www.npr.org/live-updates/sitemap.xml … 106 lines not about AI bots
Notes about AI written in the file: "Disallowing Google Bard and Vertex AI web crawlers"
Everything else we checked
| Front door | We did not visit: the rules file did not load, and the standard says to treat that as "keep out". |
|---|---|
| llms.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| ai.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| TDMRep | Disallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
44 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Oct 2026More blocking Now blocked: Claude-SearchBot and Claude-User.Found in the Internet Archive's copies: changed between 1 Sep 2026 and 1 Oct 2026. Not confirmed yet: seen once so far. Evidence: before · after
Show exactly what changed
+ User-agent: Claude-SearchBot+ Disallow: /++ User-agent: Claude-User+ Disallow: /+User-agent: Claude-Web…- # Explicitly allowing the TTD contextual crawler for Sponsorship- User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)+ # Allowing the TTD contextual crawler for Sponsorship+ User-agent: TTD-ContentAllow: /- # Explicitly allowing the Google-Mediapartners crawler for Sponsorship+ # Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship+ User-agent: Trade Desk ads.txt & sellers.json crawler+ Allow: /++ # Allowing the AdsBot-Google crawler for Sponsorship+ User-agent: AdsBot-Google+ Allow: /++ # Allowing the AdsBot-Google-Mobile crawler for Sponsorship+ User-agent: AdsBot-Google-Mobile+ Allow: /++ # Allowing the Google-Mediapartners crawler for SponsorshipUser-agent: Mediapartners-Google…- # Explicitly allowing the Google-Display-Ads-Bot crawler for Sponsorship+ # Allowing the Google-Display-Ads-Bot crawler for SponsorshipUser-agent: Google-Display-Ads-Bot…- # Explicitly allowing the IAB tech lab crawler for Sponsorship+ # Allowing the IAB tech lab crawler for SponsorshipUser-agent: IAB-Tech-Lab…- # Explicitly allowing the TTD ads fraud/IVT prevention crawler for Sponsorship- User-agent: Trade Desk ads.txt & sellers.json crawler+ # Allowing the IAS crawler for Sponsorship+ User-agent: ias_crawlerAllow: /- # Explicitly allowing the IAS crawler for Sponsorship- User-agent: ias_crawler; http://integralads.com/site-indexing-policy/+ # Allowing the IAS Wombles crawler for Sponsorship+ User-agent: ias_womblesAllow: /- # Explicitly allowing the IAS Wombles crawler for Sponsorship- User-agent: ias_wombles; http://integralads.com/site-indexing-policy/+ # Allowing the Amazon Standards crawler for Sponsorship+ User-agent: Amazon-Advertising-ad-standards-bot/1.0Allow: /… 56 more lines - Jul 2025More blocking Now blocked: OAI-SearchBot.Found in the Internet Archive's copies: changed between 1 May 2025 and 1 Jul 2025. Evidence: before · after
Show exactly what changed
Disallow: /*/partials*- Disallow: /*?live=*- Disallow: /*?cacheKill=*- Disallow: /*?skipCache=*- Disallow: /*?forceCds=*- Disallow: /*?forceXml=*-- # Disallowing non-canonical URLs that contain query strings- User-agent: *Disallow: /*?*-- # Disallowing non-audience-facing URLs- User-agent: *Disallow: /proxy/*…Disallow: /get/*+ Disallow: /geolocation…+ User-agent: OAI-SearchBot+ Disallow: /+# Disallowing Google Bard and Vertex AI web crawlers…+ User-agent: PerplexityUser+ Disallow: /+# Allow Google Search Console for sitemap crawling…- # Explicitly allowing the TTD contextual crawler+ # Explicitly allowing the TTD contextual crawler for SponsorshipUser-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)…+ # Explicitly allowing the Google-Mediapartners crawler for Sponsorship+ User-agent: Mediapartners-Google+ Allow: /++ # Explicitly allowing the Google-Display-Ads-Bot crawler for Sponsorship+ User-agent: Google-Display-Ads-Bot+ Allow: /++ # Explicitly allowing the IAB tech lab crawler for Sponsorship+ User-agent: IAB-Tech-Lab+ Allow: /++ # Explicitly allowing the TTD ads fraud/IVT prevention crawler for Sponsorship+ User-agent: Trade Desk ads.txt & sellers.json crawler+ Allow: /++ # Explicitly allowing the IAS crawler for Sponsorship+ User-agent: ias_crawler; http://integralads.com/site-indexing-policy/+ Allow: /++ # Explicitly allowing the IAS Wombles crawler for Sponsorship+ User-agent: ias_wombles; http://integralads.com/site-indexing-policy/+ Allow: /… 19 more lines - Jul 2024More blocking Now blocked: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and PerplexityBot.Found in the Internet Archive's copies: changed between 1 Jun 2024 and 1 Jul 2024. Evidence: before · after
Show exactly what changed
+ # Disallowing non-canonical URLs that contain query strings+ User-agent: *+ Disallow: /*?*++ # Disallowing non-audience-facing URLs+ User-agent: *+ Disallow: /proxy/*+ Disallow: /player/*+ Disallow: /get/*+# Disallowing the OpenAI web crawler…+ # Disallowing Common Crawl+ User-agent: CCBot+ Disallow: /+# Disallowing Google Bard and Vertex AI web crawlers…+ # Disallowing various bots+ User-agent: anthropic-ai+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: omgilibot+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ # Allow Google Search Console for sitemap crawling+ User-agent: Google-InspectionTool+ Allow: /+ User-agent: Google-Image+ Allow: /… 18 more lines - Mar 2024More blocking Now blocked: Google-Extended.Found in the Internet Archive's copies: changed between 1 Feb 2024 and 1 Mar 2024. Evidence: before · after
Show exactly what changed
Disallow: /transcripts/480128005*+ Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent+ Disallow: /transcripts/830237502*+ Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time+ Disallow: /transcripts/406744975*Disallow: /sureroute…+ # Disallowing Google Bard and Vertex AI web crawlers+ User-agent: Google-Extended+ Disallow: /++ # Explicitly allowing the TTD contextual crawler+ User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)+ Allow: /+# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept - Oct 2023More blocking Now blocked: GPTBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Sep 2023 and 1 Oct 2023. Evidence: before · after
Show exactly what changed
Disallow: /transcripts/480128005*- Disallow: /tags*Disallow: /sureroute…+ # Disallowing the OpenAI web crawler+ User-agent: GPTBot+ Disallow: /++ # Disallowing OpenAI plugins+ User-agent: ChatGPT-User+ Disallow: /+# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.