{"domain":"npr.org","name":"NPR","category":"news","subcategory":"broadcaster","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent and Perplexity-User can reach most of it. (We used the Internet Archive's copy from 1 Oct 2026, because the site would not give our bot its robots.txt: no answer within 15 seconds.)","computed_at":"2026-10-20T03:43:15.253Z","last_checked":"2026-10-19T13:35:11.010Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://npr.org"},"robots":{"via":{"type":"archive","date":"2026-10-01","url":"https://web.archive.org/web/20261001001651/https://www.npr.org/robots.txt","blob":"53f9d15a3e089e2df2e24dc31beef07de952703ea14defd9f2e96afea4edc651"},"kind":"unreachable","http":null,"error":"TimeoutError","url":"https://www.npr.org/robots.txt","hops":[{"from":"https://npr.org/robots.txt","to":"https://www.npr.org/robots.txt","status":301}],"blob":"53f9d15a3e089e2df2e24dc31beef07de952703ea14defd9f2e96afea4edc651","text":"# robots.txt for www.npr.org\n# Changes are tracked in www-render\n\nUser-agent: *\nDisallow: /mpx/\nDisallow: /cgi-bin\nDisallow: /ramfiles/\nDisallow: /oauth2/\nDisallow: /account/\nDisallow: /proxy/\nDisallow: /*.smil\nDisallow: /*.asx\nDisallow: /*.ram\nDisallow: /*.wav\nDisallow: /*.rmm\nDisallow: /*.js\nDisallow: /*.au\nDisallow: /stations/force/force_localization.php?\nDisallow: /rundowns/segment.php?\nDisallow: /templates/search/*\nDisallow: /2013/03/21/174840895/\nDisallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html\nDisallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade\nDisallow: /transcripts/470280334*\nDisallow: /2015/07/04/419570939/chasing-memories-in-their-refugee-camp-40-years-after-they-fled-vietnam\nDisallow: /transcripts/419570939*\nDisallow: /sections/parallels/2016/08/15/480128005/for-french-teens-smoking-still-has-more-allure-than-stigma\nDisallow: /transcripts/480128005*\nDisallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent\nDisallow: /transcripts/830237502*\nDisallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time\nDisallow: /transcripts/406744975*\nDisallow: /sureroute\nDisallow: /*/partials*\nDisallow: /*?*\nDisallow: /proxy/*\nDisallow: /player/*\nDisallow: /get/*\nDisallow: /geolocation\n\n# Disallowing the OpenAI web crawler\nUser-agent: GPTBot\nDisallow: /\n\n# Disallowing OpenAI plugins\nUser-agent: ChatGPT-User\nDisallow: /\n\n# Disallowing Common Crawl\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\n\n# Disallowing Google Bard and Vertex AI web crawlers\nUser-agent: Google-Extended\nDisallow: /\n\n# Disallowing various bots\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: Claude-User\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: FacebookBot\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: omgilibot\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: PerplexityUser\nDisallow: /\n\n# Allow Google Search Console for sitemap crawling\nUser-agent: Google-InspectionTool\nAllow: /\nUser-agent: Google-Image\nAllow: /\nUser-agent: Google-Video\nAllow: /\nUser-agent: Googlebot\nAllow: /\n\n# Allowing the TTD contextual crawler for Sponsorship\nUser-agent: TTD-Content\nAllow: /\n\n# Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship\nUser-agent: Trade Desk ads.txt & sellers.json crawler\nAllow: /\n\n# Allowing the AdsBot-Google crawler for Sponsorship\nUser-agent: AdsBot-Google\nAllow: /\n\n# Allowing the AdsBot-Google-Mobile crawler for Sponsorship\nUser-agent: AdsBot-Google-Mobile\nAllow: /\n\n# Allowing the Google-Mediapartners crawler for Sponsorship\nUser-agent: Mediapartners-Google\nAllow: /\n\n# Allowing the Google-Display-Ads-Bot crawler for Sponsorship\nUser-agent: Google-Display-Ads-Bot\nAllow: /\n\n# Allowing the IAB tech lab crawler for Sponsorship\nUser-agent: IAB-Tech-Lab\nAllow: /\n\n# Allowing the IAS crawler for Sponsorship\nUser-agent: ias_crawler\nAllow: /\n\n# Allowing the IAS Wombles crawler for Sponsorship\nUser-agent: ias_wombles\nAllow: /\n\n# Allowing the Amazon Standards crawler for Sponsorship\nUser-agent: Amazon-Advertising-ad-standards-bot/1.0\nAllow: /\n# (fetches ads.txt, app-ads.txt, sellers.json)\nUser-agent: APS-ad-standards-bot/1.0\nAllow: /\n\n# New requirement as of 3/2/26 to be eligible for demand from Amazon ads.\nUser-agent: AmazonAdBot\nAllow: /\n\n# Allowing the Audigent crawler for Sponsorship\nUser-agent: AudigentAdBot\nAllow: /\n\n# Allowing the Concert crawler for Sponsorship\nUser-agent: Concert/1.0\nAllow: /\n\n# Allowing Centro/Basis\nUser-agent: Centro\nAllow: /\n\n# Allowing BidSwitch (AdX DSP)\nUser-agent: bidswitchbot/1.0\nAllow: /\n\n# Allowing Roku (AdX DSP)\nUser-agent: DataXu/1.0\nAllow: /\n\n# Allowing PubMatic\nUser-agent: PubMatic\nAllow: /\n\n# Allowing AdForm\nUser-agent: IAB ATQ team adform robot site.adform.com/general/url-fetcher\nAllow: /\n\n# Allowing StackAdapt\nUser-agent: StackAdapt\nAllow: /\n\n# Allowing Quantcast\nUser-agent: Quantcastbot/1.0\nAllow: /\nUser-agent: Quantcastbot/2.0\nAllow: /\n\n# Facebook sharing\nUser-agent: facebookexternalhit\nAllow: /\n\n# Ensures that we're using the correct sitemap. The fact that this is googlecrawl*.npr.org is OK because the crawler will only accept\n# URLs in this sitemap to match www*.npr.org\nSitemap: https://googlecrawl.npr.org/standard/sitemap_index.xml\nSitemap: https://googlecrawl.npr.org/news/sitemap_news.xml\nSitemap: https://googlecrawl.npr.org/video/sitemap_index.xml\nSitemap: https://www.npr.org/live-updates/sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":56,"text":"Disallowing Google Bard and Vertex AI web crawlers"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","GPTBot","ChatGPT-User","CCBot","OAI-SearchBot","Google-Extended","anthropic-ai","Applebot-Extended","Bytespider","ClaudeBot","Claude-SearchBot","Claude-User","Claude-Web","cohere-ai","Diffbot","FacebookBot","omgili","omgilibot","PerplexityBot","PerplexityUser","Google-InspectionTool","Google-Image","Google-Video","Googlebot","TTD-Content","Trade Desk ads.txt & sellers.json crawler","AdsBot-Google","AdsBot-Google-Mobile","Mediapartners-Google","Google-Display-Ads-Bot","IAB-Tech-Lab","ias_crawler","ias_wombles","Amazon-Advertising-ad-standards-bot/1.0","APS-ad-standards-bot/1.0","AmazonAdBot","AudigentAdBot","Concert/1.0","Centro","bidswitchbot/1.0","DataXu/1.0","PubMatic","IAB ATQ team adform robot site.adform.com/general/url-fetcher","StackAdapt","Quantcastbot/1.0","Quantcastbot/2.0","facebookexternalhit"],"lines":208},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":43},"any":{"allowed":false,"line":43},"search":{"allowed":false,"line":43},"cart":{"allowed":false,"line":43},"checkout":{"allowed":false,"line":43},"account":{"allowed":false,"line":43}},"disallows":1,"lines":[42,43],"groupRanges":[[42,43]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":71},"any":{"allowed":false,"line":71},"search":{"allowed":false,"line":71},"cart":{"allowed":false,"line":71},"checkout":{"allowed":false,"line":71},"account":{"allowed":false,"line":71}},"disallows":1,"lines":[70,71],"groupRanges":[[70,71]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":58},"any":{"allowed":false,"line":58},"search":{"allowed":false,"line":58},"cart":{"allowed":false,"line":58},"checkout":{"allowed":false,"line":58},"account":{"allowed":false,"line":58}},"disallows":1,"lines":[57,58],"groupRanges":[[57,58]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":65},"any":{"allowed":false,"line":65},"search":{"allowed":false,"line":65},"cart":{"allowed":false,"line":65},"checkout":{"allowed":false,"line":65},"account":{"allowed":false,"line":65}},"disallows":1,"lines":[64,65],"groupRanges":[[64,65]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":51},"any":{"allowed":false,"line":51},"search":{"allowed":false,"line":51},"cart":{"allowed":false,"line":51},"checkout":{"allowed":false,"line":51},"account":{"allowed":false,"line":51}},"disallows":1,"lines":[50,51],"groupRanges":[[50,51]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":68},"any":{"allowed":false,"line":68},"search":{"allowed":false,"line":68},"cart":{"allowed":false,"line":68},"checkout":{"allowed":false,"line":68},"account":{"allowed":false,"line":68}},"disallows":1,"lines":[67,68],"groupRanges":[[67,68]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":62},"any":{"allowed":false,"line":62},"search":{"allowed":false,"line":62},"cart":{"allowed":false,"line":62},"checkout":{"allowed":false,"line":62},"account":{"allowed":false,"line":62}},"disallows":1,"lines":[61,62],"groupRanges":[[61,62]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":54},"any":{"allowed":false,"line":54},"search":{"allowed":false,"line":54},"cart":{"allowed":false,"line":54},"checkout":{"allowed":false,"line":54},"account":{"allowed":false,"line":54}},"disallows":1,"lines":[53,54],"groupRanges":[[53,54]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":74},"any":{"allowed":false,"line":74},"search":{"allowed":false,"line":74},"cart":{"allowed":false,"line":74},"checkout":{"allowed":false,"line":74},"account":{"allowed":false,"line":74}},"disallows":1,"lines":[73,74],"groupRanges":[[73,74]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":98},"any":{"allowed":false,"line":98},"search":{"allowed":false,"line":98},"cart":{"allowed":false,"line":98},"checkout":{"allowed":false,"line":98},"account":{"allowed":false,"line":98}},"disallows":1,"lines":[97,98],"groupRanges":[[97,98]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":47},"any":{"allowed":false,"line":47},"search":{"allowed":false,"line":47},"cart":{"allowed":false,"line":47},"checkout":{"allowed":false,"line":47},"account":{"allowed":false,"line":47}},"disallows":1,"lines":[46,47],"groupRanges":[[46,47]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":77},"any":{"allowed":false,"line":77},"search":{"allowed":false,"line":77},"cart":{"allowed":false,"line":77},"checkout":{"allowed":false,"line":77},"account":{"allowed":false,"line":77}},"disallows":1,"lines":[76,77],"groupRanges":[[76,77]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":0,"lines":[110,111],"groupRanges":[[110,111]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":35,"lines":[4,39],"groupRanges":[[4,39]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":5,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":2,"full":5,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":44,"first":"2023-01-01","archive_points":44,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":4},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11}]},"events":[{"domain":"npr.org","name":"NPR","category":"news","date":"2026-10","precision":"month","source":"archive","provisional":true,"between":["2026-09-01","2026-10-01"],"summary":"Now blocked: Claude-SearchBot and Claude-User.","dir":"up","up":["Claude-SearchBot","Claude-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260901065413/https://www.npr.org/robots.txt","https://web.archive.org/web/20261001001651/https://www.npr.org/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the TTD contextual crawler for Sponsorship"},{"t":"del","s":"User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)"},{"t":"add","s":"# Allowing the TTD contextual crawler for Sponsorship"},{"t":"add","s":"User-agent: TTD-Content"},{"t":"ctx","s":"Allow: /"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the Google-Mediapartners crawler for Sponsorship"},{"t":"add","s":"# Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship"},{"t":"add","s":"User-agent: Trade Desk ads.txt & sellers.json crawler"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Allowing the AdsBot-Google crawler for Sponsorship"},{"t":"add","s":"User-agent: AdsBot-Google"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Allowing the AdsBot-Google-Mobile crawler for Sponsorship"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Allowing the Google-Mediapartners crawler for Sponsorship"},{"t":"ctx","s":"User-agent: Mediapartners-Google"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the Google-Display-Ads-Bot crawler for Sponsorship"},{"t":"add","s":"# Allowing the Google-Display-Ads-Bot crawler for Sponsorship"},{"t":"ctx","s":"User-agent: Google-Display-Ads-Bot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the IAB tech lab crawler for Sponsorship"},{"t":"add","s":"# Allowing the IAB tech lab crawler for Sponsorship"},{"t":"ctx","s":"User-agent: IAB-Tech-Lab"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the TTD ads fraud/IVT prevention crawler for Sponsorship"},{"t":"del","s":"User-agent: Trade Desk ads.txt & sellers.json crawler"},{"t":"add","s":"# Allowing the IAS crawler for Sponsorship"},{"t":"add","s":"User-agent: ias_crawler"},{"t":"ctx","s":"Allow: /"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the IAS crawler for Sponsorship"},{"t":"del","s":"User-agent: ias_crawler; http://integralads.com/site-indexing-policy/"},{"t":"add","s":"# Allowing the IAS Wombles crawler for Sponsorship"},{"t":"add","s":"User-agent: ias_wombles"},{"t":"ctx","s":"Allow: /"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the IAS Wombles crawler for Sponsorship"},{"t":"del","s":"User-agent: ias_wombles; http://integralads.com/site-indexing-policy/"},{"t":"add","s":"# Allowing the Amazon Standards crawler for Sponsorship"},{"t":"add","s":"User-agent: Amazon-Advertising-ad-standards-bot/1.0"},{"t":"ctx","s":"Allow: /"},{"t":"gap","s":"… 56 more lines"}]},{"domain":"npr.org","name":"NPR","category":"news","date":"2025-07","precision":"month","source":"archive","provisional":false,"between":["2025-05-01","2025-07-01"],"summary":"Now blocked: OAI-SearchBot.","dir":"up","up":["OAI-SearchBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250501011518/https://www.npr.org/robots.txt","https://web.archive.org/web/20250701012213/https://www.npr.org/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /*/partials*"},{"t":"del","s":"Disallow: /*?live=*"},{"t":"del","s":"Disallow: /*?cacheKill=*"},{"t":"del","s":"Disallow: /*?skipCache=*"},{"t":"del","s":"Disallow: /*?forceCds=*"},{"t":"del","s":"Disallow: /*?forceXml=*"},{"t":"del","s":""},{"t":"del","s":"# Disallowing non-canonical URLs that contain query strings"},{"t":"del","s":"User-agent: *"},{"t":"ctx","s":"Disallow: /*?*"},{"t":"del","s":""},{"t":"del","s":"# Disallowing non-audience-facing URLs"},{"t":"del","s":"User-agent: *"},{"t":"ctx","s":"Disallow: /proxy/*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /get/*"},{"t":"add","s":"Disallow: /geolocation"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Disallowing Google Bard and Vertex AI web crawlers"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: PerplexityUser"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Allow Google Search Console for sitemap crawling"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Explicitly allowing the TTD contextual crawler"},{"t":"add","s":"# Explicitly allowing the TTD contextual crawler for Sponsorship"},{"t":"ctx","s":"User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Explicitly allowing the Google-Mediapartners crawler for Sponsorship"},{"t":"add","s":"User-agent: Mediapartners-Google"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the Google-Display-Ads-Bot crawler for Sponsorship"},{"t":"add","s":"User-agent: Google-Display-Ads-Bot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the IAB tech lab crawler for Sponsorship"},{"t":"add","s":"User-agent: IAB-Tech-Lab"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the TTD ads fraud/IVT prevention crawler for Sponsorship"},{"t":"add","s":"User-agent: Trade Desk ads.txt & sellers.json crawler"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the IAS crawler for Sponsorship"},{"t":"add","s":"User-agent: ias_crawler; http://integralads.com/site-indexing-policy/"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the IAS Wombles crawler for Sponsorship"},{"t":"add","s":"User-agent: ias_wombles; http://integralads.com/site-indexing-policy/"},{"t":"add","s":"Allow: /"},{"t":"gap","s":"… 19 more lines"}]},{"domain":"npr.org","name":"NPR","category":"news","date":"2024-07","precision":"month","source":"archive","provisional":false,"between":["2024-06-01","2024-07-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and PerplexityBot.","dir":"up","up":["ClaudeBot","Applebot-Extended","CCBot","Bytespider","anthropic-ai","PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240601000427/https://www.npr.org/robots.txt","https://web.archive.org/web/20240701014200/https://www.npr.org/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# Disallowing non-canonical URLs that contain query strings"},{"t":"add","s":"User-agent: *"},{"t":"add","s":"Disallow: /*?*"},{"t":"add","s":""},{"t":"add","s":"# Disallowing non-audience-facing URLs"},{"t":"add","s":"User-agent: *"},{"t":"add","s":"Disallow: /proxy/*"},{"t":"add","s":"Disallow: /player/*"},{"t":"add","s":"Disallow: /get/*"},{"t":"add","s":""},{"t":"ctx","s":"# Disallowing the OpenAI web crawler"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Disallowing Common Crawl"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Disallowing Google Bard and Vertex AI web crawlers"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Disallowing various bots"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Allow Google Search Console for sitemap crawling"},{"t":"add","s":"User-agent: Google-InspectionTool"},{"t":"add","s":"Allow: /"},{"t":"add","s":"User-agent: Google-Image"},{"t":"add","s":"Allow: /"},{"t":"gap","s":"… 18 more lines"}]},{"domain":"npr.org","name":"NPR","category":"news","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-02-01","2024-03-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240201003630/https://www.npr.org/robots.txt","https://web.archive.org/web/20240301001905/https://www.npr.org/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /transcripts/480128005*"},{"t":"add","s":"Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent"},{"t":"add","s":"Disallow: /transcripts/830237502*"},{"t":"add","s":"Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time"},{"t":"add","s":"Disallow: /transcripts/406744975*"},{"t":"ctx","s":"Disallow: /sureroute"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Disallowing Google Bard and Vertex AI web crawlers"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Explicitly allowing the TTD contextual crawler"},{"t":"add","s":"User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept"}]},{"domain":"npr.org","name":"NPR","category":"news","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-09-01","2023-10-01"],"summary":"Now blocked: GPTBot and ChatGPT-User.","dir":"up","up":["GPTBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230901000300/https://www.npr.org/robots.txt","https://web.archive.org/web/20231001014508/https://www.npr.org/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /transcripts/480128005*"},{"t":"del","s":"Disallow: /tags*"},{"t":"ctx","s":"Disallow: /sureroute"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Disallowing the OpenAI web crawler"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Disallowing OpenAI plugins"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept"}]}]}