{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","subcategory":"national","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 8 of the 24 AI bots we track from the whole site, including ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent and PerplexityBot. GPTBot, Google-Extended, OAI-SearchBot, Claude-SearchBot and 3 more can reach most of it.","computed_at":"2026-10-20T03:43:15.130Z","last_checked":"2026-10-19T13:35:55.977Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.washingtonpost.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.washingtonpost.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.washingtonpost.com/robots.txt","hops":[{"from":"https://washingtonpost.com/robots.txt","to":"https://www.washingtonpost.com/robots.txt","status":301}],"blob":"918970c4ae59aa1f1f31577e14eec569caf79d69660bd06c381516e4d5c49426","text":"User-agent: *\nDisallow: /*_print.html\nDisallow: /*_email.html\nDisallow: /*_singlePage.html\nDisallow: /*_allComments.html\nDisallow: /*_jsn.json\nDisallow: /*_jsonpStatic.js\nDisallow: /*_nitf.xml\nDisallow: /*_newsml.html\nDisallow: /*_qa.html\nDisallow: /*_meta.xml\nDisallow: /*_jsnp.js\nDisallow: /*_json.json\nDisallow: /*_search.html\nDisallow: /*_jsonp.js\nDisallow: /*_jsnpStatic.js\nDisallow: /*_rss.xml\nDisallow: /*_mobile.mobile\nDisallow: /*_mobile.xml\nDisallow: /*_allCommentsClassicBlog.html\nDisallow: /*_seo.html\nDisallow: /*_nimbusJson.json\nDisallow: /*_nimbusJsonp.js\nDisallow: /*_nimbusJsonpStatic.js\nDisallow: /*_modal.html\nDisallow: /todays_paper/\nDisallow: /rw/WashingtonPost/Content/Epaper/\nDisallow: /ac2/\nDisallow: /blogs/slow-ride/\nDisallow: /local/blogsandcolumns/slow-ride-story-tanked\nDisallow: /local/blogsandcolumns/slow-ride-story-achenblog\nDisallow: /local/blogsandcolumns/slow-ride-stream-tanked\nDisallow: /local/blogsandcolumns/slow-ride-front\nDisallow: /utils/\nDisallow: /jobs/JS_JobSearchResult\nDisallow: /jobs/UpdateJobEmployerCounterServlet\nDisallow: /jobs/JS_Login\nDisallow: /jobs/EU_UpdateJobEmployerCounter\nDisallow: /blogs/nationals-journal-beta/\nDisallow: /blogs/test/\nDisallow: /posttv-beta/\nDisallow: /posttv/sponsored-video/\nDisallow: /posttv/c/trendex/\nDisallow: /posttv/c/video_search/\nDisallow: /posttv/posttv/trendex\nDisallow: /posttv/c/embed/\nDisallow: /rweb/\nDisallow: /wp-stat/vrroom/\nDisallow: /classic-apps/\nDisallow: /news/test/\nDisallow: /tablet/\nDisallow: /news/tablet/\nDisallow: /sf/test/\nDisallow: /news/test-liveblog/\nDisallow: /pb/\nAllow: /pb/resources/\nAllow: /pb/gr/\nAllow: /pb/resource/\nDisallow: /homepage-video-test\nDisallow: /testpage-forhomepage\nDisallow: /knowmore\nDisallow: /test/\nDisallow: /sslsingle\nDisallow: /amphtml/news/test/\nDisallow: /amphtml/blogs/test/\nDisallow: /amphtml/classic-apps/\nDisallow: /amphtml/utils/\nDisallow: /newsletter/\nDisallow: /wp-dyn/\nDisallow: /wp-srv/\nDisallow: /bandito/\nDisallow: /Fragment/SysConfig/\nDisallow: /recipes/search/\nDisallow: /talk/\nDisallow: /wp-stat/ad/\nDisallow: /*?*outputType=comment\nDisallow: /pwapi-proxy/pwproxy/*\nDisallow: /pwapiv2/\nDisallow: /*?*outputType=accessibility\nDisallow: /wp-adv/\nDisallow: /newssearch/\nDisallow: /wp-admin/\nDisallow: /gdpr-consent/\nDisallow: /*?*outputType=tracking\nDisallow: /tetro/\nDisallow: /comments/\nDisallow: /comments\nDisallow: /search\nDisallow: /s/*\nDisallow: /embed/\nDisallow: /native/\nDisallow: /subscribe/braintree/\nDisallow: /subscribe/enterpriseportal/\nDisallow: /subscribe/foryouapi/\nDisallow: /subscribe/lagoon/\nDisallow: /subscribe/offers/service/\nDisallow: /subscribe/onsiteapi/\nDisallow: /subscribe/paywall/\nDisallow: /subscribe/person/\nDisallow: /subscribe/preferenceapi/\nDisallow: /subscribe/subscriptionapi/\nDisallow: /subscribe/user/\nDisallow: /subscribe/signin/\nDisallow: /subscribe/signup/\nDisallow: /wpost/proxy\nDisallow: /ehf/\nDisallow: /ehf/*\nDisallow: /subscribe/logging/*\nDisallow: /blogs/*\nDisallow: /gog/*\nDisallow: /arcio/fact-checker/\nDisallow: /recipes/*?q=\nDisallow: /cookie-policy/\nDisallow: /discussions/2021/09/13/cookie-policy/\nDisallow: /discussions/2021/09/13/cookie-policy-eea/\nDisallow: /privacy-policy/\nDisallow: /partials/\n\nUser-agent: Twitterbot\nAllow: /posttv-beta/\nDisallow: /amphtml/*\n\nUser-agent: SemrushBot\nDisallow: /licensing-syndication\nDisallow: /licensing-syndication/*\n\nUser-agent: Amazonbot\nDisallow: /\nAllow: /wp-apps/imrs.php\n\n\n# Disallow Rules\n\nUser-agent: AhrefsBot\nDisallow: /\n\nUser-agent: anthropic-ai\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: Applebot-Extended\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: archive.org_bot\nDisallow: /\n\nUser-agent: ArchiveBot\nDisallow: /\n\nUser-agent: Arquivo-web-crawler\nDisallow: /\n\nUser-agent: AwarioRssBot\nDisallow: /\n\nUser-agent: AwarioSmartBot\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: CCBot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: Claude-Web\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: ClaudeBot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: cohere-training-data-crawler\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: DataForSeoBot\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: europarchive.org\nDisallow: /\n\nUser-agent: FacebookBot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: heritrix\nDisallow: /\n\nUser-agent: ia_archiver\nDisallow: /\n\nUser-agent: ia_archiver-web.archive.org\nDisallow: /\n\nUser-agent: ImagesiftBot\nDisallow: /\n\nUser-agent: LinkArchiver\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: meta-externalfetcher\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: Nicecrawler\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: omgilibot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: PerplexityBot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nUser-agent: Primalbot\nDisallow: /\n\nUser-agent: web-archive-net.com.bot\nDisallow: /\n\nUser-agent: YandexBot\nDisallow: /\nAllow: /creativegroup/\nAllow: /advertising/\n\nSitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz\nSitemap: https://www.washingtonpost.com/sitemaps/news-sitemap.xml.gz\nSitemap: https://www.washingtonpost.com/sitemaps/author-sitemap.xml.gz\nSitemap: https://www.washingtonpost.com/sitemaps/section-sitemap.xml.gz\nSitemap: https://www.washingtonpost.com/elections/results/sitemap.xml\nSitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/\nSitemap: https://www.washingtonpost.com/creativegroup/sitemaps/sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Twitterbot","SemrushBot","Amazonbot","AhrefsBot","anthropic-ai","Applebot-Extended","archive.org_bot","ArchiveBot","Arquivo-web-crawler","AwarioRssBot","AwarioSmartBot","Bytespider","CCBot","Claude-Web","ClaudeBot","cohere-training-data-crawler","DataForSeoBot","Diffbot","europarchive.org","FacebookBot","heritrix","ia_archiver","ia_archiver-web.archive.org","ImagesiftBot","LinkArchiver","meta-externalagent","meta-externalfetcher","Nicecrawler","omgili","omgilibot","PerplexityBot","Primalbot","web-archive-net.com.bot","YandexBot"],"lines":264},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":178},"any":{"allowed":false,"line":178},"search":{"allowed":false,"line":178},"cart":{"allowed":false,"line":178},"checkout":{"allowed":false,"line":178},"account":{"allowed":false,"line":178}},"disallows":1,"lines":[177,180],"groupRanges":[[177,180]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":143},"any":{"allowed":false,"line":143},"search":{"allowed":false,"line":143},"cart":{"allowed":false,"line":143},"checkout":{"allowed":false,"line":143},"account":{"allowed":false,"line":143}},"disallows":1,"lines":[142,145],"groupRanges":[[142,145]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":168},"any":{"allowed":false,"line":168},"search":{"allowed":false,"line":168},"cart":{"allowed":false,"line":168},"checkout":{"allowed":false,"line":168},"account":{"allowed":false,"line":168}},"disallows":1,"lines":[167,170],"groupRanges":[[167,170]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":219},"any":{"allowed":false,"line":219},"search":{"allowed":false,"line":219},"cart":{"allowed":false,"line":219},"checkout":{"allowed":false,"line":219},"account":{"allowed":false,"line":219}},"disallows":1,"lines":[218,221],"groupRanges":[[218,221]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":163},"any":{"allowed":false,"line":163},"search":{"allowed":false,"line":163},"cart":{"allowed":false,"line":163},"checkout":{"allowed":false,"line":163},"account":{"allowed":false,"line":163}},"disallows":1,"lines":[162,165],"groupRanges":[[162,165]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":138},"any":{"allowed":false,"line":138},"search":{"allowed":false,"line":138},"cart":{"allowed":false,"line":138},"checkout":{"allowed":false,"line":138},"account":{"allowed":false,"line":138}},"disallows":1,"lines":[137,140],"groupRanges":[[137,140]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":242},"any":{"allowed":false,"line":242},"search":{"allowed":false,"line":242},"cart":{"allowed":false,"line":242},"checkout":{"allowed":false,"line":242},"account":{"allowed":false,"line":242}},"disallows":1,"lines":[241,244],"groupRanges":[[241,244]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":128},"any":{"allowed":false,"line":128},"search":{"allowed":false,"line":128},"cart":{"allowed":false,"line":128},"checkout":{"allowed":false,"line":128},"account":{"allowed":false,"line":128}},"disallows":1,"lines":[127,129],"groupRanges":[[127,129]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":224},"any":{"allowed":false,"line":224},"search":{"allowed":false,"line":224},"cart":{"allowed":false,"line":224},"checkout":{"allowed":false,"line":224},"account":{"allowed":false,"line":224}},"disallows":1,"lines":[223,226],"groupRanges":[[223,226]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":88},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":113,"lines":[1,117],"groupRanges":[[1,117]]}],"lanes":{"training":{"level":"mixed","none":5,"full":4,"restricted":0,"n":9},"search":{"level":"mixed","none":2,"full":6,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":1,"full":6,"restricted":0,"n":7}},"door":{"door":"error","detail":"TimeoutError"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"error","error":"TimeoutError","url":"https://www.washingtonpost.com/llms.txt"},"ai_txt":{"state":"error","error":"TimeoutError","url":"https://www.washingtonpost.com/ai.txt"},"tdmrep":{"state":"refused","http":403,"url":"https://www.washingtonpost.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":46,"first":"2023-01-01","archive_points":45,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8}]},"events":[{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-07-01","2025-08-03"],"summary":"Now allowed: Google-Extended.","dir":"down","up":[],"down":["Google-Extended"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250701004129/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20250803231953/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: heritrix"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2025-04","precision":"month","source":"archive","provisional":false,"between":["2025-03-01","2025-04-01"],"summary":"Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.","dir":"down","up":[],"down":["GPTBot","OAI-SearchBot","ChatGPT-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250301021544/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20250401005741/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"Allow: /wp-apps/imrs.php"},{"t":"ctx","s":""},{"t":"add","s":""},{"t":"ctx","s":"# Disallow Rules"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Amazonbot"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: cohere-training-data-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: DataForSeoBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: heritrix"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: magpie-crawler"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: meta-externalagent"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: OAI-SearchBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: omgili"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2025-02","precision":"month","source":"archive","provisional":false,"between":["2025-01-06","2025-02-14"],"summary":"Now blocked: OAI-SearchBot.","dir":"up","up":["OAI-SearchBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250106222839/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20250214064611/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /arcio/fact-checker/"},{"t":"add","s":"Disallow: /recipes/*?q="},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: AhrefsBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: SemrushBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":""},{"t":"add","s":"# Disallow Rules"},{"t":"add","s":""},{"t":"add","s":"User-agent: AhrefsBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: CCBot"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Amazonbot"},{"t":"add","s":"User-agent: archive.org_bot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: YandexBot"},{"t":"add","s":"User-agent: ArchiveBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: anthropic-ai"},{"t":"add","s":"User-agent: Arquivo-web-crawler"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: AwarioRssBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: AwarioSmartBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ClaudeBot"},{"t":"add","s":"User-agent: CCBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 75 more lines"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2024-12","precision":"month","source":"archive","provisional":false,"between":["2024-11-01","2024-12-01"],"summary":"Now blocked: Meta-ExternalAgent and Meta-ExternalFetcher.","dir":"up","up":["Meta-ExternalAgent","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241101005614/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20241201001451/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /gog/*"},{"t":"add","s":"Disallow: /arcio/fact-checker/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: meta-externalfetcher"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/elections/results/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/section-fronts/list/"},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2024-09","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-09-01"],"summary":"Now blocked: Applebot-Extended.","dir":"up","up":["Applebot-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801002032/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20240901092356/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /ac2/"},{"t":"del","s":"Disallow: /archive/lifestyle/wellness/1999/01/19/an-illness-no-one-understands-tourettes-syndrome/852df0cc-2164-4c83-be8d-0c770d028547/"},{"t":"ctx","s":"Disallow: /blogs/slow-ride/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /wp-dyn/"},{"t":"add","s":"Disallow: /wp-srv/"},{"t":"ctx","s":"Disallow: /bandito/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /subscribe/user/"},{"t":"add","s":"Disallow: /subscribe/signin/"},{"t":"add","s":"Disallow: /subscribe/signup/"},{"t":"ctx","s":"Disallow: /wpost/proxy"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /ehf/*"},{"t":"add","s":"Disallow: /subscribe/logging/*"},{"t":"add","s":"Disallow: /blogs/*"},{"t":"add","s":"Disallow: /gog/*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: YandexBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/"},{"t":"del","s":"Sitemap: https://www.washingtonpost.com/arcio/news-sitemap/"},{"t":"del","s":"Sitemap: https://www.washingtonpost.com/arcio/news-sitemap-elections/"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/sitemaps/news-sitemap.xml.gz"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/sitemaps/author-sitemap.xml.gz"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/sitemaps/section-sitemap.xml.gz"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/elections/results/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/section-fronts/list/"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-02-01","2024-03-05"],"summary":"Now blocked: ClaudeBot, Bytespider, anthropic-ai, PerplexityBot and Amazonbot.","dir":"up","up":["ClaudeBot","Bytespider","anthropic-ai","PerplexityBot","Amazonbot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240201011720/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20240305223339/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AwarioRssBot"},{"t":"add","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: peer39_crawler"},{"t":"add","s":"User-agent: peer39_crawler/1.0"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/"},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/arcio/news-sitemap/"},{"t":"add","s":"Sitemap: https://www.washingtonpost.com/arcio/news-sitemap-elections/"},{"t":"ctx","s":""}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2023-12","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-12-01"],"summary":"Now blocked: Google-Extended and CCBot.","dir":"up","up":["Google-Extended","CCBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001001645/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20231201043917/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /wpost/proxy"},{"t":"add","s":"Disallow: /ehf/"},{"t":"add","s":"Disallow: /ehf/*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/"}]},{"domain":"washingtonpost.com","name":"The Washington Post","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot and ChatGPT-User.","dir":"up","up":["GPTBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801004136/https://www.washingtonpost.com/robots.txt","https://web.archive.org/web/20230901052156/https://www.washingtonpost.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /ac2/"},{"t":"add","s":"Disallow: /archive/lifestyle/wellness/1999/01/19/an-illness-no-one-understands-tourettes-syndrome/852df0cc-2164-4c83-be8d-0c770d028547/"},{"t":"ctx","s":"Disallow: /blogs/slow-ride/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /native/"},{"t":"add","s":"Disallow: /subscribe/braintree/"},{"t":"add","s":"Disallow: /subscribe/enterpriseportal/"},{"t":"add","s":"Disallow: /subscribe/foryouapi/"},{"t":"add","s":"Disallow: /subscribe/lagoon/"},{"t":"add","s":"Disallow: /subscribe/offers/service/"},{"t":"add","s":"Disallow: /subscribe/onsiteapi/"},{"t":"add","s":"Disallow: /subscribe/paywall/"},{"t":"add","s":"Disallow: /subscribe/person/"},{"t":"add","s":"Disallow: /subscribe/preferenceapi/"},{"t":"add","s":"Disallow: /subscribe/subscriptionapi/"},{"t":"add","s":"Disallow: /subscribe/user/"},{"t":"add","s":"Disallow: /wpost/proxy"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/"}]}]}