{"domain":"apnews.com","name":"Associated Press","category":"news","subcategory":"wire","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 8 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Applebot-Extended, CCBot and Claude-SearchBot. Google-Extended, Meta-ExternalAgent, OAI-SearchBot, ChatGPT-User and 1 more can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.211Z","last_checked":"2026-10-19T13:34:19.572Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://apnews.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://apnews.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://apnews.com/robots.txt","hops":[],"blob":"9eebdaf40c8b8a0fad6b0a80e97286ccbdcc66927c75648cc6fb22c12d4191ea","text":"User-Agent: *\nDisallow: *_ptid=*\nDisallow: *?prx_t=*\nDisallow: /zephr*\nDisallow: /blaize*\nDisallow: /search?q=*\nDisallow: /*.rss\nDisallow: /buyline-shopping/*\nDisallow: /buyline-personal-finance/*\nDisallow: *?jw_start\nDisallow: *&jw_start\nDisallow: /gallery/\nDisallow: /config/newrelic/prod\nDisallow: /parsely-api-endpoint\nDisallow: /apdata/\nDisallow: /article/border-immigration-election-2182fec11fa008ec96c4202e5656d19e\nDisallow: /api/v2/feed/\nDisallow: /press-release/*\nDisallow: /*/widgets/\nDisallow: /*/widget/\nDisallow: /*/scripts/\n\nSitemap: https://apnews.com/ap-sitemap.xml\nSitemap: https://apnews.com/news-sitemap-content.xml\nSitemap: https://apnews.com/hubs-sitemap-content.xml\nSitemap: https://apnews.com/video-sitemap.xml\nSitemap: https://apnews.com/author-sitemap-content.xml\nSitemap: https://apnews.com/projects/elections-2026/sitemap-index.xml\n\n# Disallow Rules\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Bytedance\nDisallow: /\n\nUser-agent: Timpibot\nDisallow: /\n\nUser-agent: Claude-User\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","CCBot","GPTBot","anthropic-ai","ClaudeBot","Claude-Web","cohere-ai","PerplexityBot","Amazonbot","Applebot-Extended","Bytedance","Timpibot","Claude-User","Claude-SearchBot"],"lines":69},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":36},"any":{"allowed":false,"line":36},"search":{"allowed":false,"line":36},"cart":{"allowed":false,"line":36},"checkout":{"allowed":false,"line":36},"account":{"allowed":false,"line":36}},"disallows":1,"lines":[35,36],"groupRanges":[[35,36]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":42},"any":{"allowed":false,"line":42},"search":{"allowed":false,"line":42},"cart":{"allowed":false,"line":42},"checkout":{"allowed":false,"line":42},"account":{"allowed":false,"line":42}},"disallows":1,"lines":[41,42],"groupRanges":[[41,42]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":57},"any":{"allowed":false,"line":57},"search":{"allowed":false,"line":57},"cart":{"allowed":false,"line":57},"checkout":{"allowed":false,"line":57},"account":{"allowed":false,"line":57}},"disallows":1,"lines":[56,57],"groupRanges":[[56,57]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":33},"any":{"allowed":false,"line":33},"search":{"allowed":false,"line":33},"cart":{"allowed":false,"line":33},"checkout":{"allowed":false,"line":33},"account":{"allowed":false,"line":33}},"disallows":1,"lines":[32,33],"groupRanges":[[32,33]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":39},"any":{"allowed":false,"line":39},"search":{"allowed":false,"line":39},"cart":{"allowed":false,"line":39},"checkout":{"allowed":false,"line":39},"account":{"allowed":false,"line":39}},"disallows":1,"lines":[38,39],"groupRanges":[[38,39]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":51},"any":{"allowed":false,"line":51},"search":{"allowed":false,"line":51},"cart":{"allowed":false,"line":51},"checkout":{"allowed":false,"line":51},"account":{"allowed":false,"line":51}},"disallows":1,"lines":[50,51],"groupRanges":[[50,51]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":54},"any":{"allowed":false,"line":54},"search":{"allowed":false,"line":54},"cart":{"allowed":false,"line":54},"checkout":{"allowed":false,"line":54},"account":{"allowed":false,"line":54}},"disallows":1,"lines":[53,54],"groupRanges":[[53,54]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":66},"any":{"allowed":false,"line":66},"search":{"allowed":false,"line":66},"cart":{"allowed":false,"line":66},"checkout":{"allowed":false,"line":66},"account":{"allowed":false,"line":66}},"disallows":1,"lines":[65,66],"groupRanges":[[65,66]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[1,21],"groupRanges":[[1,21]]}],"lanes":{"training":{"level":"mixed","none":4,"full":5,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":5,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":1,"full":6,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"absent","http":404,"url":"https://apnews.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://apnews.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://apnews.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8}]},"events":[{"domain":"apnews.com","name":"Associated Press","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-07-01","2025-08-01"],"summary":"Now blocked: Claude-SearchBot and Claude-User.","dir":"up","up":["Claude-SearchBot","Claude-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250701002414/https://www.apnews.com/robots.txt","https://web.archive.org/web/20250801003654/https://www.apnews.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: *?prx_t=*"},{"t":"del","s":"Disallow: /press-release/*"},{"t":"del","s":"Disallow: /blaize*"},{"t":"ctx","s":"Disallow: /zephr*"},{"t":"add","s":"Disallow: /blaize*"},{"t":"ctx","s":"Disallow: /search?q=*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /buyline-personal-finance/*"},{"t":"add","s":"Disallow: *?jw_start"},{"t":"add","s":"Disallow: *&jw_start"},{"t":"add","s":"Disallow: */config/newrelic/prod*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"}]},{"domain":"apnews.com","name":"Associated Press","category":"news","date":"2025-05","precision":"month","source":"archive","provisional":false,"between":["2025-04-01","2025-05-01"],"summary":"Now blocked: Applebot-Extended.","dir":"up","up":["Applebot-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250401001015/https://apnews.com/robots.txt","https://web.archive.org/web/20250501013832/https://apnews.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"Sitemap: https://apnews.com/ap-sitemap.xml"},{"t":"add","s":"Sitemap: https://apnews.com/news-sitemap-content.xml"},{"t":"add","s":"Sitemap: https://apnews.com/hubs-sitemap-content.xml"},{"t":"add","s":"Sitemap: https://apnews.com/video-sitemap.xml"},{"t":"add","s":""},{"t":"add","s":"# Disallow Rules"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: CCBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://apnews.com/ap-sitemap.xml"},{"t":"del","s":"Sitemap: https://apnews.com/news-sitemap.xml"},{"t":"del","s":"Sitemap: https://apnews.com/video-sitemap.xml"},{"t":"del","s":""},{"t":"del","s":"# Disallow Rules"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Timpibot"},{"t":"add","s":"Disallow: /"}]},{"domain":"apnews.com","name":"Associated Press","category":"news","date":"2024-12","precision":"month","source":"archive","provisional":false,"between":["2024-11-01","2024-12-01"],"summary":"Now blocked: Amazonbot.","dir":"up","up":["Amazonbot"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241101001416/https://apnews.com/robots.txt","https://web.archive.org/web/20241201010309/https://apnews.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /press-release/*"},{"t":"del","s":"Disallow: /*blaize"},{"t":"del","s":"Disallow: /*zephr"},{"t":"add","s":"Disallow: /blaize*"},{"t":"add","s":"Disallow: /zephr*"},{"t":"add","s":"Disallow: /search?q=*"},{"t":"add","s":"Disallow: /*.rss"},{"t":"add","s":"Disallow: /buyline-shopping/*"},{"t":"add","s":"Disallow: /buyline-personal-finance/*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://apnews.com/ap-sitemap.xml"},{"t":"del","s":"Sitemap: https://apnews.com/news-sitemap-content.xml"},{"t":"add","s":"Sitemap: https://apnews.com/news-sitemap.xml"},{"t":"ctx","s":"Sitemap: https://apnews.com/video-sitemap.xml"},{"t":"del","s":"Sitemap: https://apnews.com/buyline-personal-finance/sitemap.xml"},{"t":"del","s":"Sitemap: https://apnews.com/buyline-shopping/sitemap.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytedance"},{"t":"add","s":"Disallow: /"}]},{"domain":"apnews.com","name":"Associated Press","category":"news","date":"2024-05","precision":"month","source":"archive","provisional":false,"between":["2024-04-01","2024-05-01"],"summary":"Now blocked: ClaudeBot, anthropic-ai and PerplexityBot.","dir":"up","up":["ClaudeBot","anthropic-ai","PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240401000928/https://www.apnews.com/robots.txt","https://web.archive.org/web/20240501004555/https://www.apnews.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /press-release/*"},{"t":"add","s":"Disallow: /*blaize"},{"t":"add","s":"Disallow: /*zephr"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://apnews.com/video-sitemap.xml"},{"t":"add","s":"Sitemap: https://apnews.com/buyline-personal-finance/sitemap.xml"},{"t":"add","s":"Sitemap: https://apnews.com/buyline-shopping/sitemap.xml"},{"t":"add","s":""},{"t":"add","s":"# Disallow Rules"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"}]},{"domain":"apnews.com","name":"Associated Press","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: GPTBot and CCBot.","dir":"up","up":["GPTBot","CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20231001005428/https://apnews.com/robots.txt","https://web.archive.org/web/20231101000433/https://apnews.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow:"},{"t":"del","s":""},{"t":"ctx","s":"Disallow: *_ptid=*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://apnews.com/sitemap/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://apnews.com/sitemap/google-news-sitemap/sitemap_index.xml"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"Sitemap: https://apnews.com/ap-sitemap.xml"},{"t":"add","s":"Sitemap: https://apnews.com/news-sitemap-content.xml"},{"t":"add","s":"Sitemap: https://apnews.com/video-sitemap.xml"}]}]}