{"domain":"theatlantic.com","name":"The Atlantic","category":"news","subcategory":"magazine","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 13 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot and ChatGPT-User can reach most of it. The homepage also turned our bot, which says who it is, away (HTTP 403).","computed_at":"2026-10-20T03:43:15.237Z","last_checked":"2026-10-19T13:34:50.496Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.theatlantic.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.theatlantic.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.theatlantic.com/robots.txt","hops":[{"from":"https://theatlantic.com/robots.txt","to":"https://www.theatlantic.com/robots.txt","status":301}],"blob":"500c1ee9d0c39ec3890bc0d1433e8506008290f278ce37daf6eed7358ef1bd5a","text":"# As a condition of accessing this website, you agree to abide by the\n# following content signals:\n\n# (a)  If a content-signal = yes, you may collect content for the\n#      corresponding use.\n# (b)  If a content-signal = no, you may not collect content for the\n#      corresponding use.\n# (c)  If the website operator does not include a content signal for a\n#      corresponding use, the website operator neither grants nor restricts\n#      permission via content signal with respect to the corresponding use.\n\n# The content signals and their meanings are:\n\n# search: building a search index and providing search results (e.g., returning\n#         hyperlinks and short excerpts from your website's contents). Search\n#         does not include providing AI-generated search summaries.\n# ai-input: inputting content into one or more AI models (e.g., retrieval\n#           augmented generation, grounding, or other real-time taking of\n#           content for generative AI search answers).\n# ai-train: training or fine-tuning AI models.\n\n# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF\n# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT\n# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.\n\nUser-agent: *\nDisallow: /4624/TheAtlanticOnline/*\nDisallow: /magazine/archive/2010/11/letters-to-the-editor/308258/\nDisallow: /magazine/archive/2010/11/letters-to-the-editor/308258/*\nDisallow: /ab/*\nDisallow: /video/embed/\nDisallow: /zephr/*\nDisallow: /video/iframe/*\nDisallow: /search/?*q=*\nAllow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/$\nDisallow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/*\nCrawl-delay: 1\nAllow: /\n\nUser-agent: AmazonAdBot\nAllow: /\n\nUser-agent: Amazonbot\nDisallow: /\nAllow: /feed/*\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: Applebot\nAllow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: ArcSpanScraper\nAllow: /\n\nUser-agent: AwarioRssBot\nUser-agent: AwarioSmartBot\nDisallow: /\n\nUser-agent: BingBot\nContent-signal: search=yes, ai-input=no, ai-train=no\nAllow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nAllow: /sponsored/\nDisallow: /\n\nUser-agent: Claude-User\nAllow: /sponsored/\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: DataForSeoBot\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: DuckAssistBot\nContent-signal: search=yes, ai-input=no, ai-train=no\nAllow: /\n\nUser-agent: FacebookBot\nContent-signal: search=yes, ai-input=no, ai-train=no\nAllow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: Googlebot\nContent-signal: search=yes, ai-input=no, ai-train=no\nDisallow: /zephr/*\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: Meta-ExternalAgent\nDisallow: /\n\nUser-agent: Meta-ExternalFetcher\nDisallow: /\n\nUser-agent: MistralAI-User\nDisallow: /\n\nUser-agent: NewsNow\nDisallow: /\n\nUser-agent: news-please\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: omgilibot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: Operator\nAllow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: PerplexityBot\nAllow: /sponsored/\nDisallow: /\n\nUser-agent: Perplexity-User\nAllow: /sponsored/\nDisallow: /\n\nUser-agent: ProRataInc\nAllow: /\n\nUser-agent: Quora-Bot\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: TerraCotta\nAllow: /\n\nUser-agent: TimpiBot\nDisallow: /\n\nUser-agent: TurnitinBot\nDisallow: /\n\nUser-agent: archive.org_bot\nDisallow: /\n\nSitemap: https://www.theatlantic.com/sitemap.xml\nSitemap: https://www.theatlantic.com/sponsored/sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[{"value":"search=yes, ai-input=no, ai-train=no","signals":{"search":"yes","ai-input":"no","ai-train":"no"},"line":64,"agents":["bingbot"]},{"value":"search=yes, ai-input=no, ai-train=no","signals":{"search":"yes","ai-input":"no","ai-train":"no"},"line":100,"agents":["duckassistbot"]},{"value":"search=yes, ai-input=no, ai-train=no","signals":{"search":"yes","ai-input":"no","ai-train":"no"},"line":104,"agents":["facebookbot"]},{"value":"search=yes, ai-input=no, ai-train=no","signals":{"search":"yes","ai-input":"no","ai-train":"no"},"line":111,"agents":["googlebot"]}],"contentUsage":[],"licenses":[],"aiComments":[{"line":16,"text":"does not include providing AI-generated search summaries."},{"line":17,"text":"ai-input: inputting content into one or more AI models (e.g., retrieval"},{"line":19,"text":"content for generative AI search answers)."},{"line":20,"text":"ai-train: training or fine-tuning AI models."}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","AmazonAdBot","Amazonbot","anthropic-ai","Applebot","Applebot-Extended","ArcSpanScraper","AwarioRssBot","AwarioSmartBot","BingBot","Bytespider","CCBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","Claude-Web","cohere-ai","DataForSeoBot","Diffbot","DuckAssistBot","FacebookBot","Google-Extended","Googlebot","GPTBot","magpie-crawler","Meta-ExternalAgent","Meta-ExternalFetcher","MistralAI-User","NewsNow","news-please","omgili","omgilibot","OAI-SearchBot","Operator","PetalBot","PerplexityBot","Perplexity-User","ProRataInc","Quora-Bot","Scrapy","TerraCotta","TimpiBot","TurnitinBot","archive.org_bot"],"lines":182},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":116},"any":{"allowed":true,"line":116},"search":{"allowed":true,"line":116},"cart":{"allowed":true,"line":116},"checkout":{"allowed":true,"line":116},"account":{"allowed":true,"line":116}},"disallows":0,"lines":[115,116],"groupRanges":[[115,116]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":77},"any":{"allowed":false,"line":77},"search":{"allowed":false,"line":77},"cart":{"allowed":false,"line":77},"checkout":{"allowed":false,"line":77},"account":{"allowed":false,"line":77}},"disallows":1,"lines":[76,77],"groupRanges":[[76,77]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":108},"any":{"allowed":false,"line":108},"search":{"allowed":false,"line":108},"cart":{"allowed":false,"line":108},"checkout":{"allowed":false,"line":108},"account":{"allowed":false,"line":108}},"disallows":1,"lines":[107,108],"groupRanges":[[107,108]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":54},"any":{"allowed":false,"line":54},"search":{"allowed":false,"line":54},"cart":{"allowed":false,"line":54},"checkout":{"allowed":false,"line":54},"account":{"allowed":false,"line":54}},"disallows":1,"lines":[53,54],"groupRanges":[[53,54]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":71},"any":{"allowed":false,"line":71},"search":{"allowed":false,"line":71},"cart":{"allowed":false,"line":71},"checkout":{"allowed":false,"line":71},"account":{"allowed":false,"line":71}},"disallows":1,"lines":[70,71],"groupRanges":[[70,71]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":122},"any":{"allowed":false,"line":122},"search":{"allowed":false,"line":122},"cart":{"allowed":false,"line":122},"checkout":{"allowed":false,"line":122},"account":{"allowed":false,"line":122}},"disallows":1,"lines":[121,122],"groupRanges":[[121,122]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":68},"any":{"allowed":false,"line":68},"search":{"allowed":false,"line":68},"cart":{"allowed":false,"line":68},"checkout":{"allowed":false,"line":68},"account":{"allowed":false,"line":68}},"disallows":1,"lines":[67,68],"groupRanges":[[67,68]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":48},"any":{"allowed":false,"line":48},"search":{"allowed":false,"line":48},"cart":{"allowed":false,"line":48},"checkout":{"allowed":false,"line":48},"account":{"allowed":false,"line":48}},"disallows":1,"lines":[47,48],"groupRanges":[[47,48]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":143},"any":{"allowed":true,"line":143},"search":{"allowed":true,"line":143},"cart":{"allowed":true,"line":143},"checkout":{"allowed":true,"line":143},"account":{"allowed":true,"line":143}},"disallows":0,"lines":[142,143],"groupRanges":[[142,143]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":81},"any":{"allowed":false,"line":81},"search":{"allowed":false,"line":81},"cart":{"allowed":false,"line":81},"checkout":{"allowed":false,"line":81},"account":{"allowed":false,"line":81}},"disallows":1,"lines":[79,81],"groupRanges":[[79,81]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":153},"any":{"allowed":false,"line":153},"search":{"allowed":false,"line":153},"cart":{"allowed":false,"line":153},"checkout":{"allowed":false,"line":153},"account":{"allowed":false,"line":153}},"disallows":1,"lines":[151,153],"groupRanges":[[151,153]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":44},"any":{"allowed":false,"line":44},"search":{"allowed":false,"line":44},"cart":{"allowed":false,"line":44},"checkout":{"allowed":false,"line":44},"account":{"allowed":false,"line":44}},"disallows":1,"lines":[43,45],"groupRanges":[[43,45]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":0,"lines":[99,101],"groupRanges":[[99,101]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":74},"any":{"allowed":true,"line":74},"search":{"allowed":true,"line":74},"cart":{"allowed":true,"line":74},"checkout":{"allowed":true,"line":74},"account":{"allowed":true,"line":74}},"disallows":0,"lines":[73,74],"groupRanges":[[73,74]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":85},"any":{"allowed":false,"line":85},"search":{"allowed":false,"line":85},"cart":{"allowed":false,"line":85},"checkout":{"allowed":false,"line":85},"account":{"allowed":false,"line":85}},"disallows":1,"lines":[83,85],"groupRanges":[[83,85]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":157},"any":{"allowed":false,"line":157},"search":{"allowed":false,"line":157},"cart":{"allowed":false,"line":157},"checkout":{"allowed":false,"line":157},"account":{"allowed":false,"line":157}},"disallows":1,"lines":[155,157],"groupRanges":[[155,157]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":125},"any":{"allowed":false,"line":125},"search":{"allowed":false,"line":125},"cart":{"allowed":false,"line":125},"checkout":{"allowed":false,"line":125},"account":{"allowed":false,"line":125}},"disallows":1,"lines":[124,125],"groupRanges":[[124,125]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":38},"any":{"allowed":true,"line":38},"search":{"allowed":true,"line":38},"cart":{"allowed":true,"line":38},"checkout":{"allowed":true,"line":38},"account":{"allowed":true,"line":38}},"disallows":9,"lines":[26,38],"groupRanges":[[26,38]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":128},"any":{"allowed":false,"line":128},"search":{"allowed":false,"line":128},"cart":{"allowed":false,"line":128},"checkout":{"allowed":false,"line":128},"account":{"allowed":false,"line":128}},"disallows":1,"lines":[127,128],"groupRanges":[[127,128]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":113},"any":{"allowed":true,"line":113},"search":{"allowed":true,"line":113},"cart":{"allowed":true,"line":113},"checkout":{"allowed":true,"line":113},"account":{"allowed":true,"line":113}},"disallows":1,"lines":[110,113],"groupRanges":[[110,113]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":65},"any":{"allowed":true,"line":65},"search":{"allowed":true,"line":65},"cart":{"allowed":true,"line":65},"checkout":{"allowed":true,"line":65},"account":{"allowed":true,"line":65}},"disallows":0,"lines":[63,65],"groupRanges":[[63,65]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":5,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":4,"full":3,"restricted":0,"n":7}},"door":{"door":"refused","http":403},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"refused","http":403,"url":"https://www.theatlantic.com/llms.txt"},"ai_txt":{"state":"refused","http":403,"url":"https://www.theatlantic.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.theatlantic.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13}]},"events":[{"domain":"theatlantic.com","name":"The Atlantic","category":"news","date":"2026-04","precision":"month","source":"archive","provisional":false,"between":["2026-02-01","2026-04-01"],"summary":"Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added <code>Content-Signal: search=yes, ai-input=no, ai-train=no</code>.","dir":"up","up":["Meta-ExternalAgent","Claude-SearchBot","Claude-User","Meta-ExternalFetcher","MistralAI-User"],"down":[],"major":true,"signals":true,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260201014820/https://www.theatlantic.com/robots.txt","https://web.archive.org/web/20260401001519/https://www.theatlantic.com/robots.txt"],"diff":[{"t":"add","s":"# As a condition of accessing this website, you agree to abide by the"},{"t":"add","s":"# following content signals:"},{"t":"add","s":""},{"t":"add","s":"# (a)  If a content-signal = yes, you may collect content for the"},{"t":"add","s":"#      corresponding use."},{"t":"add","s":"# (b)  If a content-signal = no, you may not collect content for the"},{"t":"add","s":"#      corresponding use."},{"t":"add","s":"# (c)  If the website operator does not include a content signal for a"},{"t":"add","s":"#      corresponding use, the website operator neither grants nor restricts"},{"t":"add","s":"#      permission via content signal with respect to the corresponding use."},{"t":"add","s":""},{"t":"add","s":"# The content signals and their meanings are:"},{"t":"add","s":""},{"t":"add","s":"# search: building a search index and providing search results (e.g., returning"},{"t":"add","s":"#         hyperlinks and short excerpts from your website's contents). Search"},{"t":"add","s":"#         does not include providing AI-generated search summaries."},{"t":"add","s":"# ai-input: inputting content into one or more AI models (e.g., retrieval"},{"t":"add","s":"#           augmented generation, grounding, or other real-time taking of"},{"t":"add","s":"#           content for generative AI search answers)."},{"t":"add","s":"# ai-train: training or fine-tuning AI models."},{"t":"add","s":""},{"t":"add","s":"# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF"},{"t":"add","s":"# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT"},{"t":"add","s":"# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET."},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /video/embed/"},{"t":"add","s":"Disallow: /zephr/*"},{"t":"ctx","s":"Disallow: /video/iframe/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: AmazonAdBot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Applebot-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: BingBot"},{"t":"add","s":"Content-signal: search=yes, ai-input=no, ai-train=no"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Bytespider"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Allow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: ClaudeBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"gap","s":"… 70 more lines"}]},{"domain":"theatlantic.com","name":"The Atlantic","category":"news","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-09-11"],"summary":"Now blocked: Perplexity-User.","dir":"up","up":["Perplexity-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250801110117/https://www.theatlantic.com/robots.txt","https://web.archive.org/web/20250911204856/https://www.theatlantic.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /feed/*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Quora-Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Scrapy"}]},{"domain":"theatlantic.com","name":"The Atlantic","category":"news","date":"2024-07","precision":"month","source":"archive","provisional":false,"between":["2024-05-01","2024-07-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, Bytespider, anthropic-ai, PerplexityBot and Amazonbot. Now allowed: GPTBot and ChatGPT-User.","dir":"mixed","up":["ClaudeBot","Applebot-Extended","Bytespider","anthropic-ai","PerplexityBot","Amazonbot"],"down":["GPTBot","ChatGPT-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240501001750/https://www.theatlantic.com/robots.txt","https://web.archive.org/web/20240701014252/https://www.theatlantic.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AwarioRssBot"},{"t":"add","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: CCBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: NewsNow"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: news-please"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 11 more lines"}]},{"domain":"theatlantic.com","name":"The Atlantic","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001011212/https://www.theatlantic.com/robots.txt","https://web.archive.org/web/20231101030644/https://www.theatlantic.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: ChatGPT-User"}]},{"domain":"theatlantic.com","name":"The Atlantic","category":"news","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot, CCBot and ChatGPT-User.","dir":"up","up":["GPTBot","CCBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801002703/https://www.theatlantic.com/robots.txt","https://web.archive.org/web/20230901001836/https://www.theatlantic.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.theatlantic.com/sitemap.xml"}]}]}