The Atlantic

Bot by bot

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

# As a condition of accessing this website, you agree to abide by the# following content signals: # (a)  If a content-signal = yes, you may collect content for the#      corresponding use.# (b)  If a content-signal = no, you may not collect content for the#      corresponding use.# (c)  If the website operator does not include a content signal for a#      corresponding use, the website operator neither grants nor restricts#      permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning#         hyperlinks and short excerpts from your website's contents). Search#         does not include providing AI-generated search summaries.# ai-input: inputting content into one or more AI models (e.g., retrieval#           augmented generation, grounding, or other real-time taking of#           content for generative AI search answers).# ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. User-agent: *Disallow: /4624/TheAtlanticOnline/*Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/*Disallow: /ab/*Disallow: /video/embed/Disallow: /zephr/*Disallow: /video/iframe/*Disallow: /search/?*q=*Allow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/$Disallow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/*Crawl-delay: 1Allow: / User-agent: AmazonAdBotAllow: /… 41 lines not about AI bots User-agent: AmazonbotDisallow: /Allow: /feed/* User-agent: anthropic-aiDisallow: / User-agent: ApplebotAllow: / User-agent: Applebot-ExtendedDisallow: / User-agent: ArcSpanScraperAllow: / User-agent: AwarioRssBotUser-agent: AwarioSmartBotDisallow: / … 7 lines not about AI botsUser-agent: BingBotContent-signal: search=yes, ai-input=no, ai-train=noAllow: / User-agent: BytespiderDisallow: / User-agent: CCBotDisallow: / User-agent: ChatGPT-UserAllow: / User-agent: ClaudeBotDisallow: / User-agent: Claude-SearchBotAllow: /sponsored/Disallow: / User-agent: Claude-UserAllow: /sponsored/Disallow: / User-agent: Claude-WebDisallow: / User-agent: cohere-aiDisallow: / User-agent: DataForSeoBotDisallow: /… 2 lines not about AI bots User-agent: DiffbotDisallow: / User-agent: DuckAssistBotContent-signal: search=yes, ai-input=no, ai-train=noAllow: / User-agent: FacebookBotContent-signal: search=yes, ai-input=no, ai-train=noAllow: / User-agent: Google-ExtendedDisallow: / User-agent: GooglebotContent-signal: search=yes, ai-input=no, ai-train=noDisallow: /zephr/*Allow: /… 1 line not about AI bots User-agent: GPTBotAllow: / User-agent: magpie-crawlerDisallow: /… 2 lines not about AI bots User-agent: Meta-ExternalAgentDisallow: / User-agent: Meta-ExternalFetcherDisallow: / User-agent: MistralAI-UserDisallow: / User-agent: NewsNowDisallow: / User-agent: news-pleaseDisallow: /… 5 lines not about AI bots User-agent: omgiliDisallow: / User-agent: omgilibotDisallow: / User-agent: OAI-SearchBotAllow: / User-agent: OperatorAllow: /… 2 lines not about AI bots User-agent: PetalBotDisallow: / User-agent: PerplexityBotAllow: /sponsored/Disallow: / User-agent: Perplexity-UserAllow: /sponsored/Disallow: / User-agent: ProRataIncAllow: / User-agent: Quora-BotDisallow: / User-agent: ScrapyDisallow: / User-agent: TerraCottaAllow: /… 11 lines not about AI bots User-agent: TimpiBotDisallow: / User-agent: TurnitinBotDisallow: / User-agent: archive.org_botDisallow: / Sitemap: https://www.theatlantic.com/sitemap.xmlSitemap: https://www.theatlantic.com/sponsored/sitemap.xml … 9 lines not about AI bots

Notes about AI written in the file: "does not include providing AI-generated search summaries." · "ai-input: inputting content into one or more AI models (e.g., retrieval" · "content for generative AI search answers)." · "ai-train: training or fine-tuning AI models."

Everything else we checked

History

47 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

• Apr 2026
  More blocking Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added Content-Signal: search=yes, ai-input=no, ai-train=no.

  Found in the Internet Archive's copies: changed between 1 Feb 2026 and 1 Apr 2026. Evidence: before · after

  Show exactly what changed

  + # As a condition of accessing this website, you agree to abide by the

  + # following content signals:

  +

  + # (a) If a content-signal = yes, you may collect content for the

  + # corresponding use.

  + # (b) If a content-signal = no, you may not collect content for the

  + # corresponding use.

  + # (c) If the website operator does not include a content signal for a

  + # corresponding use, the website operator neither grants nor restricts

  + # permission via content signal with respect to the corresponding use.

  +

  + # The content signals and their meanings are:

  +

  + # search: building a search index and providing search results (e.g., returning

  + # hyperlinks and short excerpts from your website's contents). Search

  + # does not include providing AI-generated search summaries.

  + # ai-input: inputting content into one or more AI models (e.g., retrieval

  + # augmented generation, grounding, or other real-time taking of

  + # content for generative AI search answers).

  + # ai-train: training or fine-tuning AI models.

  +

  + # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF

  + # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT

  + # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

  +

  User-agent: *

  …

  Disallow: /video/embed/

  + Disallow: /zephr/*

  Disallow: /video/iframe/*

  …

  + User-agent: AmazonAdBot

  + Allow: /

  +

  User-agent: Amazonbot

  …

  + User-agent: Applebot

  + Disallow: /

  +

  User-agent: Applebot-Extended

  …

  + User-agent: BingBot

  + Content-signal: search=yes, ai-input=no, ai-train=no

  + Allow: /

  +

  User-agent: Bytespider

  …

  + User-agent: ChatGPT-User

  + Allow: /

  +

  User-agent: ClaudeBot

  …

  + User-agent: Claude-SearchBot

  + Disallow: /

  +

  … 70 more lines
• Sep 2025
  More blocking Now blocked: Perplexity-User.

  Found in the Internet Archive's copies: changed between 1 Aug 2025 and 11 Sep 2025. Evidence: before · after

  Show exactly what changed

  Disallow: /

  + Allow: /feed/*

  …

  + User-agent: Perplexity-User

  + Disallow: /

  +

  + User-agent: Quora-Bot

  + Disallow: /

  +

  User-agent: Scrapy
• Jul 2024
  Mixed Now blocked: ClaudeBot, Applebot-Extended, Bytespider, anthropic-ai, PerplexityBot and Amazonbot. Now allowed: GPTBot and ChatGPT-User.

  Found in the Internet Archive's copies: changed between 1 May 2024 and 1 Jul 2024. Evidence: before · after

  Show exactly what changed

  + User-agent: Amazonbot

  + Disallow: /

  +

  + User-agent: anthropic-ai

  + Disallow: /

  +

  + User-agent: Applebot-Extended

  + Disallow: /

  +

  + User-agent: AwarioRssBot

  + User-agent: AwarioSmartBot

  + Disallow: /

  +

  + User-agent: Bytespider

  + Disallow: /

  +

  User-agent: CCBot

  …

  - User-agent: GPTBot

  + User-agent: ClaudeBot

  Disallow: /

  + User-agent: Claude-Web

  + Disallow: /

  +

  + User-agent: cohere-ai

  + Disallow: /

  +

  + User-agent: DataForSeoBot

  + Disallow: /

  +

  + User-agent: Diffbot

  + Disallow: /

  +

  + User-agent: FacebookBot

  + Disallow: /

  +

  User-agent: Google-Extended

  …

  - User-agent: ChatGPT-User

  + User-agent: magpie-crawler

  Disallow: /

  + User-agent: NewsNow

  + Disallow: /

  +

  + User-agent: news-please

  + Disallow: /

  +

  + User-agent: omgili

  + Disallow: /

  +

  + User-agent: omgilibot

  + Disallow: /

  +

  + User-agent: PerplexityBot

  + Disallow: /

  … 11 more lines
• Nov 2023
  More blocking Now blocked: Google-Extended.

  Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after

  Show exactly what changed

  + User-agent: Google-Extended

  + Disallow: /

  +

  User-agent: ChatGPT-User
• Sep 2023
  More blocking Now blocked: GPTBot, CCBot and ChatGPT-User.

  Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after

  Show exactly what changed

  + User-agent: CCBot

  + Disallow: /

  +

  + User-agent: GPTBot

  + Disallow: /

  +

  + User-agent: ChatGPT-User

  + Disallow: /

  +

  Sitemap: https://www.theatlantic.com/sitemap.xml

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.