The Walled Web Newsletter

All sites / News and publishing

USA Today

usatoday.com
Blocks AI

Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 22 of the 24 bots in total. It names every major AI bot one by one.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

8 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

7 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Blocked

7 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI Blocked Named, line 55 Says yes
ClaudeBotAnthropic Blocked Named, line 30 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Blocked Named, line 71 Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Named, line 139 Says yes
CCBotCommon Crawl Blocked Named, line 68 Says yes
Meta-ExternalAgentMeta Blocked Named, line 142 Suggests yes
BytespiderByteDance Blocked Named, line 161 No statement found
MistralAI-TrainingMistral AI Allowed Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Blocked Named, line 170 Does not say
anthropic-aiAnthropic (legacy) · old name Blocked Named, line 3 Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI Blocked Named, line 148 Says yes
Claude-SearchBotAnthropic Blocked Named, line 167 Says yes
PerplexityBotPerplexity Blocked Named, line 120 Suggests yes
AmazonbotAmazon Blocked Named, line 6 Says yes
Amzn-SearchBotAmazon Blocked Named, line 18 Says yes
DuckAssistBotDuckDuckGo Blocked Named, line 212 Says yes
MistralAI-IndexMistral AI Allowed Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Blocked Named, line 525 Suggests yes
AI assistants · Blocked
ChatGPT-UserOpenAI Blocked Named, line 33 Says not always
Claude-UserAnthropic Blocked Named, line 164 Says yes
Perplexity-UserPerplexity Blocked Named, line 555 Says it mostly ignores it
Google-AgentGoogle Blocked Named, line 492 Says it mostly ignores it
Meta-ExternalFetcherMeta Blocked Named, line 263 Says not always
Amzn-UserAmazon Blocked Named, line 21 Says not always
MistralAI-UserMistral AI Blocked Named, line 441 Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNot named; the catch-all rule applies
BingbotRegular search engine, for comparisonAllowedNot named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

ClaudeBlockedline 9Not in our verified list
Claude-WebBlockedline 12Not in our verified list
claudewebBlockedline 15Not in our verified list
cohere-aiBlockedline 27Not in our verified list
DiffbotBlockedline 87Documented by its operator
Diffbot-userBlockedline 90Documented by its operator
FacebookBotBlockedline 96Not in our verified list
omgiliBlockedline 108Not in our verified list
omgilibotBlockedline 111Not in our verified list
Ai2Bot-DolmaBlockedline 176Not in our verified list
ApplebotBlockedline 179Documented by its operator
cohere-training-data-crawlerBlockedline 200Not in our verified list
Google-CloudVertexBotBlockedline 218Documented by its operator
ImagesiftBotBlockedline 239Not in our verified list
PetalBotBlockedline 287Not in our verified list
TimpibotBlockedline 305Not in our verified list
Webzio-ExtendedBlockedline 314Not in our verified list
YouBotBlockedline 359Not in our verified list
Big Sur AIBlockedline 377Not in our verified list
ChatGPT OperatorBlockedline 386Not in our verified list
ChatGPT-PageFetcherBlockedline 399Not in our verified list
OpenAI-CheckBotBlockedline 402Not in our verified list
OpenAI-ExploreBlockedline 405Not in our verified list
OpenAI FeedbackBlockedline 408Not in our verified list
OpenAI-SnippetBotBlockedline 411Not in our verified list
GrokBlockedline 429Operator publishes no such name
DeepSeekBotBlockedline 480Not in our verified list
GoogleOtherBlockedline 486Documented by its operator
Google AIBlockedline 489Not in our verified list
FirecrawlAgentBlockedline 498Not in our verified list
Gemini-Deep-ResearchBlockedline 549Not in our verified list
Perplexity Search BotBlockedline 558Not in our verified list
AI2Bot-DeepResearchEvalBlockedline 588Not in our verified list
bigsur.aiBlockedline 603Not in our verified list
ChatGPT AgentBlockedline 606Not in our verified list
GoogleAgent-MarinerBlockedline 609Not in our verified list
Manus-UserBlockedline 615Not in our verified list
Jina-AIBlockedline 633Not in our verified list
WebScraping.AIBlockedline 765Not in our verified list
webscraping-aiBlockedline 771Not in our verified list
Forage-AIBlockedline 777Not in our verified list
Dumpling-AIBlockedline 786Not in our verified list
Browse-AIBlockedline 804Not in our verified list
skrape.aiBlockedline 852Not in our verified list
Exa-AIBlockedline 915Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 986 lines · 200 lines about AI bots Source
# robots.txt file for https://www.usatoday.com/… 1 line not about AI bots User-agent: anthropic-aiDisallow: / User-agent: AmazonBotDisallow: / User-agent: ClaudeDisallow: / User-agent: Claude-WebDisallow: / User-agent: claudewebDisallow: / User-agent: Amzn-SearchBotDisallow: / User-agent: Amzn-UserDisallow: / User-agent: AndiBotDisallow: /… 2 lines not about AI bots User-agent: cohere-aiDisallow: / User-agent: ClaudeBotDisallow: / User-agent: ChatGPT-UserDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: Googlebot-NewsDisallow: /story/sponsor-story/Disallow: /picture-gallery/sponsor-story/Disallow: /videos/sponsor-story/Disallow: /longform/sponsor-story/Disallow: /pages/interactives/sponsor-story/Disallow: /interactives/sponsor-story/Disallow: /videos/embed/… 8 lines not about AI bots User-agent: GPTBotDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: CCBotDisallow: / User-agent: Google-ExtendedDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: DataForSeoBotDisallow: /… 2 lines not about AI bots User-agent: DiffbotDisallow: / User-agent: Diffbot-userDisallow: / User-agent: KagibotDisallow: /… 2 lines not about AI bots User-agent: FacebookBotDisallow: / User-agent: magpie-crawlerDisallow: / User-agent: NewsNowDisallow: / User-agent: news-pleaseDisallow: /… 8 lines not about AI bots User-agent: omgiliDisallow: / User-agent: omgilibotDisallow: / User-agent: peer39_crawlerDisallow: / User-agent: peer39_crawler/1.0Disallow: /… 5 lines not about AI bots User-agent: PerplexityBotDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: ScrapyDisallow: / User-agent: TurnitinBotDisallow: /… 5 lines not about AI bots User-agent: Applebot-ExtendedDisallow: / User-agent: Meta-ExternalAgentDisallow: / User-agent: meta-externalagentDisallow: / User-agent: OAI-SearchBotDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: BytespiderDisallow: / User-agent: Claude-UserDisallow: / User-agent: Claude-SearchBotDisallow: / User-agent: AI2BotDisallow: / User-agent: AhrefsBotDisallow: /… 2 lines not about AI bots User-agent: Ai2Bot-DolmaDisallow: / User-agent: ApplebotDisallow: / User-agent: Arquivo-web-crawlerDisallow: / User-agent: AspiegelBotDisallow: / User-agent: AwarioRssBotDisallow: / User-agent: AwarioSmartBotDisallow: / User-agent: BarkrowlerDisallow: / User-agent: coccocbot-webDisallow: /… 17 lines not about AI bots User-agent: cohere-training-data-crawlerDisallow: / User-agent: crawler4jDisallow: / User-agent: CrawlspaceDisallow: / User-agent: dotbotDisallow: /… 8 lines not about AI bots User-agent: DuckAssistBotDisallow: / User-agent: FriendlyCrawlerDisallow: /… 2 lines not about AI bots User-agent: Google-CloudVertexBotDisallow: / User-agent: https://hada.newsDisallow: / User-agent: https://www.imediaethics.orgDisallow: / User-agent: iaskspider/2.0Disallow: / User-agent: ICC-CrawlerDisallow: / User-agent: icc-crawlerDisallow: / User-agent: ICC CrawlerDisallow: /… 17 lines not about AI bots User-agent: ImagesiftBotDisallow: / User-agent: img2datasetDisallow: / User-agent: ISSCyberRiskCrawlerDisallow: / User-agent: JenkersBotDisallow: / User-agent: Kangaroo BotDisallow: / User-agent: LivelapBotDisallow: / User-agent: MauiBotDisallow: / User-agent: MeltwaterDisallow: /… 20 lines not about AI bots User-agent: Meta-ExternalFetcherDisallow: / User-agent: MJ12botDisallow: / User-agent: MojeekDisallow: / User-agent: MojeekBotDisallow: / User-agent: MoodleBotDisallow: / User-agent: NeticleBotDisallow: / User-agent: NetvibesDisallow: / User-agent: PanguBotDisallow: /… 20 lines not about AI bots User-agent: PetalBotDisallow: / User-agent: SeekportBotDisallow: / User-agent: SeekrDisallow: / User-agent: SemrushBotDisallow: / User-agent: SeznamBotDisallow: / User-agent: Sidetrade Indexer BotDisallow: /… 14 lines not about AI bots User-agent: TimpibotDisallow: / User-agent: TurnitinDisallow: / User-agent: VelenPublicWebCrawlerDisallow: /… 5 lines not about AI bots User-agent: Webzio-ExtendedDisallow: / User-agent: yacyDisallow: / User-agent: yacybotDisallow: / User-agent: YandexDisallow: / User-agent: YandexAdditionalDisallow: / User-agent: YandexAdditionalBotDisallow: / User-agent: YandexBotDisallow: / User-agent: YandexImagesDisallow: / User-agent: YandexNewsDisallow: / User-agent: BaiduspiderDisallow: / User-agent: Baiduspider-imageDisallow: / User-agent: BaiduSpider-renderDisallow: / User-agent: Baiduspider-videoDisallow: / User-agent: Sogou Web SpiderDisallow: / User-agent: yisouspiderDisallow: /… 41 lines not about AI bots User-agent: YouBotDisallow: / User-agent: ia_archiverDisallow: / User-agent: archive.org_botDisallow: / User-agent: ia_archiver-web.archive.orgDisallow: / User-agent: aiHitBotDisallow: / User-agent: Amazon KendraDisallow: /… 14 lines not about AI bots User-agent: Big Sur AIDisallow: / User-agent: BrandwatchDisallow: / User-agent: BravestDisallow: /… 5 lines not about AI bots User-agent: ChatGPT OperatorDisallow: /Allow: /branded-story/Allow: /contributor-content/Allow: /story/branded-story/Allow: /story/special/contributor-content/Allow: /money/money-management/Allow: /shopping/Allow: /story/shopping/Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ User-agent: ChatGPT-PageFetcherDisallow: / User-agent: OpenAI-CheckBotDisallow: / User-agent: OpenAI-ExploreDisallow: / User-agent: OpenAI FeedbackDisallow: / User-agent: OpenAI-SnippetBotDisallow: / User-agent: CotoyogiDisallow: / User-agent: DigitalOceanGenAICrawlerDisallow: / User-agent: Echobot BotDisallow: / User-agent: EchoboxBotDisallow: / User-agent: Factset_spyderbotDisallow: /… 14 lines not about AI bots User-agent: GrokDisallow: / User-agent: iaskspiderDisallow: / User-agent: imgproxyDisallow: / User-agent: LINER BotDisallow: /… 8 lines not about AI bots User-agent: MistralAI-UserDisallow: / User-agent: netEstate Imprint CrawlerDisallow: / User-agent: NovaActDisallow: / User-agent: QualifiedBotDisallow: / User-agent: SBIntuitionsBotDisallow: / User-agent: sbintuitionsbotDisallow: / User-agent: SemrushBot-OCOBDisallow: / User-agent: SemrushBotSwaDisallow: / User-agent: SidetradeDisallow: / User-agent: TikTokSpiderDisallow: / User-agent: WARDBotDisallow: / User-agent: TaraGroup Intelligent Bot V1Disallow: / User-agent: TaraGroup Intelligent BotDisallow: /… 35 lines not about AI bots User-agent: DeepSeekBotDisallow: / User-agent: ChatGLM-SpiderDisallow: /… 2 lines not about AI bots User-agent: GoogleOtherDisallow: / User-agent: Google AIDisallow: / User-agent: Google-AgentDisallow: / User-agent: shapbotDisallow: /… 2 lines not about AI bots User-agent: FirecrawlAgentDisallow: / User-agent: Crawl4AIDisallow: / User-agent: imageSpiderDisallow: / User-agent: laion-huggingface-processorDisallow: / User-agent: LCCDisallow: / User-agent: SpiderDisallow: / User-agent: Datenbank CrawlerDisallow: / User-agent: Cloudflare-AutoRAGDisallow: / User-agent: CloudflareBrowserRenderingCrawlerDisallow: /… 23 lines not about AI bots User-agent: meta-webindexerDisallow: / User-agent: BravebotDisallow: / User-agent: AddSearchBotDisallow: / User-agent: AnomuraDisallow: / User-agent: atlassian-botDisallow: / User-agent: Channel3BotDisallow: / User-agent: LinkupBotDisallow: / User-agent: ZanistaBotDisallow: /… 20 lines not about AI bots User-agent: Gemini-Deep-ResearchDisallow: / User-agent: Google-NotebookLMDisallow: /… 2 lines not about AI bots User-agent: Perplexity-UserDisallow: / User-agent: Perplexity Search BotDisallow: / User-agent: PoetbotDisallow: / User-agent: Poseidon Research CrawlerDisallow: / User-agent: ProRataIncDisallow: / User-agent: QuillBotDisallow: / User-agent: Tiingo News AgentDisallow: / User-agent: iboubotDisallow: / User-agent: TavilyBotDisallow: / User-agent: NeevaBotDisallow: / User-agent: PhindBotDisallow: /… 26 lines not about AI bots User-agent: AI2Bot-DeepResearchEvalDisallow: / User-agent: KlaviyoAIBotDisallow: / User-agent: LinerBotDisallow: / User-agent: Poggio-CitationsDisallow: / User-agent: DevinDisallow: /… 11 lines not about AI bots User-agent: bigsur.aiDisallow: / User-agent: ChatGPT AgentDisallow: / User-agent: GoogleAgent-MarinerDisallow: / User-agent: AmazonBuyForMeDisallow: /… 2 lines not about AI bots User-agent: Manus-UserDisallow: / User-agent: manus-userDisallow: / User-agent: TwinAgentDisallow: / User-agent: AudigentAdBotDisallow: / User-agent: AudigentBotDisallow: / User-agent: JinaAIDisallow: /… 11 lines not about AI bots User-agent: Jina-AIDisallow: / User-agent: JinaBotDisallow: / User-agent: Jina-ReaderDisallow: / User-agent: BrightDataDisallow: / User-agent: Bright DataDisallow: / User-agent: BrightBotDisallow: / User-agent: SerpApiDisallow: / User-agent: SerpApiBotDisallow: / User-agent: ApifyDisallow: / User-agent: ApifyBotDisallow: / User-agent: Apify-HTTP-ClientDisallow: / User-agent: ScrapeGraphAIDisallow: / User-agent: scrapegraphDisallow: / User-agent: ScrapelessDisallow: / User-agent: ScrapelessBotDisallow: / User-agent: ZenRowsDisallow: / User-agent: ZenRowsBotDisallow: / User-agent: PromptCloudDisallow: / User-agent: PromptCloudBotDisallow: / User-agent: NimblewayDisallow: / User-agent: NimbleCrawlerDisallow: / User-agent: NimblewayBotDisallow: / User-agent: OxylabsDisallow: / User-agent: OxylabsBotDisallow: / User-agent: ZyteDisallow: / User-agent: ZyteBotDisallow: / User-agent: ScrapinghubDisallow: / User-agent: ScraperAPIDisallow: / User-agent: Scraper-APIDisallow: / User-agent: ScrapingBeeDisallow: / User-agent: ScrapingBeeBotDisallow: / User-agent: ScrapingdogDisallow: / User-agent: ScrapingdogBotDisallow: / User-agent: GrepsrDisallow: / User-agent: GrepsrBotDisallow: / User-agent: GrepsrCrawlerDisallow: / User-agent: SpiderBotDisallow: / User-agent: spider-cloudDisallow: / User-agent: SpiderCloudDisallow: / User-agent: HyperbrowserDisallow: / User-agent: HyperbrowserBotDisallow: / User-agent: HyperAgentDisallow: / User-agent: ScrapestackDisallow: / User-agent: ScrapestackBotDisallow: /… 128 lines not about AI bots User-agent: WebScraping.AIDisallow: / User-agent: WebScrapingAIDisallow: /… 2 lines not about AI bots User-agent: webscraping-aiDisallow: / User-agent: ForageAIDisallow: /… 2 lines not about AI bots User-agent: Forage-AIDisallow: / User-agent: ForageBotDisallow: / User-agent: DumplingAIDisallow: /… 5 lines not about AI bots User-agent: Dumpling-AIDisallow: / User-agent: DumplingBotDisallow: / User-agent: Import.ioDisallow: / User-agent: ImportIODisallow: / User-agent: importioDisallow: / User-agent: BrowseAIDisallow: /… 14 lines not about AI bots User-agent: Browse-AIDisallow: / User-agent: BrowseAIBotDisallow: / User-agent: DecodoDisallow: / User-agent: DecodoBotDisallow: / User-agent: SmartproxyDisallow: / User-agent: ScrapflyDisallow: / User-agent: ScrapflyBotDisallow: / User-agent: ScrapflyClientDisallow: / User-agent: OctoparseDisallow: / User-agent: OctoparseBotDisallow: / User-agent: TrajectDataDisallow: / User-agent: Traject-DataDisallow: / User-agent: SerpWowDisallow: / User-agent: ScaleSERPDisallow: / User-agent: Rainforest-APIDisallow: / User-agent: BlueCart-APIDisallow: /… 44 lines not about AI bots User-agent: skrape.aiDisallow: / User-agent: SkrapeAIDisallow: / User-agent: SkrapeBotDisallow: / User-agent: FroxyDisallow: / User-agent: FroxyBotDisallow: / User-agent: MrScraperDisallow: / User-agent: Mr-ScraperDisallow: / User-agent: MrScraperBotDisallow: / User-agent: CrawlNowDisallow: / User-agent: CrawlNowBotDisallow: / User-agent: SoaxDisallow: / User-agent: SoaxBotDisallow: / User-agent: SOAXDisallow: / User-agent: KadoaDisallow: / User-agent: KadoabotDisallow: / User-agent: KadoaBotDisallow: / User-agent: ScrapeHeroDisallow: / User-agent: Scrape-HeroDisallow: / User-agent: ScrapeHeroBotDisallow: / User-agent: ExaDisallow: / User-agent: ExaBotDisallow: /… 59 lines not about AI bots User-agent: Exa-AIDisallow: / User-agent: ExaCrawlerDisallow: / User-agent: ExabotDisallow: / User-agent: *Disallow: /errorsDisallow: /interactive/Disallow: /userauth/Disallow: /ugc/Disallow: /feeds/Disallow: /services/Disallow: /facebook/Disallow: /version-info/Disallow: /longform/draft/Disallow: /story/draft/Disallow: /topic/*/smart/Disallow: /searchDisallow: /module-showcase/Disallow: /newsletter/Disallow: /blended-newsletter/Disallow: /story/nletter/Disallow: /sports/services/photos/Disallow: /optimusDisallow: /ux-trainDisallow: /story/advisory/Disallow: /.cam-tangent/Disallow: /pbd/Disallow: /gciaf/Disallow: /gcdn/gciaf/Disallow: /dcc/Disallow: /gcdn/dcc/Disallow: /dc/Disallow: /gcdn/dc/Disallow: /dcjs/Disallow: /gcdn/dcjs/Disallow: /content-queries/Disallow: /zxc/Disallow: /money/homefront/Disallow: /exp-cruiseDisallow: /exp-las-vegas2Disallow: /exp-fawDisallow: /exp-caribbeanDisallow: /exp-beachDisallow: /exp-cruise2Disallow: /yourtakeDisallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-daniel-kelly-amfam/6661213/Disallow: /story/2014/03/20/ge-cfo-challenge-david-bartlett-amway/6653003/Disallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-art-mccarthy-neulion/6655521/Disallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-david-gross-major-league-lacrosse/6646987/Disallow: /money/lookup/stocks/Disallow: /money/blueprint/archive/Disallow: /money/blueprint/transfer?pname= User-agent: AmazonAdBotAllow: / User-agent: NewsBank.comAllow: /obituaries/ Sitemap: https://www.usatoday.com/news-sitemap.xmlSitemap: https://www.usatoday.com/web-sitemap-index.xmlSitemap: https://www.usatoday.com/video-sitemap-index.xmlSitemap: https://www.usatoday.com/money/blueprint/sitemap.xmlSitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xmlSitemap: https://www.usatoday.com/online-betting/sitemap.xmlSitemap: https://www.usatoday.com/online-betting/news-sitemap.xml … 69 lines not about AI bots

Everything else we checked

Front doorLet in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.usatoday.com/ as WalledWebBot, from a residential connection in the United States
Page tags<meta name="robots" content="NOODP, NOYDIR, max-image-preview:large"> X-Robots-Tag: noarchive,nocache
Anti-bot softwareNo sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIStructured data types: Organization, WebSite, SearchAction, WebPage, CreativeWork, Product. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result.
Terms of Servicecm.usatoday.com/terms/ (found on the homepage, page text too short to be the full terms; probably rendered by JavaScript)

No sentences mention bots, scraping, automated access or AI.

Not foundNo llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

47 versions on record since 1 Jan 2023

OverallJan 2023: Overall allowedFeb 2023: Overall allowedMar 2023: Overall allowedApr 2023: Overall allowedMay 2023: Overall allowedJun 2023: Overall allowedJul 2023: Overall allowedAug 2023: Overall allowedSep 2023: Overall partialOct 2023: Overall partialNov 2023: Overall partialDec 2023: Overall partialJan 2024: Overall partialFeb 2024: Overall partialMar 2024: Overall partialApr 2024: Overall partialMay 2024: Overall partialJun 2024: Overall partialJul 2024: Overall partialAug 2024: Overall partialSep 2024: Overall partialOct 2024: Overall partialNov 2024: Overall partialDec 2024: Overall partialJan 2025: Overall partialFeb 2025: Overall partialMar 2025: Overall partialApr 2025: Overall partialMay 2025: Overall partialJun 2025: Overall partialJul 2025: Overall partialAug 2025: Overall partialSep 2025: Overall partialOct 2025: Overall partialNov 2025: Overall partialDec 2025: Overall partialJan 2026: Overall partialFeb 2026: Overall partialMar 2026: Overall partialApr 2026: Overall partialMay 2026: Overall blockedJun 2026: Overall blockedJul 2026: Overall blockedAug 2026: Overall blockedSep 2026: Overall blockedOct 2026: Overall blockedTrainingJan 2023: Training AllowedFeb 2023: Training AllowedMar 2023: Training AllowedApr 2023: Training AllowedMay 2023: Training AllowedJun 2023: Training AllowedJul 2023: Training AllowedAug 2023: Training AllowedSep 2023: Training Some blockedOct 2023: Training Some blockedNov 2023: Training Some blockedDec 2023: Training Some blockedJan 2024: Training Some blockedFeb 2024: Training Some blockedMar 2024: Training Some blockedApr 2024: Training Some blockedMay 2024: Training Some blockedJun 2024: Training Some blockedJul 2024: Training Some blockedAug 2024: Training Some blockedSep 2024: Training Some blockedOct 2024: Training Some blockedNov 2024: Training Some blockedDec 2024: Training Some blockedJan 2025: Training Some blockedFeb 2025: Training Some blockedMar 2025: Training Some blockedApr 2025: Training Some blockedMay 2025: Training Some blockedJun 2025: Training Some blockedJul 2025: Training Some blockedAug 2025: Training Some blockedSep 2025: Training Some blockedOct 2025: Training Some blockedNov 2025: Training Some blockedDec 2025: Training Some blockedJan 2026: Training Some blockedFeb 2026: Training Some blockedMar 2026: Training Some blockedApr 2026: Training Some blockedMay 2026: Training Some blockedJun 2026: Training Some blockedJul 2026: Training Some blockedAug 2026: Training Some blockedSep 2026: Training Some blockedOct 2026: Training Some blockedSearchJan 2023: Search AllowedFeb 2023: Search AllowedMar 2023: Search AllowedApr 2023: Search AllowedMay 2023: Search AllowedJun 2023: Search AllowedJul 2023: Search AllowedAug 2023: Search AllowedSep 2023: Search AllowedOct 2023: Search AllowedNov 2023: Search AllowedDec 2023: Search AllowedJan 2024: Search AllowedFeb 2024: Search AllowedMar 2024: Search AllowedApr 2024: Search Some blockedMay 2024: Search Some blockedJun 2024: Search Some blockedJul 2024: Search Some blockedAug 2024: Search Some blockedSep 2024: Search Some blockedOct 2024: Search Some blockedNov 2024: Search Some blockedDec 2024: Search Some blockedJan 2025: Search Some blockedFeb 2025: Search Some blockedMar 2025: Search Some blockedApr 2025: Search Some blockedMay 2025: Search Some blockedJun 2025: Search Some blockedJul 2025: Search Some blockedAug 2025: Search Some blockedSep 2025: Search Some blockedOct 2025: Search Some blockedNov 2025: Search Some blockedDec 2025: Search Some blockedJan 2026: Search Some blockedFeb 2026: Search Some blockedMar 2026: Search Some blockedApr 2026: Search Some blockedMay 2026: Search Some blockedJun 2026: Search Some blockedJul 2026: Search Some blockedAug 2026: Search Some blockedSep 2026: Search Some blockedOct 2026: Search Some blockedAssistantsJan 2023: Assistants AllowedFeb 2023: Assistants AllowedMar 2023: Assistants AllowedApr 2023: Assistants AllowedMay 2023: Assistants AllowedJun 2023: Assistants AllowedJul 2023: Assistants AllowedAug 2023: Assistants AllowedSep 2023: Assistants AllowedOct 2023: Assistants AllowedNov 2023: Assistants AllowedDec 2023: Assistants AllowedJan 2024: Assistants AllowedFeb 2024: Assistants AllowedMar 2024: Assistants AllowedApr 2024: Assistants Some blockedMay 2024: Assistants Some blockedJun 2024: Assistants Some blockedJul 2024: Assistants Some blockedAug 2024: Assistants Some blockedSep 2024: Assistants Some blockedOct 2024: Assistants Some blockedNov 2024: Assistants Some blockedDec 2024: Assistants Some blockedJan 2025: Assistants Some blockedFeb 2025: Assistants Some blockedMar 2025: Assistants Some blockedApr 2025: Assistants Some blockedMay 2025: Assistants Some blockedJun 2025: Assistants Some blockedJul 2025: Assistants Some blockedAug 2025: Assistants Some blockedSep 2025: Assistants Some blockedOct 2025: Assistants Some blockedNov 2025: Assistants Some blockedDec 2025: Assistants Some blockedJan 2026: Assistants Some blockedFeb 2026: Assistants Some blockedMar 2026: Assistants Some blockedApr 2026: Assistants Some blockedMay 2026: Assistants Some blockedJun 2026: Assistants BlockedJul 2026: Assistants BlockedAug 2026: Assistants BlockedSep 2026: Assistants BlockedOct 2026: Assistants BlockedBotsblockedSep 2023: 1 of 24 AI bots blockedOct 2023: 2 of 24 AI bots blockedNov 2023: 2 of 24 AI bots blockedDec 2023: 2 of 24 AI bots blockedJan 2024: 3 of 24 AI bots blockedFeb 2024: 3 of 24 AI bots blockedMar 2024: 3 of 24 AI bots blockedApr 2024: 6 of 24 AI bots blockedMay 2024: 6 of 24 AI bots blockedJun 2024: 7 of 24 AI bots blockedJul 2024: 7 of 24 AI bots blockedAug 2024: 7 of 24 AI bots blockedSep 2024: 10 of 24 AI bots blockedOct 2024: 10 of 24 AI bots blockedNov 2024: 10 of 24 AI bots blockedDec 2024: 11 of 24 AI bots blockedJan 2025: 11 of 24 AI bots blockedFeb 2025: 11 of 24 AI bots blockedMar 2025: 11 of 24 AI bots blockedApr 2025: 11 of 24 AI bots blockedMay 2025: 13 of 24 AI bots blockedJun 2025: 13 of 24 AI bots blockedJul 2025: 16 of 24 AI bots blockedAug 2025: 16 of 24 AI bots blockedSep 2025: 17 of 24 AI bots blockedOct 2025: 17 of 24 AI bots blockedNov 2025: 17 of 24 AI bots blockedDec 2025: 17 of 24 AI bots blockedJan 2026: 17 of 24 AI bots blockedFeb 2026: 17 of 24 AI bots blockedMar 2026: 17 of 24 AI bots blockedApr 2026: 17 of 24 AI bots blockedMay 2026: 19 of 24 AI bots blockedJun 2026: 22 of 24 AI bots blockedJul 2026: 22 of 24 AI bots blockedAug 2026: 22 of 24 AI bots blockedSep 2026: 22 of 24 AI bots blockedOct 2026: 22 of 24 AI bots blocked222023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Apr to Jun 2026
    More blocking Now blocked: Amzn-SearchBot, Meta-WebIndexer, Perplexity-User, Google-Agent and Amzn-User.
    Found in the Internet Archive's copies: changed between 1 Apr 2026 and 30 Jun 2026. Evidence: before · after
    Show exactly what changed
    + User-agent: Claude
    + Disallow: /
    +
    User-agent: Claude-Web
    …
    + User-agent: claudeweb
    + Disallow: /
    +
    + User-agent: Amzn-SearchBot
    + Disallow: /
    +
    + User-agent: Amzn-User
    + Disallow: /
    +
    + User-agent: AndiBot
    + Disallow: /
    +
    User-agent: cohere-ai
    …
    Disallow: /
    + Allow: /contributor-content/
    + Allow: /story/special/contributor-content/
    + Allow: /shopping/
    …
    Disallow: /
    + Allow: /contributor-content/
    + Allow: /story/special/contributor-content/
    + Allow: /shopping/
    …
    Disallow: /
    + Allow: /contributor-content/
    + Allow: /story/special/contributor-content/
    + Allow: /shopping/
    …
    + User-agent: Diffbot-user
    + Disallow: /
    +
    + User-agent: Kagibot
    + Disallow: /
    +
    User-agent: FacebookBot
    …
    Disallow: /
    + Allow: /contributor-content/
    + Allow: /story/special/contributor-content/
    + Allow: /shopping/
    …
    Disallow: /
    + Allow: /contributor-content/
    + Allow: /story/special/contributor-content/
    + Allow: /shopping/
    …
    … 531 more lines
  • Sep 2025
    More blocking Now blocked: MistralAI-User.
    Found in the Internet Archive's copies: changed between 1 Aug 2025 and 10 Sep 2025. Evidence: before · after
    Show exactly what changed
    + User-agent: ia_archiver
    + Disallow: /
    +
    + User-agent: archive.org_bot
    + Disallow: /
    +
    + User-agent: ia_archiver-web.archive.org
    + Disallow: /
    +
    + User-agent: aiHitBot
    + Disallow: /
    +
    + User-agent: Amazon Kendra
    + Disallow: /
    +
    + User-agent: Big Sur AI
    + Disallow: /
    +
    + User-agent: Brandwatch
    + Disallow: /
    +
    + User-agent: Bravest
    + Disallow: /
    +
    + User-agent: ChatGPT Operator
    + Disallow: /
    +
    + User-agent: Cotoyogi
    + Disallow: /
    +
    + User-agent: DigitalOceanGenAICrawler
    + Disallow: /
    +
    + User-agent: Echobot Bot
    + Disallow: /
    +
    + User-agent: EchoboxBot
    + Disallow: /
    +
    + User-agent: Factset_spyderbot
    + Disallow: /
    +
    + User-agent: Grok
    + Disallow: /
    +
    + User-agent: iaskspider
    + Disallow: /
    +
    + User-agent: ICC Crawler
    + Disallow: /
    +
    + User-agent: imgproxy
    + Disallow: /
    +
    + User-agent: LINER Bot
    + Disallow: /
    +
    + User-agent: MistralAI-User
    + Disallow: /
    … 29 more lines
  • Jul 2025
    More blocking Now blocked: AI2Bot, DuckAssistBot and Meta-ExternalFetcher.
    Found in the Internet Archive's copies: changed between 1 Jun 2025 and 1 Jul 2025. Evidence: before · after
    Show exactly what changed
    + User-agent: AI2Bot
    + Disallow: /
    +
    + User-agent: AhrefsBot
    + Disallow: /
    +
    + User-agent: Ai2Bot-Dolma
    + Disallow: /
    +
    + User-agent: Applebot
    + Disallow: /
    +
    + User-agent: Arquivo-web-crawler
    + Disallow: /
    +
    + User-agent: AspiegelBot
    + Disallow: /
    +
    + User-agent: AwarioRssBot
    + Disallow: /
    +
    + User-agent: AwarioSmartBot
    + Disallow: /
    +
    + User-agent: Barkrowler
    + Disallow: /
    +
    + User-agent: coccocbot-web
    + Disallow: /
    +
    + User-agent: cohere-training-data-crawler
    + Disallow: /
    +
    + User-agent: crawler4j
    + Disallow: /
    +
    + User-agent: Crawlspace
    + Disallow: /
    +
    + User-agent: dotbot
    + Disallow: /
    +
    + User-agent: DuckAssistBot
    + Disallow: /
    +
    + User-agent: FriendlyCrawler
    + Disallow: /
    +
    + User-agent: Google-CloudVertexBot
    + Disallow: /
    +
    + User-agent: https://hada.news
    + Disallow: /
    +
    + User-agent: https://www.imediaethics.org
    + Disallow: /
    +
    + User-agent: iaskspider/2.0
    + Disallow: /
    … 105 more lines
  • May 2025
    More blocking Now blocked: Claude-SearchBot and Claude-User.
    Found in the Internet Archive's copies: changed between 1 Apr 2025 and 1 May 2025. Evidence: before · after
    Show exactly what changed
    + User-agent: Claude-User
    + Disallow: /
    +
    + User-agent: Claude-SearchBot
    + Disallow: /
    +
    User-agent: *
    …
    Disallow: /gciaf/
    + Disallow: /gcdn/gciaf/
    + Disallow: /dcc/
    + Disallow: /gcdn/dcc/
    + Disallow: /dc/
    + Disallow: /gcdn/dc/
    + Disallow: /dcjs/
    + Disallow: /gcdn/dcjs/
    + Disallow: /content-queries/
    + Disallow: /zxc/
    + Disallow: /money/homefront/
    Disallow: /exp-cruise
    …
    Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xml
    - Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
    Sitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml
  • Dec 2024
    More blocking Now blocked: Bytespider.
    Found in the Internet Archive's copies: changed between 1 Nov 2024 and 1 Dec 2024. Evidence: before · after
    Show exactly what changed
    + User-agent: Bytespider
    + Disallow: /
    +
    User-agent: *
    …
    Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
    - Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml
    + Sitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml
  • Sep 2024
    More blocking Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.
    Found in the Internet Archive's copies: changed between 1 Aug 2024 and 1 Sep 2024. Evidence: before · after
    Show exactly what changed
    + User-agent: Applebot-Extended
    + Disallow: /
    +
    + User-agent: Meta-ExternalAgent
    + Disallow: /
    +
    + User-agent: meta-externalagent
    + Disallow: /
    +
    + User-agent: OAI-SearchBot
    + Disallow: /
    +
    User-agent: *
    …
    Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml
    - Sitemap: https://www.usatoday.com/betting/sitemap.xml
    - Sitemap: https://www.usatoday.com/betting/news-sitemap.xml
    + Sitemap: https://www.usatoday.com/online-betting/sitemap.xml
    + Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xml
    Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
  • Jun 2024
    More blocking Now blocked: PerplexityBot.
    Found in the Internet Archive's copies: changed between 1 May 2024 and 1 Jun 2024. Evidence: before · after
    Show exactly what changed
    + User-agent: DataForSeoBot
    + Disallow: /
    +
    + User-agent: Diffbot
    + Disallow: /
    +
    + User-agent: FacebookBot
    + Disallow: /
    +
    + User-agent: magpie-crawler
    + Disallow: /
    +
    + User-agent: NewsNow
    + Disallow: /
    +
    + User-agent: news-please
    + Disallow: /
    +
    + User-agent: omgili
    + Disallow: /
    +
    + User-agent: omgilibot
    + Disallow: /
    +
    + User-agent: peer39_crawler
    + Disallow: /
    +
    + User-agent: peer39_crawler/1.0
    + Disallow: /
    +
    + User-agent: PerplexityBot
    + Disallow: /
    +
    + User-agent: Scrapy
    + Disallow: /
    +
    + User-agent: TurnitinBot
    + Disallow: /
    +
    User-agent: *
    …
    Disallow: /money/lookup/stocks/
    + Disallow: /money/blueprint/archive/
    + Disallow: /money/blueprint/transfer?pname=
  • Apr 2024
    More blocking Now blocked: ClaudeBot, anthropic-ai, Amazonbot and ChatGPT-User.
    Found in the Internet Archive's copies: changed between 1 Mar 2024 and 1 Apr 2024. Evidence: before · after
    Show exactly what changed
    + User-agent: anthropic-ai
    + Disallow: /
    +
    + User-agent: AmazonBot
    + Disallow: /
    +
    + User-agent: Claude-Web
    + Disallow: /
    +
    + User-agent: cohere-ai
    + Disallow: /
    +
    + User-agent: ClaudeBot
    + Disallow: /
    +
    + User-agent: ChatGPT-User
    + Disallow: /
    +
    User-agent: Googlebot-News
    …
    Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
    + Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml
  • Jan 2024
    More blocking Now blocked: Google-Extended.
    Found in the Internet Archive's copies: changed between 1 Dec 2023 and 1 Jan 2024. Evidence: before · after
    Show exactly what changed
    + User-agent: Google-Extended
    + Disallow: /
    +
    User-agent: *
    …
    Sitemap: https://www.usatoday.com/betting/news-sitemap.xml
    + Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
  • Oct 2023
    More blocking Now blocked: GPTBot and CCBot.
    Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Oct 2023. Evidence: before · after
    Show exactly what changed
    + User-agent: GPTBot
    + Disallow: /
    +
    + User-agent: CCBot
    + Disallow: /
    +
    User-agent: *
    …
    Disallow: /.cam-tangent/
    + Disallow: /pbd/
    + Disallow: /gciaf/
    Disallow: /exp-cruise
    …
    Sitemap: https://www.usatoday.com/video-sitemap-index.xml
    + Sitemap: https://www.usatoday.com/money/blueprint/sitemap.xml
    + Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml
    + Sitemap: https://www.usatoday.com/betting/sitemap.xml
    + Sitemap: https://www.usatoday.com/betting/news-sitemap.xml

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.