{"domain":"usatoday.com","name":"USA Today","category":"news","subcategory":"national","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 22 of the 24 bots in total. It names every major AI bot one by one.","computed_at":"2026-10-20T03:43:15.268Z","last_checked":"2026-10-19T13:35:31.141Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.usatoday.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.usatoday.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.usatoday.com/robots.txt","hops":[{"from":"https://usatoday.com/robots.txt","to":"https://www.usatoday.com/robots.txt","status":301}],"blob":"7ac5cfceb70ab70e88bc5023e98ea9e25efa1b553481003be1d1c9ed3c465186","text":"# robots.txt file for https://www.usatoday.com/\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: AmazonBot\nDisallow: /\n\nUser-agent: Claude\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: claudeweb\nDisallow: /\n\nUser-agent: Amzn-SearchBot\nDisallow: /\n\nUser-agent: Amzn-User\nDisallow: /\n\nUser-agent: AndiBot\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: ChatGPT-User\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: Googlebot-News\nDisallow: /story/sponsor-story/\nDisallow: /picture-gallery/sponsor-story/\nDisallow: /videos/sponsor-story/\nDisallow: /longform/sponsor-story/\nDisallow: /pages/interactives/sponsor-story/\nDisallow: /interactives/sponsor-story/\nDisallow: /videos/embed/\n\nUser-agent: GPTBot\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: DataForSeoBot\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: Diffbot-user\nDisallow: /\n\nUser-agent: Kagibot\nDisallow: /\n\nUser-agent: FacebookBot\nDisallow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: NewsNow\nDisallow: /\n\nUser-agent: news-please\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: omgilibot\nDisallow: /\n\nUser-agent: peer39_crawler\nDisallow: /\n\nUser-agent: peer39_crawler/1.0\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: TurnitinBot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Meta-ExternalAgent\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: Claude-User\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: AI2Bot\nDisallow: /\n\nUser-agent: AhrefsBot\nDisallow: /\n\nUser-agent: Ai2Bot-Dolma\nDisallow: /\n\nUser-agent: Applebot\nDisallow: /\n\nUser-agent: Arquivo-web-crawler\nDisallow: /\n\nUser-agent: AspiegelBot\nDisallow: /\n\nUser-agent: AwarioRssBot\nDisallow: /\n\nUser-agent: AwarioSmartBot\nDisallow: /\n\nUser-agent: Barkrowler\nDisallow: /\n\nUser-agent: coccocbot-web\nDisallow: /\n\nUser-agent: cohere-training-data-crawler\nDisallow: /\n\nUser-agent: crawler4j\nDisallow: /\n\nUser-agent: Crawlspace\nDisallow: /\n\nUser-agent: dotbot\nDisallow: /\n\nUser-agent: DuckAssistBot\nDisallow: /\n\nUser-agent: FriendlyCrawler\nDisallow: /\n\nUser-agent: Google-CloudVertexBot\nDisallow: /\n\nUser-agent: https://hada.news\nDisallow: /\n\nUser-agent: https://www.imediaethics.org\nDisallow: /\n\nUser-agent: iaskspider/2.0\nDisallow: /\n\nUser-agent: ICC-Crawler\nDisallow: /\n\nUser-agent: icc-crawler\nDisallow: /\n\nUser-agent: ICC Crawler\nDisallow: /\n\nUser-agent: ImagesiftBot\nDisallow: /\n\nUser-agent: img2dataset\nDisallow: /\n\nUser-agent: ISSCyberRiskCrawler\nDisallow: /\n\nUser-agent: JenkersBot\nDisallow: /\n\nUser-agent: Kangaroo Bot\nDisallow: /\n\nUser-agent: LivelapBot\nDisallow: /\n\nUser-agent: MauiBot\nDisallow: /\n\nUser-agent: Meltwater\nDisallow: /\n\nUser-agent: Meta-ExternalFetcher\nDisallow: /\n\nUser-agent: MJ12bot\nDisallow: /\n\nUser-agent: Mojeek\nDisallow: /\n\nUser-agent: MojeekBot\nDisallow: /\n\nUser-agent: MoodleBot\nDisallow: /\n\nUser-agent: NeticleBot\nDisallow: /\n\nUser-agent: Netvibes\nDisallow: /\n\nUser-agent: PanguBot\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: SeekportBot\nDisallow: /\n\nUser-agent: Seekr\nDisallow: /\n\nUser-agent: SemrushBot\nDisallow: /\n\nUser-agent: SeznamBot\nDisallow: /\n\nUser-agent: Sidetrade Indexer Bot\nDisallow: /\n\nUser-agent: Timpibot\nDisallow: /\n\nUser-agent: Turnitin\nDisallow: /\n\nUser-agent: VelenPublicWebCrawler\nDisallow: /\n\nUser-agent: Webzio-Extended\nDisallow: /\n\nUser-agent: yacy\nDisallow: /\n\nUser-agent: yacybot\nDisallow: /\n\nUser-agent: Yandex\nDisallow: /\n\nUser-agent: YandexAdditional\nDisallow: /\n\nUser-agent: YandexAdditionalBot\nDisallow: /\n\nUser-agent: YandexBot\nDisallow: /\n\nUser-agent: YandexImages\nDisallow: /\n\nUser-agent: YandexNews\nDisallow: /\n\nUser-agent: Baiduspider\nDisallow: /\n\nUser-agent: Baiduspider-image\nDisallow: /\n\nUser-agent: BaiduSpider-render\nDisallow: /\n\nUser-agent: Baiduspider-video\nDisallow: /\n\nUser-agent: Sogou Web Spider\nDisallow: /\n\nUser-agent: yisouspider\nDisallow: /\n\nUser-agent: YouBot\nDisallow: /\n\nUser-agent: ia_archiver\nDisallow: /\n\nUser-agent: archive.org_bot\nDisallow: /\n\nUser-agent: ia_archiver-web.archive.org\nDisallow: /\n\nUser-agent: aiHitBot\nDisallow: /\n\nUser-agent: Amazon Kendra\nDisallow: /\n\nUser-agent: Big Sur AI\nDisallow: /\n\nUser-agent: Brandwatch\nDisallow: /\n\nUser-agent: Bravest\nDisallow: /\n\nUser-agent: ChatGPT Operator\nDisallow: /\nAllow: /branded-story/\nAllow: /contributor-content/\nAllow: /story/branded-story/\nAllow: /story/special/contributor-content/\nAllow: /money/money-management/\nAllow: /shopping/\nAllow: /story/shopping/\nAllow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/\nAllow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/\nAllow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/\n\nUser-agent: ChatGPT-PageFetcher\nDisallow: /\n\nUser-agent: OpenAI-CheckBot\nDisallow: /\n\nUser-agent: OpenAI-Explore\nDisallow: /\n\nUser-agent: OpenAI Feedback\nDisallow: /\n\nUser-agent: OpenAI-SnippetBot\nDisallow: /\n\nUser-agent: Cotoyogi\nDisallow: /\n\nUser-agent: DigitalOceanGenAICrawler\nDisallow: /\n\nUser-agent: Echobot Bot\nDisallow: /\n\nUser-agent: EchoboxBot\nDisallow: /\n\nUser-agent: Factset_spyderbot\nDisallow: /\n\nUser-agent: Grok\nDisallow: /\n\nUser-agent: iaskspider\nDisallow: /\n\nUser-agent: imgproxy\nDisallow: /\n\nUser-agent: LINER Bot\nDisallow: /\n\nUser-agent: MistralAI-User\nDisallow: /\n\nUser-agent: netEstate Imprint Crawler\nDisallow: /\n\nUser-agent: NovaAct\nDisallow: /\n\nUser-agent: QualifiedBot\nDisallow: /\n\nUser-agent: SBIntuitionsBot\nDisallow: /\n\nUser-agent: sbintuitionsbot\nDisallow: /\n\nUser-agent: SemrushBot-OCOB\nDisallow: /\n\nUser-agent: SemrushBotSwa\nDisallow: /\n\nUser-agent: Sidetrade\nDisallow: /\n\nUser-agent: TikTokSpider\nDisallow: /\n\nUser-agent: WARDBot\nDisallow: /\n\nUser-agent: TaraGroup Intelligent Bot V1\nDisallow: /\n\nUser-agent: TaraGroup Intelligent Bot\nDisallow: /\n\nUser-agent: DeepSeekBot\nDisallow: /\n\nUser-agent: ChatGLM-Spider\nDisallow: /\n\nUser-agent: GoogleOther\nDisallow: /\n\nUser-agent: Google AI\nDisallow: /\n\nUser-agent: Google-Agent\nDisallow: /\n\nUser-agent: shapbot\nDisallow: /\n\nUser-agent: FirecrawlAgent\nDisallow: /\n\nUser-agent: Crawl4AI\nDisallow: /\n\nUser-agent: imageSpider\nDisallow: /\n\nUser-agent: laion-huggingface-processor\nDisallow: /\n\nUser-agent: LCC\nDisallow: /\n\nUser-agent: Spider\nDisallow: /\n\nUser-agent: Datenbank Crawler\nDisallow: /\n\nUser-agent: Cloudflare-AutoRAG\nDisallow: /\n\nUser-agent: CloudflareBrowserRenderingCrawler\nDisallow: /\n\nUser-agent: meta-webindexer\nDisallow: /\n\nUser-agent: Bravebot\nDisallow: /\n\nUser-agent: AddSearchBot\nDisallow: /\n\nUser-agent: Anomura\nDisallow: /\n\nUser-agent: atlassian-bot\nDisallow: /\n\nUser-agent: Channel3Bot\nDisallow: /\n\nUser-agent: LinkupBot\nDisallow: /\n\nUser-agent: ZanistaBot\nDisallow: /\n\nUser-agent: Gemini-Deep-Research\nDisallow: /\n\nUser-agent: Google-NotebookLM\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: Perplexity Search Bot\nDisallow: /\n\nUser-agent: Poetbot\nDisallow: /\n\nUser-agent: Poseidon Research Crawler\nDisallow: /\n\nUser-agent: ProRataInc\nDisallow: /\n\nUser-agent: QuillBot\nDisallow: /\n\nUser-agent: Tiingo News Agent\nDisallow: /\n\nUser-agent: iboubot\nDisallow: /\n\nUser-agent: TavilyBot\nDisallow: /\n\nUser-agent: NeevaBot\nDisallow: /\n\nUser-agent: PhindBot\nDisallow: /\n\nUser-agent: AI2Bot-DeepResearchEval\nDisallow: /\n\nUser-agent: KlaviyoAIBot\nDisallow: /\n\nUser-agent: LinerBot\nDisallow: /\n\nUser-agent: Poggio-Citations\nDisallow: /\n\nUser-agent: Devin\nDisallow: /\n\nUser-agent: bigsur.ai\nDisallow: /\n\nUser-agent: ChatGPT Agent\nDisallow: /\n\nUser-agent: GoogleAgent-Mariner\nDisallow: /\n\nUser-agent: AmazonBuyForMe\nDisallow: /\n\nUser-agent: Manus-User\nDisallow: /\n\nUser-agent: manus-user\nDisallow: /\n\nUser-agent: TwinAgent\nDisallow: /\n\nUser-agent: AudigentAdBot\nDisallow: /\n\nUser-agent: AudigentBot\nDisallow: /\n\nUser-agent: JinaAI\nDisallow: /\n\nUser-agent: Jina-AI\nDisallow: /\n\nUser-agent: JinaBot\nDisallow: /\n\nUser-agent: Jina-Reader\nDisallow: /\n\nUser-agent: BrightData\nDisallow: /\n\nUser-agent: Bright Data\nDisallow: /\n\nUser-agent: BrightBot\nDisallow: /\n\nUser-agent: SerpApi\nDisallow: /\n\nUser-agent: SerpApiBot\nDisallow: /\n\nUser-agent: Apify\nDisallow: /\n\nUser-agent: ApifyBot\nDisallow: /\n\nUser-agent: Apify-HTTP-Client\nDisallow: /\n\nUser-agent: ScrapeGraphAI\nDisallow: /\n\nUser-agent: scrapegraph\nDisallow: /\n\nUser-agent: Scrapeless\nDisallow: /\n\nUser-agent: ScrapelessBot\nDisallow: /\n\nUser-agent: ZenRows\nDisallow: /\n\nUser-agent: ZenRowsBot\nDisallow: /\n\nUser-agent: PromptCloud\nDisallow: /\n\nUser-agent: PromptCloudBot\nDisallow: /\n\nUser-agent: Nimbleway\nDisallow: /\n\nUser-agent: NimbleCrawler\nDisallow: /\n\nUser-agent: NimblewayBot\nDisallow: /\n\nUser-agent: Oxylabs\nDisallow: /\n\nUser-agent: OxylabsBot\nDisallow: /\n\nUser-agent: Zyte\nDisallow: /\n\nUser-agent: ZyteBot\nDisallow: /\n\nUser-agent: Scrapinghub\nDisallow: /\n\nUser-agent: ScraperAPI\nDisallow: /\n\nUser-agent: Scraper-API\nDisallow: /\n\nUser-agent: ScrapingBee\nDisallow: /\n\nUser-agent: ScrapingBeeBot\nDisallow: /\n\nUser-agent: Scrapingdog\nDisallow: /\n\nUser-agent: ScrapingdogBot\nDisallow: /\n\nUser-agent: Grepsr\nDisallow: /\n\nUser-agent: GrepsrBot\nDisallow: /\n\nUser-agent: GrepsrCrawler\nDisallow: /\n\nUser-agent: SpiderBot\nDisallow: /\n\nUser-agent: spider-cloud\nDisallow: /\n\nUser-agent: SpiderCloud\nDisallow: /\n\nUser-agent: Hyperbrowser\nDisallow: /\n\nUser-agent: HyperbrowserBot\nDisallow: /\n\nUser-agent: HyperAgent\nDisallow: /\n\nUser-agent: Scrapestack\nDisallow: /\n\nUser-agent: ScrapestackBot\nDisallow: /\n\nUser-agent: WebScraping.AI\nDisallow: /\n\nUser-agent: WebScrapingAI\nDisallow: /\n\nUser-agent: webscraping-ai\nDisallow: /\n\nUser-agent: ForageAI\nDisallow: /\n\nUser-agent: Forage-AI\nDisallow: /\n\nUser-agent: ForageBot\nDisallow: /\n\nUser-agent: DumplingAI\nDisallow: /\n\nUser-agent: Dumpling-AI\nDisallow: /\n\nUser-agent: DumplingBot\nDisallow: /\n\nUser-agent: Import.io\nDisallow: /\n\nUser-agent: ImportIO\nDisallow: /\n\nUser-agent: importio\nDisallow: /\n\nUser-agent: BrowseAI\nDisallow: /\n\nUser-agent: Browse-AI\nDisallow: /\n\nUser-agent: BrowseAIBot\nDisallow: /\n\nUser-agent: Decodo\nDisallow: /\n\nUser-agent: DecodoBot\nDisallow: /\n\nUser-agent: Smartproxy\nDisallow: /\n\nUser-agent: Scrapfly\nDisallow: /\n\nUser-agent: ScrapflyBot\nDisallow: /\n\nUser-agent: ScrapflyClient\nDisallow: /\n\nUser-agent: Octoparse\nDisallow: /\n\nUser-agent: OctoparseBot\nDisallow: /\n\nUser-agent: TrajectData\nDisallow: /\n\nUser-agent: Traject-Data\nDisallow: /\n\nUser-agent: SerpWow\nDisallow: /\n\nUser-agent: ScaleSERP\nDisallow: /\n\nUser-agent: Rainforest-API\nDisallow: /\n\nUser-agent: BlueCart-API\nDisallow: /\n\nUser-agent: skrape.ai\nDisallow: /\n\nUser-agent: SkrapeAI\nDisallow: /\n\nUser-agent: SkrapeBot\nDisallow: /\n\nUser-agent: Froxy\nDisallow: /\n\nUser-agent: FroxyBot\nDisallow: /\n\nUser-agent: MrScraper\nDisallow: /\n\nUser-agent: Mr-Scraper\nDisallow: /\n\nUser-agent: MrScraperBot\nDisallow: /\n\nUser-agent: CrawlNow\nDisallow: /\n\nUser-agent: CrawlNowBot\nDisallow: /\n\nUser-agent: Soax\nDisallow: /\n\nUser-agent: SoaxBot\nDisallow: /\n\nUser-agent: SOAX\nDisallow: /\n\nUser-agent: Kadoa\nDisallow: /\n\nUser-agent: Kadoabot\nDisallow: /\n\nUser-agent: KadoaBot\nDisallow: /\n\nUser-agent: ScrapeHero\nDisallow: /\n\nUser-agent: Scrape-Hero\nDisallow: /\n\nUser-agent: ScrapeHeroBot\nDisallow: /\n\nUser-agent: Exa\nDisallow: /\n\nUser-agent: ExaBot\nDisallow: /\n\nUser-agent: Exa-AI\nDisallow: /\n\nUser-agent: ExaCrawler\nDisallow: /\n\nUser-agent: Exabot\nDisallow: /\n\nUser-agent: *\nDisallow: /errors\nDisallow: /interactive/\nDisallow: /userauth/\nDisallow: /ugc/\nDisallow: /feeds/\nDisallow: /services/\nDisallow: /facebook/\nDisallow: /version-info/\nDisallow: /longform/draft/\nDisallow: /story/draft/\nDisallow: /topic/*/smart/\nDisallow: /search\nDisallow: /module-showcase/\nDisallow: /newsletter/\nDisallow: /blended-newsletter/\nDisallow: /story/nletter/\nDisallow: /sports/services/photos/\nDisallow: /optimus\nDisallow: /ux-train\nDisallow: /story/advisory/\nDisallow: /.cam-tangent/\nDisallow: /pbd/\nDisallow: /gciaf/\nDisallow: /gcdn/gciaf/\nDisallow: /dcc/\nDisallow: /gcdn/dcc/\nDisallow: /dc/\nDisallow: /gcdn/dc/\nDisallow: /dcjs/\nDisallow: /gcdn/dcjs/\nDisallow: /content-queries/\nDisallow: /zxc/\nDisallow: /money/homefront/\nDisallow: /exp-cruise\nDisallow: /exp-las-vegas2\nDisallow: /exp-faw\nDisallow: /exp-caribbean\nDisallow: /exp-beach\nDisallow: /exp-cruise2\nDisallow: /yourtake\nDisallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-daniel-kelly-amfam/6661213/\nDisallow: /story/2014/03/20/ge-cfo-challenge-david-bartlett-amway/6653003/\nDisallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-art-mccarthy-neulion/6655521/\nDisallow: /story/sports/ncaab/2014/03/20/ge-cfo-challenge-david-gross-major-league-lacrosse/6646987/\nDisallow: /money/lookup/stocks/\nDisallow: /money/blueprint/archive/\nDisallow: /money/blueprint/transfer?pname=\n\nUser-agent: AmazonAdBot\nAllow: /\n\nUser-agent: NewsBank.com\nAllow: /obituaries/\n\nSitemap: https://www.usatoday.com/news-sitemap.xml\nSitemap: https://www.usatoday.com/web-sitemap-index.xml\nSitemap: https://www.usatoday.com/video-sitemap-index.xml\nSitemap: https://www.usatoday.com/money/blueprint/sitemap.xml\nSitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml\nSitemap: https://www.usatoday.com/online-betting/sitemap.xml\nSitemap: https://www.usatoday.com/online-betting/news-sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["anthropic-ai","AmazonBot","Claude","Claude-Web","claudeweb","Amzn-SearchBot","Amzn-User","AndiBot","cohere-ai","ClaudeBot","ChatGPT-User","Googlebot-News","GPTBot","CCBot","Google-Extended","DataForSeoBot","Diffbot","Diffbot-user","Kagibot","FacebookBot","magpie-crawler","NewsNow","news-please","omgili","omgilibot","peer39_crawler","peer39_crawler/1.0","PerplexityBot","Scrapy","TurnitinBot","Applebot-Extended","Meta-ExternalAgent","meta-externalagent","OAI-SearchBot","Bytespider","Claude-User","Claude-SearchBot","AI2Bot","AhrefsBot","Ai2Bot-Dolma","Applebot","Arquivo-web-crawler","AspiegelBot","AwarioRssBot","AwarioSmartBot","Barkrowler","coccocbot-web","cohere-training-data-crawler","crawler4j","Crawlspace","dotbot","DuckAssistBot","FriendlyCrawler","Google-CloudVertexBot","https://hada.news","https://www.imediaethics.org","iaskspider/2.0","ICC-Crawler","icc-crawler","ICC Crawler","ImagesiftBot","img2dataset","ISSCyberRiskCrawler","JenkersBot","Kangaroo Bot","LivelapBot","MauiBot","Meltwater","Meta-ExternalFetcher","MJ12bot","Mojeek","MojeekBot","MoodleBot","NeticleBot","Netvibes","PanguBot","PetalBot","SeekportBot","Seekr","SemrushBot","SeznamBot","Sidetrade Indexer Bot","Timpibot","Turnitin","VelenPublicWebCrawler","Webzio-Extended","yacy","yacybot","Yandex","YandexAdditional","YandexAdditionalBot","YandexBot","YandexImages","YandexNews","Baiduspider","Baiduspider-image","BaiduSpider-render","Baiduspider-video","Sogou Web Spider","yisouspider","YouBot","ia_archiver","archive.org_bot","ia_archiver-web.archive.org","aiHitBot","Amazon Kendra","Big Sur AI","Brandwatch","Bravest","ChatGPT Operator","ChatGPT-PageFetcher","OpenAI-CheckBot","OpenAI-Explore","OpenAI Feedback","OpenAI-SnippetBot","Cotoyogi","DigitalOceanGenAICrawler","Echobot Bot","EchoboxBot","Factset_spyderbot","Grok","iaskspider","imgproxy","LINER Bot","MistralAI-User","netEstate Imprint Crawler","NovaAct","QualifiedBot","SBIntuitionsBot","sbintuitionsbot","SemrushBot-OCOB","SemrushBotSwa","Sidetrade","TikTokSpider","WARDBot","TaraGroup Intelligent Bot V1","TaraGroup Intelligent Bot","DeepSeekBot","ChatGLM-Spider","GoogleOther","Google AI","Google-Agent","shapbot","FirecrawlAgent","Crawl4AI","imageSpider","laion-huggingface-processor","LCC","Spider","Datenbank Crawler","Cloudflare-AutoRAG","CloudflareBrowserRenderingCrawler","meta-webindexer","Bravebot","AddSearchBot","Anomura","atlassian-bot","Channel3Bot","LinkupBot","ZanistaBot","Gemini-Deep-Research","Google-NotebookLM","Perplexity-User","Perplexity Search Bot","Poetbot","Poseidon Research Crawler","ProRataInc","QuillBot","Tiingo News Agent","iboubot","TavilyBot","NeevaBot","PhindBot","AI2Bot-DeepResearchEval","KlaviyoAIBot","LinerBot","Poggio-Citations","Devin","bigsur.ai","ChatGPT Agent","GoogleAgent-Mariner","AmazonBuyForMe","Manus-User","manus-user","TwinAgent","AudigentAdBot","AudigentBot","JinaAI","Jina-AI","JinaBot","Jina-Reader","BrightData","Bright Data","BrightBot","SerpApi","SerpApiBot","Apify","ApifyBot","Apify-HTTP-Client","ScrapeGraphAI","scrapegraph","Scrapeless","ScrapelessBot","ZenRows","ZenRowsBot","PromptCloud","PromptCloudBot","Nimbleway","NimbleCrawler","NimblewayBot","Oxylabs","OxylabsBot","Zyte","ZyteBot","Scrapinghub","ScraperAPI","Scraper-API","ScrapingBee","ScrapingBeeBot","Scrapingdog","ScrapingdogBot","Grepsr","GrepsrBot","GrepsrCrawler","SpiderBot","spider-cloud","SpiderCloud","Hyperbrowser","HyperbrowserBot","HyperAgent","Scrapestack","ScrapestackBot","WebScraping.AI","WebScrapingAI","webscraping-ai","ForageAI","Forage-AI","ForageBot","DumplingAI","Dumpling-AI","DumplingBot","Import.io","ImportIO","importio","BrowseAI","Browse-AI","BrowseAIBot","Decodo","DecodoBot","Smartproxy","Scrapfly","ScrapflyBot","ScrapflyClient","Octoparse","OctoparseBot","TrajectData","Traject-Data","SerpWow","ScaleSERP","Rainforest-API","BlueCart-API","skrape.ai","SkrapeAI","SkrapeBot","Froxy","FroxyBot","MrScraper","Mr-Scraper","MrScraperBot","CrawlNow","CrawlNowBot","Soax","SoaxBot","SOAX","Kadoa","Kadoabot","KadoaBot","ScrapeHero","Scrape-Hero","ScrapeHeroBot","Exa","ExaBot","Exa-AI","ExaCrawler","Exabot","*","AmazonAdBot","NewsBank.com"],"lines":986},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":56},"any":{"allowed":false,"line":56},"search":{"allowed":false,"line":56},"cart":{"allowed":false,"line":56},"checkout":{"allowed":false,"line":56},"account":{"allowed":false,"line":56}},"disallows":1,"lines":[55,66],"groupRanges":[[55,66]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":31},"any":{"allowed":false,"line":31},"search":{"allowed":false,"line":31},"cart":{"allowed":false,"line":31},"checkout":{"allowed":false,"line":31},"account":{"allowed":false,"line":31}},"disallows":1,"lines":[30,31],"groupRanges":[[30,31]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":72},"any":{"allowed":false,"line":72},"search":{"allowed":false,"line":72},"cart":{"allowed":false,"line":72},"checkout":{"allowed":false,"line":72},"account":{"allowed":false,"line":72}},"disallows":1,"lines":[71,82],"groupRanges":[[71,82]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":140},"any":{"allowed":false,"line":140},"search":{"allowed":false,"line":140},"cart":{"allowed":false,"line":140},"checkout":{"allowed":false,"line":140},"account":{"allowed":false,"line":140}},"disallows":1,"lines":[139,140],"groupRanges":[[139,140]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":143},"any":{"allowed":false,"line":143},"search":{"allowed":false,"line":143},"cart":{"allowed":false,"line":143},"checkout":{"allowed":false,"line":143},"account":{"allowed":false,"line":143}},"disallows":2,"lines":[142,143,145,146],"groupRanges":[[142,143],[145,146]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":162},"any":{"allowed":false,"line":162},"search":{"allowed":false,"line":162},"cart":{"allowed":false,"line":162},"checkout":{"allowed":false,"line":162},"account":{"allowed":false,"line":162}},"disallows":1,"lines":[161,162],"groupRanges":[[161,162]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":936},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":47,"lines":[924,971],"groupRanges":[[924,971]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":171},"any":{"allowed":false,"line":171},"search":{"allowed":false,"line":171},"cart":{"allowed":false,"line":171},"checkout":{"allowed":false,"line":171},"account":{"allowed":false,"line":171}},"disallows":1,"lines":[170,171],"groupRanges":[[170,171]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":4},"any":{"allowed":false,"line":4},"search":{"allowed":false,"line":4},"cart":{"allowed":false,"line":4},"checkout":{"allowed":false,"line":4},"account":{"allowed":false,"line":4}},"disallows":1,"lines":[3,4],"groupRanges":[[3,4]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":149},"any":{"allowed":false,"line":149},"search":{"allowed":false,"line":149},"cart":{"allowed":false,"line":149},"checkout":{"allowed":false,"line":149},"account":{"allowed":false,"line":149}},"disallows":1,"lines":[148,159],"groupRanges":[[148,159]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":168},"any":{"allowed":false,"line":168},"search":{"allowed":false,"line":168},"cart":{"allowed":false,"line":168},"checkout":{"allowed":false,"line":168},"account":{"allowed":false,"line":168}},"disallows":1,"lines":[167,168],"groupRanges":[[167,168]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":121},"any":{"allowed":false,"line":121},"search":{"allowed":false,"line":121},"cart":{"allowed":false,"line":121},"checkout":{"allowed":false,"line":121},"account":{"allowed":false,"line":121}},"disallows":1,"lines":[120,131],"groupRanges":[[120,131]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":7},"any":{"allowed":false,"line":7},"search":{"allowed":false,"line":7},"cart":{"allowed":false,"line":7},"checkout":{"allowed":false,"line":7},"account":{"allowed":false,"line":7}},"disallows":1,"lines":[6,7],"groupRanges":[[6,7]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":19},"any":{"allowed":false,"line":19},"search":{"allowed":false,"line":19},"cart":{"allowed":false,"line":19},"checkout":{"allowed":false,"line":19},"account":{"allowed":false,"line":19}},"disallows":1,"lines":[18,19],"groupRanges":[[18,19]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":213},"any":{"allowed":false,"line":213},"search":{"allowed":false,"line":213},"cart":{"allowed":false,"line":213},"checkout":{"allowed":false,"line":213},"account":{"allowed":false,"line":213}},"disallows":1,"lines":[212,213],"groupRanges":[[212,213]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":936},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":47,"lines":[924,971],"groupRanges":[[924,971]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":526},"any":{"allowed":false,"line":526},"search":{"allowed":false,"line":526},"cart":{"allowed":false,"line":526},"checkout":{"allowed":false,"line":526},"account":{"allowed":false,"line":526}},"disallows":1,"lines":[525,526],"groupRanges":[[525,526]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":34},"any":{"allowed":false,"line":34},"search":{"allowed":false,"line":34},"cart":{"allowed":false,"line":34},"checkout":{"allowed":false,"line":34},"account":{"allowed":false,"line":34}},"disallows":1,"lines":[33,44],"groupRanges":[[33,44]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":165},"any":{"allowed":false,"line":165},"search":{"allowed":false,"line":165},"cart":{"allowed":false,"line":165},"checkout":{"allowed":false,"line":165},"account":{"allowed":false,"line":165}},"disallows":1,"lines":[164,165],"groupRanges":[[164,165]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":556},"any":{"allowed":false,"line":556},"search":{"allowed":false,"line":556},"cart":{"allowed":false,"line":556},"checkout":{"allowed":false,"line":556},"account":{"allowed":false,"line":556}},"disallows":1,"lines":[555,556],"groupRanges":[[555,556]]},{"token":"Google-Agent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":493},"any":{"allowed":false,"line":493},"search":{"allowed":false,"line":493},"cart":{"allowed":false,"line":493},"checkout":{"allowed":false,"line":493},"account":{"allowed":false,"line":493}},"disallows":1,"lines":[492,493],"groupRanges":[[492,493]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":264},"any":{"allowed":false,"line":264},"search":{"allowed":false,"line":264},"cart":{"allowed":false,"line":264},"checkout":{"allowed":false,"line":264},"account":{"allowed":false,"line":264}},"disallows":1,"lines":[263,264],"groupRanges":[[263,264]]},{"token":"Amzn-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":22},"any":{"allowed":false,"line":22},"search":{"allowed":false,"line":22},"cart":{"allowed":false,"line":22},"checkout":{"allowed":false,"line":22},"account":{"allowed":false,"line":22}},"disallows":1,"lines":[21,22],"groupRanges":[[21,22]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":442},"any":{"allowed":false,"line":442},"search":{"allowed":false,"line":442},"cart":{"allowed":false,"line":442},"checkout":{"allowed":false,"line":442},"account":{"allowed":false,"line":442}},"disallows":1,"lines":[441,442],"groupRanges":[[441,442]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":936},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":47,"lines":[924,971],"groupRanges":[[924,971]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":936},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":47,"lines":[924,971],"groupRanges":[[924,971]]}],"lanes":{"training":{"level":"mixed","none":8,"full":1,"restricted":0,"n":9},"search":{"level":"mixed","none":7,"full":1,"restricted":0,"n":8},"user_directed":{"level":"none","none":7,"full":0,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.usatoday.com/llms.txt/"},"ai_txt":{"state":"absent","http":404,"url":"https://www.usatoday.com/ai.txt/"},"tdmrep":{"state":"absent","http":404,"url":"https://www.usatoday.com/.well-known/tdmrep.json/"},"meta":{"title":"USA TODAY - Breaking News and Latest News Today","robotsMeta":[{"name":"robots","content":"NOODP, NOYDIR, max-image-preview:large"}],"noai":[],"noindex":false,"jsonLdTypes":["Organization","WebSite","SearchAction","WebPage","CreativeWork","Product"],"platform":null,"textRatio":0.017,"llmsLink":false},"x_robots_tag":"noarchive,nocache","tos":{"url":"https://cm.usatoday.com/terms/","source":"discovered","http":200,"hash":"043234eb2c4df8641d3b7f6ef7f5a475f412e65d161ba46fabaeea866eb6fb1a","chars":264,"thin":true,"clauses":[],"door":{"door":"open","http":200}},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":13},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-05","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":19},{"m":"2026-06","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":22},{"m":"2026-07","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":22},{"m":"2026-08","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":22},{"m":"2026-09","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":22},{"m":"2026-10","s":"blocked","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":22}]},"events":[{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-04-01","2026-06-30"],"summary":"Now blocked: Amzn-SearchBot, Meta-WebIndexer, Perplexity-User, Google-Agent and Amzn-User.","dir":"up","up":["Amzn-SearchBot","Meta-WebIndexer","Perplexity-User","Google-Agent","Amzn-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"blocked","evidence":["https://web.archive.org/web/20260401225033/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20260630204214/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: claudeweb"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amzn-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amzn-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AndiBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: cohere-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /contributor-content/"},{"t":"add","s":"Allow: /story/special/contributor-content/"},{"t":"add","s":"Allow: /shopping/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /contributor-content/"},{"t":"add","s":"Allow: /story/special/contributor-content/"},{"t":"add","s":"Allow: /shopping/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /contributor-content/"},{"t":"add","s":"Allow: /story/special/contributor-content/"},{"t":"add","s":"Allow: /shopping/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Diffbot-user"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Kagibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /contributor-content/"},{"t":"add","s":"Allow: /story/special/contributor-content/"},{"t":"add","s":"Allow: /shopping/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /contributor-content/"},{"t":"add","s":"Allow: /story/special/contributor-content/"},{"t":"add","s":"Allow: /shopping/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"gap","s":"… 531 more lines"}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-09-10"],"summary":"Now blocked: MistralAI-User.","dir":"up","up":["MistralAI-User"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250801013743/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20250910211719/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: ia_archiver"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: archive.org_bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ia_archiver-web.archive.org"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: aiHitBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amazon Kendra"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Big Sur AI"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Brandwatch"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bravest"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT Operator"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Cotoyogi"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DigitalOceanGenAICrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Echobot Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: EchoboxBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Factset_spyderbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Grok"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: iaskspider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ICC Crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: imgproxy"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: LINER Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: MistralAI-User"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 29 more lines"}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2025-07","precision":"month","source":"archive","provisional":false,"between":["2025-06-01","2025-07-01"],"summary":"Now blocked: AI2Bot, DuckAssistBot and Meta-ExternalFetcher.","dir":"up","up":["AI2Bot","DuckAssistBot","Meta-ExternalFetcher"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250601013730/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20250701024336/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AhrefsBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Arquivo-web-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AspiegelBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AwarioRssBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AwarioSmartBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Barkrowler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: coccocbot-web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-training-data-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: crawler4j"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Crawlspace"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: dotbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FriendlyCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-CloudVertexBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: https://hada.news"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: https://www.imediaethics.org"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: iaskspider/2.0"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 105 more lines"}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2025-05","precision":"month","source":"archive","provisional":false,"between":["2025-04-01","2025-05-01"],"summary":"Now blocked: Claude-SearchBot and Claude-User.","dir":"up","up":["Claude-SearchBot","Claude-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250401004320/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20250501152555/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /gciaf/"},{"t":"add","s":"Disallow: /gcdn/gciaf/"},{"t":"add","s":"Disallow: /dcc/"},{"t":"add","s":"Disallow: /gcdn/dcc/"},{"t":"add","s":"Disallow: /dc/"},{"t":"add","s":"Disallow: /gcdn/dc/"},{"t":"add","s":"Disallow: /dcjs/"},{"t":"add","s":"Disallow: /gcdn/dcjs/"},{"t":"add","s":"Disallow: /content-queries/"},{"t":"add","s":"Disallow: /zxc/"},{"t":"add","s":"Disallow: /money/homefront/"},{"t":"ctx","s":"Disallow: /exp-cruise"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xml"},{"t":"del","s":"Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml"}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2024-12","precision":"month","source":"archive","provisional":false,"between":["2024-11-01","2024-12-01"],"summary":"Now blocked: Bytespider.","dir":"up","up":["Bytespider"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241101004820/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20241201002444/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml"},{"t":"ctx","s":""}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2024-09","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-09-01"],"summary":"Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.","dir":"up","up":["Applebot-Extended","Meta-ExternalAgent","OAI-SearchBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801010507/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20240901013121/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Meta-ExternalAgent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml"},{"t":"del","s":"Sitemap: https://www.usatoday.com/betting/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.usatoday.com/betting/news-sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/online-betting/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xml"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml"}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2024-06","precision":"month","source":"archive","provisional":false,"between":["2024-05-01","2024-06-01"],"summary":"Now blocked: PerplexityBot.","dir":"up","up":["PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240501052302/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20240601002441/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: NewsNow"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: news-please"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: peer39_crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: peer39_crawler/1.0"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Scrapy"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: TurnitinBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /money/lookup/stocks/"},{"t":"add","s":"Disallow: /money/blueprint/archive/"},{"t":"add","s":"Disallow: /money/blueprint/transfer?pname="},{"t":"ctx","s":""}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2024-04","precision":"month","source":"archive","provisional":false,"between":["2024-03-01","2024-04-01"],"summary":"Now blocked: ClaudeBot, anthropic-ai, Amazonbot and ChatGPT-User.","dir":"up","up":["ClaudeBot","anthropic-ai","Amazonbot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240301000539/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20240401004158/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AmazonBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Googlebot-News"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml"},{"t":"ctx","s":""}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2024-01","precision":"month","source":"archive","provisional":false,"between":["2023-12-01","2024-01-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231201000112/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20240101001154/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/betting/news-sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml"},{"t":"ctx","s":""}]},{"domain":"usatoday.com","name":"USA Today","category":"news","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-10-01"],"summary":"Now blocked: GPTBot and CCBot.","dir":"up","up":["GPTBot","CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801000146/https://www.usatoday.com/robots.txt","https://web.archive.org/web/20231001001255/https://www.usatoday.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /.cam-tangent/"},{"t":"add","s":"Disallow: /pbd/"},{"t":"add","s":"Disallow: /gciaf/"},{"t":"ctx","s":"Disallow: /exp-cruise"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.usatoday.com/video-sitemap-index.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/money/blueprint/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/betting/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.usatoday.com/betting/news-sitemap.xml"},{"t":"ctx","s":""}]}]}