All sites / News and publishing
Reuters
reuters.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 20 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. OAI-SearchBot and ChatGPT-User can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Blocked
9 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
6 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
5 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Blocked | |||
| GPTBotOpenAI | Blocked | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 124 | Says yes |
| CCBotCommon Crawl | Blocked | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Blocked | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Blocked | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Allowed | Named, line 11 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Blocked | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Named, line 11 | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Allowed | Named, line 11 | Says not always |
| Claude-UserAnthropic | Blocked | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Blocked | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Blocked | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Named, line 11 | Says not always |
| MistralAI-UserMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Named | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Applebot | Has rules | line 18 | Documented by its operator |
| GoogleOther | Has rules | line 43 | Documented by its operator |
| meta-externalads | Has rules | line 56 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# NOTICE: Collection of content, data and/or information from reuters.com through# automated means is prohibited unless you have prior written consent from Reuters # and may only be conducted for the purposes explicitly described in such permission. ## The terms and conditions for reuters.com may be found at # https://www.reuters.com/info-pages/terms-of-use/ ## If you would like to seek permission to license the content and/or other materials # of Reuters, please contact https://reutersagency.com/contact-us/enquire/… 9 lines not about AI bots User-agent: AASA-BotUser-agent: ADmantXUser-agent: AdmiralBotUser-agent: AdsBot-GoogleUser-agent: AdsBot-Google-MobileUser-agent: AmazonAdBotUser-agent: Amzn-UserUser-agent: ApplebotUser-agent: AppleNewsBotUser-agent: BingbotUser-agent: BingPreviewUser-agent: BrightEdgeUser-agent: BrightEdgeOnCrawlUser-agent: CensysInspectUser-agent: ChatGPT-UserUser-agent: CisionUser-agent: ClickagyUser-agent: ConcertUser-agent: ContextualBotUser-agent: CriteoBotUser-agent: DatadogSyntheticsUser-agent: datadome-pageprotect-scannerUser-agent: DiscordbotUser-agent: doubleverifyUser-agent: ElevenlabsBotUser-agent: Embedly User-agent: facebookexternalhitUser-agent: GooglebotUser-agent: Googlebot SmartphoneUser-agent: Googlebot-NewsUser-agent: Google-Display-Ads-BotUser-agent: Google-InspectionToolUser-agent: GoogleOtherUser-agent: Google-Read-AloudUser-agent: Google-SafetyUser-agent: Google-Site-VerificationUser-agent: GTmetrixUser-agent: GumGumBotUser-agent: ias_crawlerUser-agent: IframelyUser-agent: leikiUser-agent: LinkedInBotUser-agent: LinkTigerUser-agent: MantisbotUser-agent: Mediapartners-GoogleUser-agent: meta-externaladsUser-agent: meta-webindexerUser-agent: MicrosoftPreviewUser-agent: MJ12bot User-agent: MoreoverUser-agent: msnbot User-agent: NFBNewslineRobotUser-agent: OAI-SearchBotUser-agent: OncrawlUser-agent: Opebot-vUser-agent: OpointUser-agent: OptimizerUser-agent: outbrain User-agent: PinterestbotUser-agent: PrerenderUser-agent: Proofpoint User-agent: proximicUser-agent: PubMatic Crawler BotUser-agent: QuantcastbotUser-agent: QwantbotUser-agent: Reuters SEO Screaming Frog Spider 007User-agent: Reuters-NAUWIUser-agent: Scom-Crawler-For-ReutersUser-agent: SinceraSyntheticUserUser-agent: SlurpUser-agent: SmartologyBotUser-agent: snewsUser-agent: SocialFlowUser-agent: StatusCakeUser-agent: Storebot-GoogleUser-agent: StripebotUser-agent: TTD-ContentUser-agent: TwitterbotUser-agent: URLDefenseUser-agent: VerityUser-agent: vuln_scan_by_trustedsite_com_halo_securityUser-agent: WISEbotUser-agent: Xenu Link SleuthUser-agent: Yahoo Link PreviewUser-agent: Yahoo! JAPANUser-agent: YahooMailProxyDisallow: /finance/stocks/optionDisallow: /finance/stocks/financialHighlightsDisallow: /searchDisallow: /site-search/Disallow: /betaDisallow: /designtechDisallow: /featured-optimizeDisallow: /energy-testDisallow: /article/betaDisallow: /sponsored/previewcampaignDisallow: /sponsored/previewarticleDisallow: /test/Disallow: /news/archive/commentaryDisallow: /brandfeatures/venture-capitalDisallow: /assets/siteindexDisallow: /article/api/Disallow: /practical-law-the-journal/search/Disallow: /pf/api/Disallow: /fr/Disallow: /it/Disallow: /es/Disallow: /pt/Disallow: /de/Disallow: /latam/Disallow: /account/subscribe/paymentDisallow: /*/site-search/ User-agent: Applebot-ExtendedDisallow: / # Block all other botsUser-agent: *Allow: /plus/Disallow: / SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-jp-index/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/plus/sitemap-index.xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xmlSITEMAP: https://www.reuters.com/graphics/sitemap.xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xmlSITEMAP: https://www.reuters.com/static/video-sitemap/jp/sitemap_video_index.xmlSITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xmlSITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml… 19 lines not about AI bots
Everything else we checked
| Front door | We did not visit: the rules ask our bot to stay out. |
|---|---|
| llms.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| ai.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| TDMRep | Disallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Amazon CloudFront.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Sep 2026Signals Named Applebot-Extended explicitly, without changing what it may reach.Found in the Internet Archive's copies: changed between 5 Aug 2026 and 1 Sep 2026. Evidence: before · after
Show exactly what changed
User-agent: ADmantX+ User-agent: AdmiralBotUser-agent: AdsBot-Google…User-agent: doubleverify- User-agent: DuckDuckBotUser-agent: ElevenlabsBot…Disallow: /latam/- Disallow: /account/subscribe/payment/+ Disallow: /jp/+ Disallow: /account/subscribe/payment+ User-agent: Applebot-Extended+ Disallow: /+# Block all other bots…SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml+ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml+ SITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml - Apr to Jun 2026More blocking Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.Found in the Internet Archive's copies: changed between 1 Apr 2026 and 5 Jun 2026. Evidence: before · after
Show exactly what changed
- # robots.txt for www.reuters.com-- User-agent: AI2Bot- User-agent: AI2Bot-Dolma- User-agent: Alibaba.Security.Heimdall- User-agent: AliyunSecBot/Aliyun- User-agent: anthropic-ai- User-agent: Applebot-Extended- User-agent: archive.org_bot- User-agent: bedrockbot- User-agent: Bytespider- User-agent: CCBot- User-agent: ChatGLM-Spider- User-agent: Claude-SearchBot- User-agent: Claude-Web- User-agent: ClaudeBot- User-agent: cohere-ai- User-agent: cohere-training-data-crawler- User-agent: Crawlspace- User-agent: DeepSeekBot- User-agent: Diffbot- User-agent: FacebookBot- User-agent: Feedfetcher-Google- User-agent: Gemini-Deep-Research- User-agent: Google-CloudVertexBot- User-agent: Google-Extended+ User-agent: AASA-Bot+ User-agent: ADmantX+ User-agent: AdsBot-Google+ User-agent: AdsBot-Google-Mobile+ User-agent: AmazonAdBot+ User-agent: Amzn-User+ User-agent: Applebot+ User-agent: AppleNewsBot+ User-agent: Bingbot+ User-agent: BingPreview+ User-agent: BrightEdge+ User-agent: BrightEdgeOnCrawl+ User-agent: CensysInspect+ User-agent: ChatGPT-User+ User-agent: Cision+ User-agent: Clickagy+ User-agent: Concert+ User-agent: ContextualBot+ User-agent: CriteoBot+ User-agent: DatadogSynthetics+ User-agent: datadome-pageprotect-scanner+ User-agent: Discordbot+ User-agent: doubleverify+ User-agent: DuckDuckBot+ User-agent: ElevenlabsBot+ User-agent: Embedly+ User-agent: facebookexternalhit+ User-agent: Googlebot+ User-agent: Googlebot Smartphone+ User-agent: Googlebot-News+ User-agent: Google-Display-Ads-Bot+ User-agent: Google-InspectionToolUser-agent: GoogleOther- User-agent: Grok… 83 more lines - Mar 2026Mixed Now blocked: AI2Bot and MistralAI-User. Now allowed: Claude-User.Found in the Internet Archive's copies: changed between 1 Feb 2026 and 1 Mar 2026. Evidence: before · after
Show exactly what changed
- User-agent: PiplBot- Disallow: /-- User-agent: CCbot- Disallow: /-- # ADO-657700- User-agent: Meta-ExternalAgent- Disallow: /-- User-agent: Meta-ExternalFetcher- Disallow: /-- User-agent: Google-Extended- Disallow: /- #-- User-agent: FacebookBot- Disallow: /-- # ADO-658177+ User-agent: AI2Bot+ User-agent: AI2Bot-Dolma+ User-agent: Alibaba.Security.Heimdall+ User-agent: AliyunSecBot/AliyunUser-agent: anthropic-ai- Disallow: /-+ User-agent: Applebot-Extended+ User-agent: archive.org_bot+ User-agent: bedrockbot+ User-agent: Bytespider+ User-agent: CCBot+ User-agent: ChatGLM-Spider+ User-agent: Claude-SearchBot+ User-agent: Claude-WebUser-agent: ClaudeBot- Disallow: /-User-agent: cohere-ai- Disallow: /-- User-agent: Timpibot- Disallow: /-+ User-agent: cohere-training-data-crawler+ User-agent: Crawlspace+ User-agent: DeepSeekBotUser-agent: Diffbot- Disallow: /-- User-agent: Bytespider- Disallow: /-- User-agent: Claude-Web- Disallow: /-- User-agent: PerplexityBot- Disallow: /… 48 more lines - Aug 2025More blocking Now blocked: Claude-SearchBot and Claude-User.Found in the Internet Archive's copies: changed between 1 Jul 2025 and 1 Aug 2025. Evidence: before · after
Show exactly what changed
+ # ADO-726772+ User-agent: Claude-User+ Disallow: /+ User-agent: Claude-SearchBot+ Disallow: /++ # ADO-727160+ User-agent: AliyunSecBot/Aliyun+ Disallow: /++ User-agent: Alibaba.Security.Heimdall+ Disallow: /++ # ADO-730264+ User-agent: Feedfetcher-Google+ Disallow: /+User-agent: *…Disallow: /test/- Disallow: /commentaryDisallow: /news/archive/commentary…Disallow: /practical-law-the-journal/search/- Disallow: /fr/- Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/- Disallow: /info-pages/custom-services-agreement-23-10/- Disallow: /info-pages/advertising-guidelines/Disallow: /pf/api/+ Disallow: /fr/+ Disallow: /it/+ Disallow: /es/+ Disallow: /pt/+ Disallow: /de/+ Disallow: /ar/+ Disallow: /account/subscribe/payment/ - Aug to Nov 2024More blocking Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, PerplexityBot and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Aug 2024 and 1 Nov 2024. Evidence: before · after
Show exactly what changed
+ # ADO-657700+ User-agent: Meta-ExternalAgent+ Disallow: /++ User-agent: Meta-ExternalFetcher+ Disallow: /++ User-agent: Google-Extended+ Disallow: /+ #++ User-agent: FacebookBot+ Disallow: /++ # ADO-658177User-agent: anthropic-ai…+ User-agent: ClaudeBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Timpibot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Bytespider+ Disallow: /+User-agent: Claude-Web…- User-agent: Google-Extended+ User-agent: PerplexityBotDisallow: /+ # ADO-661811+ User-agent: Applebot-Extended+ Disallow: /++User-agent: *…Disallow: /brandfeatures/venture-capital- Disallow: /journalists/Disallow: /assets/siteindex…Disallow: /practical-law-the-journal/search/- Disallow: /practical-law-the-journal/contributors/Disallow: /fr/+ Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/+ Disallow: /info-pages/custom-services-agreement-23-10/+ Disallow: /info-pages/advertising-guidelines/+ Disallow: /pf/api/… 13 more lines - Aug to Nov 2023More blocking Now blocked: Google-Extended, CCBot and anthropic-ai.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Nov 2023. Evidence: before · after
Show exactly what changed
- # robots_allow.txt for www.reuters.com- # Disallow: /*/key-developments/article/*+ # robots.txt for www.reuters.com+ User-agent: PiplBot+ Disallow: /++ User-agent: CCbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: Google-Extended+ Disallow: /+User-agent: *…Disallow: /beta- Disallow: /designtech*+ Disallow: /designtechDisallow: /featured-optimize…Disallow: /news/archive/commentary- Disallow: /brandfeatures/venture-capital*+ Disallow: /brandfeatures/venture-capitalDisallow: /journalists/…Disallow: /article/api/+ Disallow: /practical-law-the-journal/search/+ Disallow: /practical-law-the-journal/contributors/+ Disallow: /fr/- SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml+ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/sitemap_video_index.xml- SITEMAP: https://www.reuters.com/brandfeature/sitemap-- User-agent: Pipl- Disallow: /-+ SITEMAP: https://www.reuters.com/plus/sitemap-index.xml+ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml+ SITEMAP: https://www.reuters.com/graphics/sitemap.xml
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.