{"domain":"reuters.com","name":"Reuters","category":"news","subcategory":"wire","country":"GB","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 20 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. OAI-SearchBot and ChatGPT-User can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.200Z","last_checked":"2026-10-19T13:33:53.304Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://www.reuters.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.reuters.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.reuters.com/robots.txt","hops":[{"from":"https://reuters.com/robots.txt","to":"https://www.reuters.com/robots.txt","status":301}],"blob":"a6610234e4c1c34b9ea02a5e3af72dd4d219e2f3cd97ba6b1a7fe37166fa64a7","text":"# NOTICE: Collection of content, data and/or information from reuters.com through\n# automated means is prohibited unless you have prior written consent from Reuters \n# and may only be conducted for the purposes explicitly described in such permission. \n#\n# The terms and conditions for reuters.com may be found at \n# https://www.reuters.com/info-pages/terms-of-use/ \n#\n# If you would like to seek permission to license the content and/or other materials \n# of Reuters, please contact https://reutersagency.com/contact-us/enquire/\n\nUser-agent: AASA-Bot\nUser-agent: ADmantX\nUser-agent: AdmiralBot\nUser-agent: AdsBot-Google\nUser-agent: AdsBot-Google-Mobile\nUser-agent: AmazonAdBot\nUser-agent: Amzn-User\nUser-agent: Applebot\nUser-agent: AppleNewsBot\nUser-agent: Bingbot\nUser-agent: BingPreview\nUser-agent: BrightEdge\nUser-agent: BrightEdgeOnCrawl\nUser-agent: CensysInspect\nUser-agent: ChatGPT-User\nUser-agent: Cision\nUser-agent: Clickagy\nUser-agent: Concert\nUser-agent: ContextualBot\nUser-agent: CriteoBot\nUser-agent: DatadogSynthetics\nUser-agent: datadome-pageprotect-scanner\nUser-agent: Discordbot\nUser-agent: doubleverify\nUser-agent: ElevenlabsBot\nUser-agent: Embedly \nUser-agent: facebookexternalhit\nUser-agent: Googlebot\nUser-agent: Googlebot Smartphone\nUser-agent: Googlebot-News\nUser-agent: Google-Display-Ads-Bot\nUser-agent: Google-InspectionTool\nUser-agent: GoogleOther\nUser-agent: Google-Read-Aloud\nUser-agent: Google-Safety\nUser-agent: Google-Site-Verification\nUser-agent: GTmetrix\nUser-agent: GumGumBot\nUser-agent: ias_crawler\nUser-agent: Iframely\nUser-agent: leiki\nUser-agent: LinkedInBot\nUser-agent: LinkTiger\nUser-agent: Mantisbot\nUser-agent: Mediapartners-Google\nUser-agent: meta-externalads\nUser-agent: meta-webindexer\nUser-agent: MicrosoftPreview\nUser-agent: MJ12bot \nUser-agent: Moreover\nUser-agent: msnbot \nUser-agent: NFBNewslineRobot\nUser-agent: OAI-SearchBot\nUser-agent: Oncrawl\nUser-agent: Opebot-v\nUser-agent: Opoint\nUser-agent: Optimizer\nUser-agent: outbrain \nUser-agent: Pinterestbot\nUser-agent: Prerender\nUser-agent: Proofpoint \nUser-agent: proximic\nUser-agent: PubMatic Crawler Bot\nUser-agent: Quantcastbot\nUser-agent: Qwantbot\nUser-agent: Reuters SEO Screaming Frog Spider 007\nUser-agent: Reuters-NAUWI\nUser-agent: Scom-Crawler-For-Reuters\nUser-agent: SinceraSyntheticUser\nUser-agent: Slurp\nUser-agent: SmartologyBot\nUser-agent: snews\nUser-agent: SocialFlow\nUser-agent: StatusCake\nUser-agent: Storebot-Google\nUser-agent: Stripebot\nUser-agent: TTD-Content\nUser-agent: Twitterbot\nUser-agent: URLDefense\nUser-agent: Verity\nUser-agent: vuln_scan_by_trustedsite_com_halo_security\nUser-agent: WISEbot\nUser-agent: Xenu Link Sleuth\nUser-agent: Yahoo Link Preview\nUser-agent: Yahoo! JAPAN\nUser-agent: YahooMailProxy\nDisallow: /finance/stocks/option\nDisallow: /finance/stocks/financialHighlights\nDisallow: /search\nDisallow: /site-search/\nDisallow: /beta\nDisallow: /designtech\nDisallow: /featured-optimize\nDisallow: /energy-test\nDisallow: /article/beta\nDisallow: /sponsored/previewcampaign\nDisallow: /sponsored/previewarticle\nDisallow: /test/\nDisallow: /news/archive/commentary\nDisallow: /brandfeatures/venture-capital\nDisallow: /assets/siteindex\nDisallow: /article/api/\nDisallow: /practical-law-the-journal/search/\nDisallow: /pf/api/\nDisallow: /fr/\nDisallow: /it/\nDisallow: /es/\nDisallow: /pt/\nDisallow: /de/\nDisallow: /latam/\nDisallow: /account/subscribe/payment\nDisallow: /*/site-search/\n\nUser-agent: Applebot-Extended\nDisallow: /\n\n# Block all other bots\nUser-agent: *\nAllow: /plus/\nDisallow: /\n\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-jp-index/?outputType=xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml\nSITEMAP: https://www.reuters.com/plus/sitemap-index.xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml\nSITEMAP: https://www.reuters.com/graphics/sitemap.xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xml\nSITEMAP: https://www.reuters.com/static/video-sitemap/jp/sitemap_video_index.xml\nSITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml\nSITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml\nSITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["AASA-Bot","ADmantX","AdmiralBot","AdsBot-Google","AdsBot-Google-Mobile","AmazonAdBot","Amzn-User","Applebot","AppleNewsBot","Bingbot","BingPreview","BrightEdge","BrightEdgeOnCrawl","CensysInspect","ChatGPT-User","Cision","Clickagy","Concert","ContextualBot","CriteoBot","DatadogSynthetics","datadome-pageprotect-scanner","Discordbot","doubleverify","ElevenlabsBot","Embedly","facebookexternalhit","Googlebot","Googlebot Smartphone","Googlebot-News","Google-Display-Ads-Bot","Google-InspectionTool","GoogleOther","Google-Read-Aloud","Google-Safety","Google-Site-Verification","GTmetrix","GumGumBot","ias_crawler","Iframely","leiki","LinkedInBot","LinkTiger","Mantisbot","Mediapartners-Google","meta-externalads","meta-webindexer","MicrosoftPreview","MJ12bot","Moreover","msnbot","NFBNewslineRobot","OAI-SearchBot","Oncrawl","Opebot-v","Opoint","Optimizer","outbrain","Pinterestbot","Prerender","Proofpoint","proximic","PubMatic Crawler Bot","Quantcastbot","Qwantbot","Reuters SEO Screaming Frog Spider 007","Reuters-NAUWI","Scom-Crawler-For-Reuters","SinceraSyntheticUser","Slurp","SmartologyBot","snews","SocialFlow","StatusCake","Storebot-Google","Stripebot","TTD-Content","Twitterbot","URLDefense","Verity","vuln_scan_by_trustedsite_com_halo_security","WISEbot","Xenu Link Sleuth","Yahoo Link Preview","Yahoo! JAPAN","YahooMailProxy","Applebot-Extended","*"],"lines":145},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":125},"any":{"allowed":false,"line":125},"search":{"allowed":false,"line":125},"cart":{"allowed":false,"line":125},"checkout":{"allowed":false,"line":125},"account":{"allowed":false,"line":125}},"disallows":1,"lines":[124,125],"groupRanges":[[124,125]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]},{"token":"Amzn-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":130},"any":{"allowed":false,"line":130},"search":{"allowed":false,"line":130},"cart":{"allowed":false,"line":130},"checkout":{"allowed":false,"line":130},"account":{"allowed":false,"line":130}},"disallows":1,"lines":[128,130],"groupRanges":[[128,130]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":99},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":26,"lines":[11,122],"groupRanges":[[11,122]]}],"lanes":{"training":{"level":"none","none":9,"full":0,"restricted":0,"n":9},"search":{"level":"mixed","none":6,"full":2,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":5,"full":2,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":["Amazon CloudFront"],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2026-06","s":"partial","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20},{"m":"2026-07","s":"partial","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20},{"m":"2026-08","s":"partial","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20},{"m":"2026-09","s":"partial","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20},{"m":"2026-10","s":"partial","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20}]},"events":[{"domain":"reuters.com","name":"Reuters","category":"news","date":"2026-09","precision":"month","source":"archive","provisional":false,"between":["2026-08-05","2026-09-01"],"summary":"Named Applebot-Extended explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260805193147/https://www.reuters.com/robots.txt","https://web.archive.org/web/20260901092027/https://www.reuters.com/robots.txt"],"diff":[{"t":"ctx","s":"User-agent: ADmantX"},{"t":"add","s":"User-agent: AdmiralBot"},{"t":"ctx","s":"User-agent: AdsBot-Google"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: doubleverify"},{"t":"del","s":"User-agent: DuckDuckBot"},{"t":"ctx","s":"User-agent: ElevenlabsBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /latam/"},{"t":"del","s":"Disallow: /account/subscribe/payment/"},{"t":"add","s":"Disallow: /jp/"},{"t":"add","s":"Disallow: /account/subscribe/payment"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Block all other bots"},{"t":"gap","s":"…"},{"t":"ctx","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml"},{"t":"add","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml"},{"t":"add","s":"SITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml"}]},{"domain":"reuters.com","name":"Reuters","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-04-01","2026-06-05"],"summary":"Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.","dir":"up","up":["GPTBot","MistralAI-Training","Amazonbot","Amzn-SearchBot","DuckAssistBot","MistralAI-Index","Claude-User","Perplexity-User","Google-Agent"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260401002117/https://www.reuters.com/robots.txt","https://web.archive.org/web/20260605150558/https://www.reuters.com/robots.txt"],"diff":[{"t":"del","s":"# robots.txt for www.reuters.com"},{"t":"del","s":""},{"t":"del","s":"User-agent: AI2Bot"},{"t":"del","s":"User-agent: AI2Bot-Dolma"},{"t":"del","s":"User-agent: Alibaba.Security.Heimdall"},{"t":"del","s":"User-agent: AliyunSecBot/Aliyun"},{"t":"del","s":"User-agent: anthropic-ai"},{"t":"del","s":"User-agent: Applebot-Extended"},{"t":"del","s":"User-agent: archive.org_bot"},{"t":"del","s":"User-agent: bedrockbot"},{"t":"del","s":"User-agent: Bytespider"},{"t":"del","s":"User-agent: CCBot"},{"t":"del","s":"User-agent: ChatGLM-Spider"},{"t":"del","s":"User-agent: Claude-SearchBot"},{"t":"del","s":"User-agent: Claude-Web"},{"t":"del","s":"User-agent: ClaudeBot"},{"t":"del","s":"User-agent: cohere-ai"},{"t":"del","s":"User-agent: cohere-training-data-crawler"},{"t":"del","s":"User-agent: Crawlspace"},{"t":"del","s":"User-agent: DeepSeekBot"},{"t":"del","s":"User-agent: Diffbot"},{"t":"del","s":"User-agent: FacebookBot"},{"t":"del","s":"User-agent: Feedfetcher-Google"},{"t":"del","s":"User-agent: Gemini-Deep-Research"},{"t":"del","s":"User-agent: Google-CloudVertexBot"},{"t":"del","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: AASA-Bot"},{"t":"add","s":"User-agent: ADmantX"},{"t":"add","s":"User-agent: AdsBot-Google"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile"},{"t":"add","s":"User-agent: AmazonAdBot"},{"t":"add","s":"User-agent: Amzn-User"},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: AppleNewsBot"},{"t":"add","s":"User-agent: Bingbot"},{"t":"add","s":"User-agent: BingPreview"},{"t":"add","s":"User-agent: BrightEdge"},{"t":"add","s":"User-agent: BrightEdgeOnCrawl"},{"t":"add","s":"User-agent: CensysInspect"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: Cision"},{"t":"add","s":"User-agent: Clickagy"},{"t":"add","s":"User-agent: Concert"},{"t":"add","s":"User-agent: ContextualBot"},{"t":"add","s":"User-agent: CriteoBot"},{"t":"add","s":"User-agent: DatadogSynthetics"},{"t":"add","s":"User-agent: datadome-pageprotect-scanner"},{"t":"add","s":"User-agent: Discordbot"},{"t":"add","s":"User-agent: doubleverify"},{"t":"add","s":"User-agent: DuckDuckBot"},{"t":"add","s":"User-agent: ElevenlabsBot"},{"t":"add","s":"User-agent: Embedly "},{"t":"add","s":"User-agent: facebookexternalhit"},{"t":"add","s":"User-agent: Googlebot"},{"t":"add","s":"User-agent: Googlebot Smartphone"},{"t":"add","s":"User-agent: Googlebot-News"},{"t":"add","s":"User-agent: Google-Display-Ads-Bot"},{"t":"add","s":"User-agent: Google-InspectionTool"},{"t":"ctx","s":"User-agent: GoogleOther"},{"t":"del","s":"User-agent: Grok"},{"t":"gap","s":"… 83 more lines"}]},{"domain":"reuters.com","name":"Reuters","category":"news","date":"2026-03","precision":"month","source":"archive","provisional":false,"between":["2026-02-01","2026-03-01"],"summary":"Now blocked: AI2Bot and MistralAI-User. Now allowed: Claude-User.","dir":"mixed","up":["AI2Bot","MistralAI-User"],"down":["Claude-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260201001928/https://www.reuters.com/robots.txt","https://web.archive.org/web/20260301002752/https://www.reuters.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: PiplBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: CCbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"# ADO-657700"},{"t":"del","s":"User-agent: Meta-ExternalAgent"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Meta-ExternalFetcher"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"#"},{"t":"del","s":""},{"t":"del","s":"User-agent: FacebookBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"# ADO-658177"},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"User-agent: AI2Bot-Dolma"},{"t":"add","s":"User-agent: Alibaba.Security.Heimdall"},{"t":"add","s":"User-agent: AliyunSecBot/Aliyun"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"User-agent: archive.org_bot"},{"t":"add","s":"User-agent: bedrockbot"},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"User-agent: ChatGLM-Spider"},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"ctx","s":"User-agent: ClaudeBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: cohere-ai"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Timpibot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"add","s":"User-agent: cohere-training-data-crawler"},{"t":"add","s":"User-agent: Crawlspace"},{"t":"add","s":"User-agent: DeepSeekBot"},{"t":"ctx","s":"User-agent: Diffbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Bytespider"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Claude-Web"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: PerplexityBot"},{"t":"del","s":"Disallow: /"},{"t":"gap","s":"… 48 more lines"}]},{"domain":"reuters.com","name":"Reuters","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-07-01","2025-08-01"],"summary":"Now blocked: Claude-SearchBot and Claude-User.","dir":"up","up":["Claude-SearchBot","Claude-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250701005843/https://www.reuters.com/robots.txt","https://web.archive.org/web/20250801185400/https://www.reuters.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# ADO-726772"},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# ADO-727160"},{"t":"add","s":"User-agent: AliyunSecBot/Aliyun "},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Alibaba.Security.Heimdall "},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# ADO-730264"},{"t":"add","s":"User-agent: Feedfetcher-Google"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /test/"},{"t":"del","s":"Disallow: /commentary"},{"t":"ctx","s":"Disallow: /news/archive/commentary"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /practical-law-the-journal/search/"},{"t":"del","s":"Disallow: /fr/"},{"t":"del","s":"Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/"},{"t":"del","s":"Disallow: /info-pages/custom-services-agreement-23-10/"},{"t":"del","s":"Disallow: /info-pages/advertising-guidelines/"},{"t":"ctx","s":"Disallow: /pf/api/"},{"t":"add","s":"Disallow: /fr/"},{"t":"add","s":"Disallow: /it/"},{"t":"add","s":"Disallow: /es/"},{"t":"add","s":"Disallow: /pt/"},{"t":"add","s":"Disallow: /de/"},{"t":"add","s":"Disallow: /ar/"},{"t":"add","s":"Disallow: /account/subscribe/payment/"},{"t":"ctx","s":""}]},{"domain":"reuters.com","name":"Reuters","category":"news","date":"2024-11","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-11-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, PerplexityBot and Meta-ExternalFetcher.","dir":"up","up":["ClaudeBot","Applebot-Extended","Meta-ExternalAgent","Bytespider","PerplexityBot","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801030613/https://www.reuters.com/robots.txt","https://web.archive.org/web/20241101004338/https://www.reuters.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# ADO-657700"},{"t":"add","s":"User-agent: Meta-ExternalAgent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Meta-ExternalFetcher"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"#"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# ADO-658177"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Timpibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"# ADO-661811"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /brandfeatures/venture-capital"},{"t":"del","s":"Disallow: /journalists/"},{"t":"ctx","s":"Disallow: /assets/siteindex"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /practical-law-the-journal/search/"},{"t":"del","s":"Disallow: /practical-law-the-journal/contributors/"},{"t":"ctx","s":"Disallow: /fr/"},{"t":"add","s":"Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/"},{"t":"add","s":"Disallow: /info-pages/custom-services-agreement-23-10/"},{"t":"add","s":"Disallow: /info-pages/advertising-guidelines/"},{"t":"add","s":"Disallow: /pf/api/"},{"t":"ctx","s":""},{"t":"gap","s":"… 13 more lines"}]},{"domain":"reuters.com","name":"Reuters","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-11-01"],"summary":"Now blocked: Google-Extended, CCBot and anthropic-ai.","dir":"up","up":["Google-Extended","CCBot","anthropic-ai"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801011936/https://www.reuters.com/robots.txt","https://web.archive.org/web/20231101023513/https://www.reuters.com/robots.txt"],"diff":[{"t":"del","s":"# robots_allow.txt for www.reuters.com"},{"t":"del","s":"# Disallow: /*/key-developments/article/*"},{"t":"add","s":"# robots.txt for www.reuters.com"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: PiplBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /beta"},{"t":"del","s":"Disallow: /designtech*"},{"t":"add","s":"Disallow: /designtech"},{"t":"ctx","s":"Disallow: /featured-optimize"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /news/archive/commentary"},{"t":"del","s":"Disallow: /brandfeatures/venture-capital*"},{"t":"add","s":"Disallow: /brandfeatures/venture-capital"},{"t":"ctx","s":"Disallow: /journalists/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /article/api/"},{"t":"add","s":"Disallow: /practical-law-the-journal/search/"},{"t":"add","s":"Disallow: /practical-law-the-journal/contributors/"},{"t":"add","s":"Disallow: /fr/"},{"t":"ctx","s":""},{"t":"del","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml "},{"t":"add","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml"},{"t":"ctx","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml"},{"t":"ctx","s":"SITEMAP: https://www.reuters.com/sitemap_video_index.xml"},{"t":"del","s":"SITEMAP: https://www.reuters.com/brandfeature/sitemap"},{"t":"del","s":""},{"t":"del","s":"User-agent: Pipl"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"add","s":"SITEMAP: https://www.reuters.com/plus/sitemap-index.xml"},{"t":"add","s":"SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml"},{"t":"add","s":"SITEMAP: https://www.reuters.com/graphics/sitemap.xml"},{"t":"ctx","s":""}]}]}