{"domain":"bloomberg.com","name":"Bloomberg","category":"news","subcategory":"business","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. It keeps OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User out of parts of it. Perplexity-User can reach most of it. The homepage also showed our bot, which says who it is, a Generic bot challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.222Z","last_checked":"2026-10-19T13:34:31.057Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.bloomberg.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.bloomberg.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.bloomberg.com/robots.txt","hops":[{"from":"https://bloomberg.com/robots.txt","to":"https://www.bloomberg.com/robots.txt","status":301}],"blob":"6879b4e3afbc243a8422c6e1331cad0f7c85a7817612bf65a56d27cd6bc9ebf7","text":"# Bot rules:\n# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.\n# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.\n# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.\n# If you can read this then you should apply here https://www.bloomberg.com/careers/\nUser-agent: *\nDisallow: /polska\nAllow: /account/newsletters\nDisallow: /account/*\nDisallow: /tosv*.html\nDisallow: /search\nDisallow: /company/search/\nDisallow: /professional/search/\nDisallow: /impact/search/\nDisallow: /ux/search/\nDisallow: /wnwi/search/\nDisallow: /gei/search/\nDisallow: /impact/search/\nDisallow: /netzeropathfinders/search/\nDisallow: /notices/search/\nDisallow: /distribution/search/\nDisallow: /ukinnovators/search/\nDisallow: /latam/search/\nDisallow: /faq/search/\nDisallow: /tc/search/\nDisallow: /subscriptions/group/manage/\nDisallow: /preview/lineup\nDisallow: /preview/articles\nDisallow: /explore/\nDisallow: /press-releases/\nDisallow: /artemis/\nDisallow: /sessions-publisher/\n\nUser-agent: Google-Extended\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\n\nUser-agent: Mediapartners-Google\nDisallow: /about/careers\nDisallow: /about/careers/\nDisallow: /offlinemessage/\nDisallow: /apps/fbk\nDisallow: /bb/newsarchive/\nDisallow: /apps/news\n\nUser-agent: Spinn3r\nDisallow: /podcasts/\nDisallow: /feed/podcast/\nDisallow: /bb/avfile/\n\nUser-agent: Googlebot-News\nDisallow: /sponsor/\nDisallow: /news/sponsors/*\nDisallow: /news/terminal/*\n\nUser-agent: Twitterbot\nAllow: /en/news/thp\n\nUser-agent: CCBot\nDisallow: /\n\n# Development Tools\nUser-agent: Python-urllib\nDisallow: /\nUser-agent: python-requests\nDisallow: /\nUser-agent: Python-http-client\nDisallow: /\n\n# AI/LLM Bots\nUser-agent: Claude-Web\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: anthropic-ai\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: PerplexityBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: YouBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: DuckAssistBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: cohere-ai\nDisallow: /\n\n# AI/LLM Bots - Homepage Access\n# Homepage and regional homepages only. The $ makes each Allow an exact match\n# so it beats \"Disallow: /\"; without it the rule is a prefix match and opens\n# everything under that route. /region/ is a 302 to /region and needs its own\n# rule, since robots.txt is checked before the redirect can be followed.\n\n# Training\nUser-agent: GPTBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: ClaudeBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\n\n# Search indexing\nUser-agent: OAI-SearchBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nAllow: /$\nAllow: /europe$\nAllow: /europe/$\nAllow: /asia$\nAllow: /asia/$\nAllow: /jp$\nAllow: /jp/$\nAllow: /india$\nAllow: /india/$\nAllow: /middleeast$\nAllow: /middleeast/$\nAllow: /africa$\nAllow: /africa/$\nAllow: /canada$\nAllow: /canada/$\nAllow: /uk$\nAllow: /uk/$\nAllow: /latinamerica$\nAllow: /latinamerica/$\nAllow: /llms.txt$\nAllow: /llms.txt/$\nAllow: /europe/llms.txt$\nAllow: /europe/llms.txt/$\nAllow: /asia/llms.txt$\nAllow: /asia/llms.txt/$\nAllow: /jp/llms.txt$\nAllow: /jp/llms.txt/$\nAllow: /india/llms.txt$\nAllow: /india/llms.txt/$\nAllow: /middleeast/llms.txt$\nAllow: /middleeast/llms.txt/$\nAllow: /africa/llms.txt$\nAllow: /africa/llms.txt/$\nAllow: /canada/llms.txt$\nAllow: /canada/llms.txt/$\nAllow: /uk/llms.txt$\nAllow: /uk/llms.txt/$\nAllow: /latinamerica/llms.txt$\nAllow: /latinamerica/llms.txt/$\nUser-agent: Claude-SearchBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nAllow: /$\nAllow: /europe$\nAllow: /europe/$\nAllow: /asia$\nAllow: /asia/$\nAllow: /jp$\nAllow: /jp/$\nAllow: /india$\nAllow: /india/$\nAllow: /middleeast$\nAllow: /middleeast/$\nAllow: /africa$\nAllow: /africa/$\nAllow: /canada$\nAllow: /canada/$\nAllow: /uk$\nAllow: /uk/$\nAllow: /latinamerica$\nAllow: /latinamerica/$\nAllow: /llms.txt$\nAllow: /llms.txt/$\nAllow: /europe/llms.txt$\nAllow: /europe/llms.txt/$\nAllow: /asia/llms.txt$\nAllow: /asia/llms.txt/$\nAllow: /jp/llms.txt$\nAllow: /jp/llms.txt/$\nAllow: /india/llms.txt$\nAllow: /india/llms.txt/$\nAllow: /middleeast/llms.txt$\nAllow: /middleeast/llms.txt/$\nAllow: /africa/llms.txt$\nAllow: /africa/llms.txt/$\nAllow: /canada/llms.txt$\nAllow: /canada/llms.txt/$\nAllow: /uk/llms.txt$\nAllow: /uk/llms.txt/$\nAllow: /latinamerica/llms.txt$\nAllow: /latinamerica/llms.txt/$\n\n# User-triggered fetches\nUser-agent: ChatGPT-User\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nAllow: /$\nAllow: /europe$\nAllow: /europe/$\nAllow: /asia$\nAllow: /asia/$\nAllow: /jp$\nAllow: /jp/$\nAllow: /india$\nAllow: /india/$\nAllow: /middleeast$\nAllow: /middleeast/$\nAllow: /africa$\nAllow: /africa/$\nAllow: /canada$\nAllow: /canada/$\nAllow: /uk$\nAllow: /uk/$\nAllow: /latinamerica$\nAllow: /latinamerica/$\nAllow: /llms.txt$\nAllow: /llms.txt/$\nAllow: /europe/llms.txt$\nAllow: /europe/llms.txt/$\nAllow: /asia/llms.txt$\nAllow: /asia/llms.txt/$\nAllow: /jp/llms.txt$\nAllow: /jp/llms.txt/$\nAllow: /india/llms.txt$\nAllow: /india/llms.txt/$\nAllow: /middleeast/llms.txt$\nAllow: /middleeast/llms.txt/$\nAllow: /africa/llms.txt$\nAllow: /africa/llms.txt/$\nAllow: /canada/llms.txt$\nAllow: /canada/llms.txt/$\nAllow: /uk/llms.txt$\nAllow: /uk/llms.txt/$\nAllow: /latinamerica/llms.txt$\nAllow: /latinamerica/llms.txt/$\nUser-agent: Claude-User\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nAllow: /$\nAllow: /europe$\nAllow: /europe/$\nAllow: /asia$\nAllow: /asia/$\nAllow: /jp$\nAllow: /jp/$\nAllow: /india$\nAllow: /india/$\nAllow: /middleeast$\nAllow: /middleeast/$\nAllow: /africa$\nAllow: /africa/$\nAllow: /canada$\nAllow: /canada/$\nAllow: /uk$\nAllow: /uk/$\nAllow: /latinamerica$\nAllow: /latinamerica/$\nAllow: /llms.txt$\nAllow: /llms.txt/$\nAllow: /europe/llms.txt$\nAllow: /europe/llms.txt/$\nAllow: /asia/llms.txt$\nAllow: /asia/llms.txt/$\nAllow: /jp/llms.txt$\nAllow: /jp/llms.txt/$\nAllow: /india/llms.txt$\nAllow: /india/llms.txt/$\nAllow: /middleeast/llms.txt$\nAllow: /middleeast/llms.txt/$\nAllow: /africa/llms.txt$\nAllow: /africa/llms.txt/$\nAllow: /canada/llms.txt$\nAllow: /canada/llms.txt/$\nAllow: /uk/llms.txt$\nAllow: /uk/llms.txt/$\nAllow: /latinamerica/llms.txt$\nAllow: /latinamerica/llms.txt/$\n\n# Google Services\nUser-agent: Google-Apps\nDisallow: /\nUser-agent: Google-Apps-Script\nDisallow: /\nUser-agent: AppEngine-Google\nDisallow: /\nUser-agent: Google-Cloud\nDisallow: /\nUser-agent: Google-CloudVertexBot\nDisallow: /\n\n# Major Company Bots\nUser-agent: AmazonBot\nDisallow: /\nUser-agent: AmazonAdBot\nDisallow: /\nUser-agent: ByteSpider\nDisallow: /\nUser-agent: Meta-ExternalAgent\nDisallow: /\nUser-agent: Meta-ExternalFetcher\nDisallow: /\nUser-agent: Meta-ExternalAgent-Image\nDisallow: /\nUser-agent: Applebot-Extended\nDisallow: /\nUser-agent: Bytespider-Image\nDisallow: /\n\n# Feed & Content Aggregators\nUser-agent: Feedly\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: FeedlyBot\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: FeedlyApp\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: MWFeedParser\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\n\n# Analytics & Marketing\nUser-agent: comscore\nDisallow: /\nUser-agent: Comscore\nDisallow: /\nUser-agent: HubSpot\nDisallow: /\nUser-agent: hubspot\nDisallow: /\nUser-agent: Criteo\nDisallow: /\nUser-agent: criteo-bot\nDisallow: /\nUser-agent: Peer39_Crawler\nDisallow: /\nUser-agent: rogerbot\nDisallow: /\nUser-agent: linkdexbot\nDisallow: /\nUser-agent: NTENT\nDisallow: /\nUser-agent: AndersPinkBot\nDisallow: /\nUser-agent: IndeedBot\nDisallow: /\n\n# Search Engine Bots\nUser-agent: MJ12bot\nDisallow: /\nUser-agent: PetalBot\nDisallow: /\nUser-agent: YisouSpider\nDisallow: /\nUser-agent: 360Spider\nDisallow: /\nUser-agent: Qwantify\nDisallow: /\nUser-agent: ToutiaoSpider\nDisallow: /\n\n# Monitoring & Security\nUser-agent: Screaming Frog SEO Spider\nDisallow: /\nAllow: /professional\nAllow: /company\nAllow: /latam\nAllow: /faq\nAllow: /tc\nUser-agent: PRTG\nDisallow: /\nUser-agent: FreshpingBot\nDisallow: /\nUser-agent: Panopta\nDisallow: /\nUser-agent: DatadogSynthetics\nDisallow: /\nUser-agent: Rackspace\nDisallow: /\nUser-agent: censys\nDisallow: /\nUser-agent: burp\nDisallow: /\nUser-agent: Burp\nDisallow: /\nUser-agent: check_http\nDisallow: /\nUser-agent: DotcomMonitor\nDisallow: /\nUser-agent: WatchSumo\nDisallow: /\nUser-agent: WormlyBot\nDisallow: /\nUser-agent: CalibreBot\nDisallow: /\nUser-agent: AudistoBot\nDisallow: /\n\n# Social & Content\nUser-agent: Hatena\nDisallow: /\nUser-agent: Hatena-Bookmark\nDisallow: /\nUser-agent: EveryoneSocialBot\nDisallow: /\nUser-agent: Diffbot\nDisallow: /\nUser-agent: CCBot\nDisallow: /\nUser-agent: Netvibes\nDisallow: /\nUser-agent: WebCEO\nDisallow: /\nUser-agent: Postano\nDisallow: /\nUser-agent: RebelMouse\nDisallow: /\nUser-agent: Muck-Rack\nDisallow: /\nUser-agent: InstapaperViewer\nDisallow: /\nUser-agent: Twurly\nDisallow: /\nUser-agent: LivelapBot\nDisallow: /\nUser-agent: DatagnionBot\nDisallow: /\nUser-agent: Linespider\nDisallow: /\nUser-agent: Discourse\nDisallow: /\n\n# Potentially Malicious\nUser-agent: Medusa\nDisallow: /\nUser-agent: pingback\nDisallow: /\nUser-agent: WordPress\nDisallow: /\nUser-agent: wp_ping\nDisallow: /\nUser-agent: MauiBot\nDisallow: /\nUser-agent: ltx71\nDisallow: /\nUser-agent: WeSEE\nDisallow: /\nUser-agent: halebot\nDisallow: /\nUser-agent: BrightBot\nDisallow: /\n\n# Gaming/3D\nUser-agent: UnityPlayer\nDisallow: /\n\n# Sitemaps app\nSitemap: https://www.bloomberg.com/sitemaps/news/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/news/latest.xml\nSitemap: https://www.bloomberg.com/sitemaps/collections/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/media/video/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/media/audio/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/people/profiles/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/companies/public-company/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/companies/private-company/index.xml\nSitemap: https://www.bloomberg.com/sitemaps/securites/index.xml\n\n# Billionaires, owned by graphics\nSitemap: https://www.bloomberg.com/billionaires/sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":74,"text":"AI/LLM Bots"},{"line":113,"text":"AI/LLM Bots - Homepage Access"},{"line":119,"text":"Training"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Google-Extended","Mediapartners-Google","Spinn3r","Googlebot-News","Twitterbot","CCBot","Python-urllib","python-requests","Python-http-client","Claude-Web","anthropic-ai","PerplexityBot","YouBot","DuckAssistBot","cohere-ai","GPTBot","ClaudeBot","OAI-SearchBot","Claude-SearchBot","ChatGPT-User","Claude-User","Google-Apps","Google-Apps-Script","AppEngine-Google","Google-Cloud","Google-CloudVertexBot","AmazonBot","AmazonAdBot","ByteSpider","Meta-ExternalAgent","Meta-ExternalFetcher","Meta-ExternalAgent-Image","Applebot-Extended","Bytespider-Image","Feedly","FeedlyBot","FeedlyApp","MWFeedParser","comscore","Comscore","HubSpot","hubspot","Criteo","criteo-bot","Peer39_Crawler","rogerbot","linkdexbot","NTENT","AndersPinkBot","IndeedBot","MJ12bot","PetalBot","YisouSpider","360Spider","Qwantify","ToutiaoSpider","Screaming Frog SEO Spider","PRTG","FreshpingBot","Panopta","DatadogSynthetics","Rackspace","censys","burp","Burp","check_http","DotcomMonitor","WatchSumo","WormlyBot","CalibreBot","AudistoBot","Hatena","Hatena-Bookmark","EveryoneSocialBot","Diffbot","Netvibes","WebCEO","Postano","RebelMouse","Muck-Rack","InstapaperViewer","Twurly","LivelapBot","DatagnionBot","Linespider","Discourse","Medusa","pingback","WordPress","wp_ping","MauiBot","ltx71","WeSEE","halebot","BrightBot","UnityPlayer"],"lines":531},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":121},"any":{"allowed":false,"line":121},"search":{"allowed":false,"line":121},"cart":{"allowed":false,"line":121},"checkout":{"allowed":false,"line":121},"account":{"allowed":false,"line":121}},"disallows":1,"lines":[120,126],"groupRanges":[[120,126]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":128},"any":{"allowed":false,"line":128},"search":{"allowed":false,"line":128},"cart":{"allowed":false,"line":128},"checkout":{"allowed":false,"line":128},"account":{"allowed":false,"line":128}},"disallows":1,"lines":[127,133],"groupRanges":[[127,133]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":35},"any":{"allowed":false,"line":35},"search":{"allowed":false,"line":35},"cart":{"allowed":false,"line":35},"checkout":{"allowed":false,"line":35},"account":{"allowed":false,"line":35}},"disallows":1,"lines":[34,40],"groupRanges":[[34,40]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":349},"any":{"allowed":false,"line":349},"search":{"allowed":false,"line":349},"cart":{"allowed":false,"line":349},"checkout":{"allowed":false,"line":349},"account":{"allowed":false,"line":349}},"disallows":1,"lines":[348,349],"groupRanges":[[348,349]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":64},"any":{"allowed":false,"line":64},"search":{"allowed":false,"line":64},"cart":{"allowed":false,"line":64},"checkout":{"allowed":false,"line":64},"account":{"allowed":false,"line":64}},"disallows":2,"lines":[63,64,469,470],"groupRanges":[[63,64],[469,470]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":343},"any":{"allowed":false,"line":343},"search":{"allowed":false,"line":343},"cart":{"allowed":false,"line":343},"checkout":{"allowed":false,"line":343},"account":{"allowed":false,"line":343}},"disallows":1,"lines":[342,343],"groupRanges":[[342,343]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":341},"any":{"allowed":false,"line":341},"search":{"allowed":false,"line":341},"cart":{"allowed":false,"line":341},"checkout":{"allowed":false,"line":341},"account":{"allowed":false,"line":341}},"disallows":1,"lines":[340,341],"groupRanges":[[340,341]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":83},"any":{"allowed":false,"line":83},"search":{"allowed":false,"line":83},"cart":{"allowed":false,"line":83},"checkout":{"allowed":false,"line":83},"account":{"allowed":false,"line":83}},"disallows":1,"lines":[82,88],"groupRanges":[[82,88]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"restricted","probes":{"root":{"allowed":true,"line":143},"any":{"allowed":false,"line":137},"search":{"allowed":false,"line":137},"cart":{"allowed":false,"line":137},"checkout":{"allowed":false,"line":137},"account":{"allowed":false,"line":137}},"disallows":1,"lines":[136,181],"groupRanges":[[136,181]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"restricted","probes":{"root":{"allowed":true,"line":189},"any":{"allowed":false,"line":183},"search":{"allowed":false,"line":183},"cart":{"allowed":false,"line":183},"checkout":{"allowed":false,"line":183},"account":{"allowed":false,"line":183}},"disallows":1,"lines":[182,227],"groupRanges":[[182,227]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":90},"any":{"allowed":false,"line":90},"search":{"allowed":false,"line":90},"cart":{"allowed":false,"line":90},"checkout":{"allowed":false,"line":90},"account":{"allowed":false,"line":90}},"disallows":1,"lines":[89,95],"groupRanges":[[89,95]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":337},"any":{"allowed":false,"line":337},"search":{"allowed":false,"line":337},"cart":{"allowed":false,"line":337},"checkout":{"allowed":false,"line":337},"account":{"allowed":false,"line":337}},"disallows":1,"lines":[336,337],"groupRanges":[[336,337]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":104},"any":{"allowed":false,"line":104},"search":{"allowed":false,"line":104},"cart":{"allowed":false,"line":104},"checkout":{"allowed":false,"line":104},"account":{"allowed":false,"line":104}},"disallows":1,"lines":[103,109],"groupRanges":[[103,109]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"restricted","probes":{"root":{"allowed":true,"line":237},"any":{"allowed":false,"line":231},"search":{"allowed":false,"line":231},"cart":{"allowed":false,"line":231},"checkout":{"allowed":false,"line":231},"account":{"allowed":false,"line":231}},"disallows":1,"lines":[230,275],"groupRanges":[[230,275]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"restricted","probes":{"root":{"allowed":true,"line":283},"any":{"allowed":false,"line":277},"search":{"allowed":false,"line":277},"cart":{"allowed":false,"line":277},"checkout":{"allowed":false,"line":277},"account":{"allowed":false,"line":277}},"disallows":1,"lines":[276,321],"groupRanges":[[276,321]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":345},"any":{"allowed":false,"line":345},"search":{"allowed":false,"line":345},"cart":{"allowed":false,"line":345},"checkout":{"allowed":false,"line":345},"account":{"allowed":false,"line":345}},"disallows":1,"lines":[344,345],"groupRanges":[[344,345]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":11},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":25,"lines":[6,32],"groupRanges":[[6,32]]}],"lanes":{"training":{"level":"mixed","none":7,"full":2,"restricted":0,"n":9},"search":{"level":"mixed","none":3,"full":3,"restricted":2,"n":8},"user_directed":{"level":"mixed","none":1,"full":4,"restricted":2,"n":7}},"door":{"door":"challenged","http":403,"detail":"Generic bot challenge page"},"edge":{"cdn":[],"botManagement":[{"vendor":"HUMAN (PerimeterX)","cookies":["_pxhd"]}]},"llms":{"state":"refused","http":403,"url":"https://www.bloomberg.com/llms.txt"},"ai_txt":{"state":"refused","http":403,"url":"https://www.bloomberg.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.bloomberg.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":5},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":5},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":5},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":5},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":5},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":15},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11}]},"events":[{"domain":"bloomberg.com","name":"Bloomberg","category":"news","date":"2026-10","precision":"month","source":"archive","provisional":false,"between":["2026-09-11","2026-10-01"],"summary":"Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.","dir":"down","up":[],"down":["OAI-SearchBot","Claude-SearchBot","ChatGPT-User","Claude-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260911092001/https://www.bloomberg.com/robots.txt","https://web.archive.org/web/20261001024331/https://www.bloomberg.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: CCBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"# Development Tools"},{"t":"gap","s":"…"},{"t":"ctx","s":"# AI/LLM Bots"},{"t":"del","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /tc"},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /tc"},{"t":"del","s":"User-agent: Claude-Web"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /tc"},{"t":"del","s":"User-agent: anthropic-ai"},{"t":"add","s":"User-agent: YouBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /tc"},{"t":"del","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /tc"},{"t":"del","s":"User-agent: YouBot"},{"t":"add","s":"User-agent: cohere-ai"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# AI/LLM Bots - Homepage Access"},{"t":"add","s":"# Homepage and regional homepages only. The $ makes each Allow an exact match"},{"t":"add","s":"# so it beats \"Disallow: /\"; without it the rule is a prefix match and opens"},{"t":"add","s":"# everything under that route. /region/ is a 302 to /region and needs its own"},{"t":"add","s":"# rule, since robots.txt is checked before the redirect can be followed."},{"t":"add","s":""},{"t":"add","s":"# Training"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"Allow: /professional"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"del","s":"User-agent: Claude-User"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"gap","s":"… 152 more lines"}]},{"domain":"bloomberg.com","name":"Bloomberg","category":"news","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-05-01","2026-06-02"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.","dir":"up","up":["ClaudeBot","Applebot-Extended","OAI-SearchBot","Claude-SearchBot","DuckAssistBot","Claude-User","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260501225650/https://www.bloomberg.com/robots.txt","https://web.archive.org/web/20260602050346/https://www.bloomberg.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":"User-agent: PerplexityBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"ctx","s":"User-agent: YouBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Allow: /professional"},{"t":"add","s":"Allow: /company"},{"t":"add","s":"Allow: /latam"},{"t":"add","s":"Allow: /faq"},{"t":"add","s":"Allow: /tc"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: cohere-ai"},{"t":"gap","s":"… 21 more lines"}]},{"domain":"bloomberg.com","name":"Bloomberg","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-07-01","2025-08-01"],"summary":"Now blocked: Meta-ExternalAgent, Bytespider and Amazonbot.","dir":"up","up":["Meta-ExternalAgent","Bytespider","Amazonbot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250701020735/https://www.bloomberg.com/robots.txt","https://web.archive.org/web/20250801055411/https://www.bloomberg.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://www.bloomberg.com/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/sitemap_news.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/bpol/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/businessweek/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/citylab/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/green/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/technology/sitemap_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_securities_index.xml"},{"t":"del","s":"Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_profiles_company_index.xml"},{"t":"add","s":"# Major Company Bots"},{"t":"add","s":"User-agent: AmazonBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: AmazonAdBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: ByteSpider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Meta-ExternalAgent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Feed & Content Aggregators"},{"t":"add","s":"User-agent: Feedly"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: FeedlyBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: FeedlyApp"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: MWFeedParser"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Analytics & Marketing"},{"t":"add","s":"User-agent: comscore"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Comscore"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: HubSpot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: hubspot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Criteo"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: criteo-bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Peer39_Crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: rogerbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: linkdexbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: NTENT"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: AndersPinkBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: IndeedBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Search Engine Bots"},{"t":"add","s":"User-agent: MJ12bot"},{"t":"add","s":"Disallow: /"},{"t":"gap","s":"… 126 more lines"}]},{"domain":"bloomberg.com","name":"Bloomberg","category":"news","date":"2025-03","precision":"month","source":"archive","provisional":false,"between":["2025-02-01","2025-03-01"],"summary":"Now blocked: anthropic-ai, PerplexityBot and ChatGPT-User.","dir":"up","up":["anthropic-ai","PerplexityBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250201081237/https://www.bloomberg.com/robots.txt","https://web.archive.org/web/20250301223848/https://www.bloomberg.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /tc/search/"},{"t":"del","s":"Disallow: /en/news/thp"},{"t":"ctx","s":"Disallow: /subscriptions/group/manage/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /explore/"},{"t":"add","s":"Disallow: /press-releases/"},{"t":"add","s":"Disallow: /artemis/"},{"t":"add","s":"Disallow: /sessions-publisher/"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Mediapartners-Google"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# Development Tools"},{"t":"add","s":"User-agent: Python-urllib"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: python-requests"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Python-http-client"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# AI/LLM Bots"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: YouBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Google Services"},{"t":"add","s":"User-agent: Google-Apps"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Google-Apps-Script"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: AppEngine-Google"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Google-Cloud"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Google-CloudVertexBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.bloomberg.com/sitemap.xml"}]},{"domain":"bloomberg.com","name":"Bloomberg","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-11-01"],"summary":"Now blocked: GPTBot, Google-Extended and CCBot.","dir":"up","up":["GPTBot","Google-Extended","CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801001431/https://www.bloomberg.com/robots.txt","https://web.archive.org/web/20231101001611/https://www.bloomberg.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /tosv*.html"},{"t":"add","s":"Disallow: /search"},{"t":"add","s":"Disallow: /company/search/"},{"t":"add","s":"Disallow: /professional/search/"},{"t":"add","s":"Disallow: /impact/search/"},{"t":"add","s":"Disallow: /ux/search/"},{"t":"add","s":"Disallow: /wnwi/search/"},{"t":"add","s":"Disallow: /gei/search/"},{"t":"add","s":"Disallow: /impact/search/"},{"t":"add","s":"Disallow: /netzeropathfinders/search/"},{"t":"add","s":"Disallow: /notices/search/"},{"t":"add","s":"Disallow: /distribution/search/"},{"t":"add","s":"Disallow: /ukinnovators/search/"},{"t":"add","s":"Disallow: /latam/search/"},{"t":"add","s":"Disallow: /faq/search/"},{"t":"add","s":"Disallow: /tc/search/"},{"t":"add","s":"Disallow: /en/news/thp"},{"t":"add","s":"Disallow: /subscriptions/group/manage/"},{"t":"add","s":"Disallow: /preview/lineup"},{"t":"add","s":"Disallow: /preview/articles"},{"t":"add","s":"Disallow: /explore/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /apps/news"},{"t":"del","s":"Disallow: /search"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Twitterbot"},{"t":"add","s":"Allow: /en/news/thp"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.bloomberg.com/sitemap.xml"}]}]}