The Walled Web Newsletter

All sites / News and publishing

Bloomberg

bloomberg.com
Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. It keeps OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User out of parts of it. Perplexity-User can reach most of it. The homepage also showed our bot, which says who it is, a Generic bot challenge page instead of the site.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

7 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

3 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Some blocked

1 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI Blocked Named, line 120 Says yes
ClaudeBotAnthropic Blocked Named, line 127 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Blocked Named, line 34 Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Named, line 348 Says yes
CCBotCommon Crawl Blocked Named, line 63 Says yes
Meta-ExternalAgentMeta Blocked Named, line 342 Suggests yes
BytespiderByteDance Blocked Named, line 340 No statement found
MistralAI-TrainingMistral AI Allowed Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Allowed Not named; the catch-all rule applies Does not say
anthropic-aiAnthropic (legacy) · old name Blocked Named, line 82 Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI Some pages blocked Named, line 136 Says yes
Claude-SearchBotAnthropic Some pages blocked Named, line 182 Says yes
PerplexityBotPerplexity Blocked Named, line 89 Suggests yes
AmazonbotAmazon Blocked Named, line 336 Says yes
Amzn-SearchBotAmazon Allowed Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Blocked Named, line 103 Says yes
MistralAI-IndexMistral AI Allowed Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Allowed Not named; the catch-all rule applies Suggests yes
AI assistants · Some blocked
ChatGPT-UserOpenAI Some pages blocked Named, line 230 Says not always
Claude-UserAnthropic Some pages blocked Named, line 276 Says yes
Perplexity-UserPerplexity Allowed Not named; the catch-all rule applies Says it mostly ignores it
Google-AgentGoogle Allowed Not named; the catch-all rule applies Says it mostly ignores it
Meta-ExternalFetcherMeta Blocked Named, line 344 Says not always
Amzn-UserAmazon Allowed Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Allowed Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNot named; the catch-all rule applies
BingbotRegular search engine, for comparisonAllowedNot named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

Claude-WebBlockedline 75Not in our verified list
YouBotBlockedline 96Not in our verified list
cohere-aiBlockedline 110Not in our verified list
Google-CloudVertexBotBlockedline 332Documented by its operator
Meta-ExternalAgent-ImageBlockedline 346Not in our verified list
Bytespider-ImageBlockedline 350Not in our verified list
PetalBotBlockedline 412Not in our verified list
DiffbotBlockedline 467Documented by its operator

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 531 lines · 266 lines about AI bots Source
# Bot rules:# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.# If you can read this then you should apply here https://www.bloomberg.com/careers/User-agent: *Disallow: /polskaAllow: /account/newslettersDisallow: /account/*Disallow: /tosv*.htmlDisallow: /searchDisallow: /company/search/Disallow: /professional/search/Disallow: /impact/search/Disallow: /ux/search/Disallow: /wnwi/search/Disallow: /gei/search/Disallow: /impact/search/Disallow: /netzeropathfinders/search/Disallow: /notices/search/Disallow: /distribution/search/Disallow: /ukinnovators/search/Disallow: /latam/search/Disallow: /faq/search/Disallow: /tc/search/Disallow: /subscriptions/group/manage/Disallow: /preview/lineupDisallow: /preview/articlesDisallow: /explore/Disallow: /press-releases/Disallow: /artemis/Disallow: /sessions-publisher/… 32 lines not about AI bots User-agent: Google-ExtendedDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tc User-agent: Mediapartners-GoogleDisallow: /about/careersDisallow: /about/careers/Disallow: /offlinemessage/Disallow: /apps/fbkDisallow: /bb/newsarchive/Disallow: /apps/news User-agent: Spinn3rDisallow: /podcasts/Disallow: /feed/podcast/Disallow: /bb/avfile/ User-agent: Googlebot-NewsDisallow: /sponsor/Disallow: /news/sponsors/*Disallow: /news/terminal/* User-agent: TwitterbotAllow: /en/news/thp… 20 lines not about AI bots User-agent: CCBotDisallow: / # Development ToolsUser-agent: Python-urllibDisallow: /User-agent: python-requestsDisallow: /User-agent: Python-http-clientDisallow: / … 8 lines not about AI bots# AI/LLM BotsUser-agent: Claude-WebDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: anthropic-aiDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: PerplexityBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: YouBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: DuckAssistBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: cohere-aiDisallow: / # AI/LLM Bots - Homepage Access# Homepage and regional homepages only. The $ makes each Allow an exact match# so it beats "Disallow: /"; without it the rule is a prefix match and opens# everything under that route. /region/ is a 302 to /region and needs its own# rule, since robots.txt is checked before the redirect can be followed. … 6 lines not about AI bots# TrainingUser-agent: GPTBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: ClaudeBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tc # Search indexingUser-agent: OAI-SearchBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcAllow: /$Allow: /europe$Allow: /europe/$Allow: /asia$Allow: /asia/$Allow: /jp$Allow: /jp/$Allow: /india$Allow: /india/$Allow: /middleeast$Allow: /middleeast/$Allow: /africa$Allow: /africa/$Allow: /canada$Allow: /canada/$Allow: /uk$Allow: /uk/$Allow: /latinamerica$Allow: /latinamerica/$Allow: /llms.txt$Allow: /llms.txt/$Allow: /europe/llms.txt$Allow: /europe/llms.txt/$Allow: /asia/llms.txt$Allow: /asia/llms.txt/$Allow: /jp/llms.txt$Allow: /jp/llms.txt/$Allow: /india/llms.txt$Allow: /india/llms.txt/$Allow: /middleeast/llms.txt$Allow: /middleeast/llms.txt/$Allow: /africa/llms.txt$Allow: /africa/llms.txt/$Allow: /canada/llms.txt$Allow: /canada/llms.txt/$Allow: /uk/llms.txt$Allow: /uk/llms.txt/$Allow: /latinamerica/llms.txt$Allow: /latinamerica/llms.txt/$User-agent: Claude-SearchBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcAllow: /$Allow: /europe$Allow: /europe/$Allow: /asia$Allow: /asia/$Allow: /jp$Allow: /jp/$Allow: /india$Allow: /india/$Allow: /middleeast$Allow: /middleeast/$Allow: /africa$Allow: /africa/$Allow: /canada$Allow: /canada/$Allow: /uk$Allow: /uk/$Allow: /latinamerica$Allow: /latinamerica/$Allow: /llms.txt$Allow: /llms.txt/$Allow: /europe/llms.txt$Allow: /europe/llms.txt/$Allow: /asia/llms.txt$Allow: /asia/llms.txt/$Allow: /jp/llms.txt$Allow: /jp/llms.txt/$Allow: /india/llms.txt$Allow: /india/llms.txt/$Allow: /middleeast/llms.txt$Allow: /middleeast/llms.txt/$Allow: /africa/llms.txt$Allow: /africa/llms.txt/$Allow: /canada/llms.txt$Allow: /canada/llms.txt/$Allow: /uk/llms.txt$Allow: /uk/llms.txt/$Allow: /latinamerica/llms.txt$Allow: /latinamerica/llms.txt/$ # User-triggered fetchesUser-agent: ChatGPT-UserDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcAllow: /$Allow: /europe$Allow: /europe/$Allow: /asia$Allow: /asia/$Allow: /jp$Allow: /jp/$Allow: /india$Allow: /india/$Allow: /middleeast$Allow: /middleeast/$Allow: /africa$Allow: /africa/$Allow: /canada$Allow: /canada/$Allow: /uk$Allow: /uk/$Allow: /latinamerica$Allow: /latinamerica/$Allow: /llms.txt$Allow: /llms.txt/$Allow: /europe/llms.txt$Allow: /europe/llms.txt/$Allow: /asia/llms.txt$Allow: /asia/llms.txt/$Allow: /jp/llms.txt$Allow: /jp/llms.txt/$Allow: /india/llms.txt$Allow: /india/llms.txt/$Allow: /middleeast/llms.txt$Allow: /middleeast/llms.txt/$Allow: /africa/llms.txt$Allow: /africa/llms.txt/$Allow: /canada/llms.txt$Allow: /canada/llms.txt/$Allow: /uk/llms.txt$Allow: /uk/llms.txt/$Allow: /latinamerica/llms.txt$Allow: /latinamerica/llms.txt/$User-agent: Claude-UserDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcAllow: /$Allow: /europe$Allow: /europe/$Allow: /asia$Allow: /asia/$Allow: /jp$Allow: /jp/$Allow: /india$Allow: /india/$Allow: /middleeast$Allow: /middleeast/$Allow: /africa$Allow: /africa/$Allow: /canada$Allow: /canada/$Allow: /uk$Allow: /uk/$Allow: /latinamerica$Allow: /latinamerica/$Allow: /llms.txt$Allow: /llms.txt/$Allow: /europe/llms.txt$Allow: /europe/llms.txt/$Allow: /asia/llms.txt$Allow: /asia/llms.txt/$Allow: /jp/llms.txt$Allow: /jp/llms.txt/$Allow: /india/llms.txt$Allow: /india/llms.txt/$Allow: /middleeast/llms.txt$Allow: /middleeast/llms.txt/$Allow: /africa/llms.txt$Allow: /africa/llms.txt/$Allow: /canada/llms.txt$Allow: /canada/llms.txt/$Allow: /uk/llms.txt$Allow: /uk/llms.txt/$Allow: /latinamerica/llms.txt$Allow: /latinamerica/llms.txt/$ # Google ServicesUser-agent: Google-AppsDisallow: /User-agent: Google-Apps-ScriptDisallow: /User-agent: AppEngine-GoogleDisallow: /User-agent: Google-Cloud… 8 lines not about AI botsDisallow: /User-agent: Google-CloudVertexBotDisallow: / # Major Company BotsUser-agent: AmazonBotDisallow: /User-agent: AmazonAdBotDisallow: /User-agent: ByteSpiderDisallow: /User-agent: Meta-ExternalAgentDisallow: /User-agent: Meta-ExternalFetcherDisallow: /User-agent: Meta-ExternalAgent-ImageDisallow: /User-agent: Applebot-ExtendedDisallow: /User-agent: Bytespider-ImageDisallow: / # Feed & Content AggregatorsUser-agent: FeedlyDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: FeedlyBotDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: FeedlyAppDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: MWFeedParserDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tc # Analytics & MarketingUser-agent: comscoreDisallow: /User-agent: ComscoreDisallow: /User-agent: HubSpotDisallow: /User-agent: hubspotDisallow: /User-agent: CriteoDisallow: /User-agent: criteo-botDisallow: /User-agent: Peer39_CrawlerDisallow: /User-agent: rogerbotDisallow: /User-agent: linkdexbotDisallow: /User-agent: NTENTDisallow: /User-agent: AndersPinkBotDisallow: /User-agent: IndeedBotDisallow: / # Search Engine BotsUser-agent: MJ12bot… 58 lines not about AI botsDisallow: /User-agent: PetalBotDisallow: /User-agent: YisouSpiderDisallow: /User-agent: 360SpiderDisallow: /User-agent: QwantifyDisallow: /User-agent: ToutiaoSpiderDisallow: / # Monitoring & SecurityUser-agent: Screaming Frog SEO SpiderDisallow: /Allow: /professionalAllow: /companyAllow: /latamAllow: /faqAllow: /tcUser-agent: PRTGDisallow: /User-agent: FreshpingBotDisallow: /User-agent: PanoptaDisallow: /User-agent: DatadogSyntheticsDisallow: /User-agent: RackspaceDisallow: /User-agent: censysDisallow: /User-agent: burpDisallow: /User-agent: BurpDisallow: /User-agent: check_httpDisallow: /User-agent: DotcomMonitorDisallow: /User-agent: WatchSumoDisallow: /User-agent: WormlyBotDisallow: /User-agent: CalibreBotDisallow: /User-agent: AudistoBotDisallow: / # Social & ContentUser-agent: HatenaDisallow: /User-agent: Hatena-BookmarkDisallow: /User-agent: EveryoneSocialBot… 51 lines not about AI botsDisallow: /User-agent: DiffbotDisallow: /User-agent: CCBotDisallow: /User-agent: NetvibesDisallow: /User-agent: WebCEODisallow: /User-agent: PostanoDisallow: /User-agent: RebelMouseDisallow: /User-agent: Muck-RackDisallow: /User-agent: InstapaperViewerDisallow: /User-agent: TwurlyDisallow: /User-agent: LivelapBotDisallow: /User-agent: DatagnionBotDisallow: /User-agent: LinespiderDisallow: /User-agent: DiscourseDisallow: / # Potentially MaliciousUser-agent: MedusaDisallow: /User-agent: pingbackDisallow: /User-agent: WordPressDisallow: /User-agent: wp_pingDisallow: /User-agent: MauiBotDisallow: /User-agent: ltx71Disallow: /User-agent: WeSEEDisallow: /User-agent: halebotDisallow: /User-agent: BrightBotDisallow: / # Gaming/3DUser-agent: UnityPlayerDisallow: / # Sitemaps appSitemap: https://www.bloomberg.com/sitemaps/news/index.xmlSitemap: https://www.bloomberg.com/sitemaps/news/latest.xmlSitemap: https://www.bloomberg.com/sitemaps/collections/index.xmlSitemap: https://www.bloomberg.com/sitemaps/media/video/index.xmlSitemap: https://www.bloomberg.com/sitemaps/media/audio/index.xmlSitemap: https://www.bloomberg.com/sitemaps/people/profiles/index.xmlSitemap: https://www.bloomberg.com/sitemaps/companies/public-company/index.xmlSitemap: https://www.bloomberg.com/sitemaps/companies/private-company/index.xmlSitemap: https://www.bloomberg.com/sitemaps/securites/index.xml # Billionaires, owned by graphicsSitemap: https://www.bloomberg.com/billionaires/sitemap.xml … 60 lines not about AI bots

Notes about AI written in the file: "AI/LLM Bots" · "AI/LLM Bots - Homepage Access" · "Training"

Everything else we checked

Front doorHuman check The homepage showed our bot, which says who it is, a Generic bot challenge page instead of the site (HTTP 403).GET https://www.bloomberg.com/ as WalledWebBot, from a residential connection in the United States
llms.txtRefused (HTTP 403)https://www.bloomberg.com/llms.txt
ai.txtRefused (HTTP 403)
Page tagsHomepage not read.
Anti-bot softwareThe homepage set a HUMAN (PerimeterX) cookie (_pxhd), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIHomepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of ServiceNo Terms of Service page found.
Not foundNo tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

47 versions on record since 1 Jan 2023

OverallJan 2023: Overall allowedFeb 2023: Overall allowedMar 2023: Overall allowedApr 2023: Overall allowedMay 2023: Overall allowedJun 2023: Overall allowedJul 2023: Overall allowedAug 2023: Overall allowedSep 2023: Overall partialOct 2023: Overall partialNov 2023: Overall partialDec 2023: Overall partialJan 2024: Overall partialFeb 2024: Overall partialMar 2024: Overall partialApr 2024: Overall partialMay 2024: Overall partialJun 2024: Overall partialJul 2024: Overall partialAug 2024: Overall partialSep 2024: Overall partialOct 2024: Overall partialNov 2024: Overall partialDec 2024: Overall partialJan 2025: Overall partialFeb 2025: Overall partialMar 2025: Overall partialApr 2025: Overall partialMay 2025: Overall partialJun 2025: Overall partialJul 2025: Overall partialAug 2025: Overall partialSep 2025: Overall partialOct 2025: Overall partialNov 2025: Overall partialDec 2025: Overall partialJan 2026: Overall partialFeb 2026: Overall partialMar 2026: Overall partialApr 2026: Overall partialMay 2026: Overall partialJun 2026: Overall partialJul 2026: Overall partialAug 2026: Overall partialSep 2026: Overall partialOct 2026: Overall partialTrainingJan 2023: Training AllowedFeb 2023: Training AllowedMar 2023: Training AllowedApr 2023: Training AllowedMay 2023: Training AllowedJun 2023: Training AllowedJul 2023: Training AllowedAug 2023: Training AllowedSep 2023: Training Some blockedOct 2023: Training Some blockedNov 2023: Training Some blockedDec 2023: Training Some blockedJan 2024: Training Some blockedFeb 2024: Training Some blockedMar 2024: Training Some blockedApr 2024: Training Some blockedMay 2024: Training Some blockedJun 2024: Training Some blockedJul 2024: Training Some blockedAug 2024: Training Some blockedSep 2024: Training Some blockedOct 2024: Training Some blockedNov 2024: Training Some blockedDec 2024: Training Some blockedJan 2025: Training Some blockedFeb 2025: Training Some blockedMar 2025: Training Some blockedApr 2025: Training Some blockedMay 2025: Training Some blockedJun 2025: Training Some blockedJul 2025: Training Some blockedAug 2025: Training Some blockedSep 2025: Training Some blockedOct 2025: Training Some blockedNov 2025: Training Some blockedDec 2025: Training Some blockedJan 2026: Training Some blockedFeb 2026: Training Some blockedMar 2026: Training Some blockedApr 2026: Training Some blockedMay 2026: Training Some blockedJun 2026: Training Some blockedJul 2026: Training Some blockedAug 2026: Training Some blockedSep 2026: Training Some blockedOct 2026: Training Some blockedSearchJan 2023: Search AllowedFeb 2023: Search AllowedMar 2023: Search AllowedApr 2023: Search AllowedMay 2023: Search AllowedJun 2023: Search AllowedJul 2023: Search AllowedAug 2023: Search AllowedSep 2023: Search AllowedOct 2023: Search AllowedNov 2023: Search AllowedDec 2023: Search AllowedJan 2024: Search AllowedFeb 2024: Search AllowedMar 2024: Search AllowedApr 2024: Search AllowedMay 2024: Search AllowedJun 2024: Search AllowedJul 2024: Search AllowedAug 2024: Search AllowedSep 2024: Search AllowedOct 2024: Search AllowedNov 2024: Search AllowedDec 2024: Search AllowedJan 2025: Search AllowedFeb 2025: Search AllowedMar 2025: Search Some blockedApr 2025: Search Some blockedMay 2025: Search Some blockedJun 2025: Search Some blockedJul 2025: Search Some blockedAug 2025: Search Some blockedSep 2025: Search Some blockedOct 2025: Search Some blockedNov 2025: Search Some blockedDec 2025: Search Some blockedJan 2026: Search Some blockedFeb 2026: Search Some blockedMar 2026: Search Some blockedApr 2026: Search Some blockedMay 2026: Search Some blockedJun 2026: Search Some blockedJul 2026: Search Some blockedAug 2026: Search Some blockedSep 2026: Search Some blockedOct 2026: Search Some blockedAssistantsJan 2023: Assistants AllowedFeb 2023: Assistants AllowedMar 2023: Assistants AllowedApr 2023: Assistants AllowedMay 2023: Assistants AllowedJun 2023: Assistants AllowedJul 2023: Assistants AllowedAug 2023: Assistants AllowedSep 2023: Assistants AllowedOct 2023: Assistants AllowedNov 2023: Assistants AllowedDec 2023: Assistants AllowedJan 2024: Assistants AllowedFeb 2024: Assistants AllowedMar 2024: Assistants AllowedApr 2024: Assistants AllowedMay 2024: Assistants AllowedJun 2024: Assistants AllowedJul 2024: Assistants AllowedAug 2024: Assistants AllowedSep 2024: Assistants AllowedOct 2024: Assistants AllowedNov 2024: Assistants AllowedDec 2024: Assistants AllowedJan 2025: Assistants AllowedFeb 2025: Assistants AllowedMar 2025: Assistants Some blockedApr 2025: Assistants Some blockedMay 2025: Assistants Some blockedJun 2025: Assistants Some blockedJul 2025: Assistants Some blockedAug 2025: Assistants Some blockedSep 2025: Assistants Some blockedOct 2025: Assistants Some blockedNov 2025: Assistants Some blockedDec 2025: Assistants Some blockedJan 2026: Assistants Some blockedFeb 2026: Assistants Some blockedMar 2026: Assistants Some blockedApr 2026: Assistants Some blockedMay 2026: Assistants Some blockedJun 2026: Assistants Some blockedJul 2026: Assistants Some blockedAug 2026: Assistants Some blockedSep 2026: Assistants Some blockedOct 2026: Assistants Some blockedBotsblockedSep 2023: 1 of 24 AI bots blockedOct 2023: 2 of 24 AI bots blockedNov 2023: 3 of 24 AI bots blockedDec 2023: 3 of 24 AI bots blockedJan 2024: 3 of 24 AI bots blockedFeb 2024: 3 of 24 AI bots blockedMar 2024: 3 of 24 AI bots blockedApr 2024: 3 of 24 AI bots blockedMay 2024: 3 of 24 AI bots blockedJun 2024: 3 of 24 AI bots blockedJul 2024: 3 of 24 AI bots blockedAug 2024: 3 of 24 AI bots blockedSep 2024: 3 of 24 AI bots blockedOct 2024: 3 of 24 AI bots blockedNov 2024: 3 of 24 AI bots blockedDec 2024: 3 of 24 AI bots blockedJan 2025: 3 of 24 AI bots blockedFeb 2025: 3 of 24 AI bots blockedMar 2025: 5 of 24 AI bots blockedApr 2025: 5 of 24 AI bots blockedMay 2025: 5 of 24 AI bots blockedJun 2025: 5 of 24 AI bots blockedJul 2025: 5 of 24 AI bots blockedAug 2025: 8 of 24 AI bots blockedSep 2025: 8 of 24 AI bots blockedOct 2025: 8 of 24 AI bots blockedNov 2025: 8 of 24 AI bots blockedDec 2025: 8 of 24 AI bots blockedJan 2026: 8 of 24 AI bots blockedFeb 2026: 8 of 24 AI bots blockedMar 2026: 8 of 24 AI bots blockedApr 2026: 8 of 24 AI bots blockedMay 2026: 8 of 24 AI bots blockedJun 2026: 15 of 24 AI bots blockedJul 2026: 15 of 24 AI bots blockedAug 2026: 15 of 24 AI bots blockedSep 2026: 15 of 24 AI bots blockedOct 2026: 11 of 24 AI bots blocked112023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Oct 2026
    Less blocking Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.
    Found in the Internet Archive's copies: changed between 11 Sep 2026 and 1 Oct 2026. Evidence: before · after
    Show exactly what changed
    - User-agent: GPTBot
    - Disallow: /
    -
    User-agent: CCBot
    …
    - User-agent: Google-Extended
    - Disallow: /
    -
    # Development Tools
    …
    # AI/LLM Bots
    - User-agent: ChatGPT-User
    + User-agent: Claude-Web
    Disallow: /
    …
    Allow: /tc
    - User-agent: GPTBot
    + User-agent: anthropic-ai
    Disallow: /
    …
    Allow: /tc
    - User-agent: Claude-Web
    + User-agent: PerplexityBot
    Disallow: /
    …
    Allow: /tc
    - User-agent: anthropic-ai
    + User-agent: YouBot
    Disallow: /
    …
    Allow: /tc
    - User-agent: PerplexityBot
    + User-agent: DuckAssistBot
    Disallow: /
    …
    Allow: /tc
    - User-agent: YouBot
    + User-agent: cohere-ai
    Disallow: /
    +
    + # AI/LLM Bots - Homepage Access
    + # Homepage and regional homepages only. The $ makes each Allow an exact match
    + # so it beats "Disallow: /"; without it the rule is a prefix match and opens
    + # everything under that route. /region/ is a 302 to /region and needs its own
    + # rule, since robots.txt is checked before the redirect can be followed.
    +
    + # Training
    + User-agent: GPTBot
    + Disallow: /
    Allow: /professional
    …
    Disallow: /
    - User-agent: Claude-User
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    … 152 more lines
  • Jun 2026
    More blocking Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
    Found in the Internet Archive's copies: changed between 1 May 2026 and 2 Jun 2026. Evidence: before · after
    Show exactly what changed
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    …
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    User-agent: GPTBot
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    User-agent: Claude-Web
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    User-agent: anthropic-ai
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    User-agent: PerplexityBot
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    User-agent: YouBot
    Disallow: /
    + Allow: /professional
    + Allow: /company
    + Allow: /latam
    + Allow: /faq
    + Allow: /tc
    + User-agent: ClaudeBot
    + Disallow: /
    + User-agent: Claude-User
    + Disallow: /
    + User-agent: Claude-SearchBot
    + Disallow: /
    + User-agent: OAI-SearchBot
    + Disallow: /
    + User-agent: DuckAssistBot
    + Disallow: /
    + User-agent: cohere-ai
    … 21 more lines
  • Aug 2025
    More blocking Now blocked: Meta-ExternalAgent, Bytespider and Amazonbot.
    Found in the Internet Archive's copies: changed between 1 Jul 2025 and 1 Aug 2025. Evidence: before · after
    Show exactly what changed
    - Sitemap: https://www.bloomberg.com/sitemap.xml
    - Sitemap: https://www.bloomberg.com/feeds/sitemap_news.xml
    - Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/bpol/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/businessweek/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/citylab/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/green/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/technology/sitemap_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_securities_index.xml
    - Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_profiles_company_index.xml
    + # Major Company Bots
    + User-agent: AmazonBot
    + Disallow: /
    + User-agent: AmazonAdBot
    + Disallow: /
    + User-agent: ByteSpider
    + Disallow: /
    + User-agent: Meta-ExternalAgent
    + Disallow: /
    +
    + # Feed & Content Aggregators
    + User-agent: Feedly
    + Disallow: /
    + User-agent: FeedlyBot
    + Disallow: /
    + User-agent: FeedlyApp
    + Disallow: /
    + User-agent: MWFeedParser
    + Disallow: /
    +
    + # Analytics & Marketing
    + User-agent: comscore
    + Disallow: /
    + User-agent: Comscore
    + Disallow: /
    + User-agent: HubSpot
    + Disallow: /
    + User-agent: hubspot
    + Disallow: /
    + User-agent: Criteo
    + Disallow: /
    + User-agent: criteo-bot
    + Disallow: /
    + User-agent: Peer39_Crawler
    + Disallow: /
    + User-agent: rogerbot
    + Disallow: /
    + User-agent: linkdexbot
    + Disallow: /
    + User-agent: NTENT
    + Disallow: /
    + User-agent: AndersPinkBot
    + Disallow: /
    + User-agent: IndeedBot
    + Disallow: /
    +
    + # Search Engine Bots
    + User-agent: MJ12bot
    + Disallow: /
    … 126 more lines
  • Mar 2025
    More blocking Now blocked: anthropic-ai, PerplexityBot and ChatGPT-User.
    Found in the Internet Archive's copies: changed between 1 Feb 2025 and 1 Mar 2025. Evidence: before · after
    Show exactly what changed
    Disallow: /tc/search/
    - Disallow: /en/news/thp
    Disallow: /subscriptions/group/manage/
    …
    Disallow: /explore/
    + Disallow: /press-releases/
    + Disallow: /artemis/
    + Disallow: /sessions-publisher/
    + User-agent: Google-Extended
    + Disallow: /
    +
    User-agent: Mediapartners-Google
    …
    + # Development Tools
    + User-agent: Python-urllib
    + Disallow: /
    + User-agent: python-requests
    + Disallow: /
    + User-agent: Python-http-client
    + Disallow: /
    +
    + # AI/LLM Bots
    + User-agent: ChatGPT-User
    + Disallow: /
    + User-agent: GPTBot
    + Disallow: /
    + User-agent: Claude-Web
    + Disallow: /
    + User-agent: anthropic-ai
    + Disallow: /
    + User-agent: PerplexityBot
    + Disallow: /
    + User-agent: YouBot
    + Disallow: /
    +
    + # Google Services
    + User-agent: Google-Apps
    + Disallow: /
    + User-agent: Google-Apps-Script
    + Disallow: /
    + User-agent: AppEngine-Google
    + Disallow: /
    + User-agent: Google-Cloud
    + Disallow: /
    + User-agent: Google-CloudVertexBot
    + Disallow: /
    +
    Sitemap: https://www.bloomberg.com/sitemap.xml
  • Aug to Nov 2023
    More blocking Now blocked: GPTBot, Google-Extended and CCBot.
    Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Nov 2023. Evidence: before · after
    Show exactly what changed
    Disallow: /tosv*.html
    + Disallow: /search
    + Disallow: /company/search/
    + Disallow: /professional/search/
    + Disallow: /impact/search/
    + Disallow: /ux/search/
    + Disallow: /wnwi/search/
    + Disallow: /gei/search/
    + Disallow: /impact/search/
    + Disallow: /netzeropathfinders/search/
    + Disallow: /notices/search/
    + Disallow: /distribution/search/
    + Disallow: /ukinnovators/search/
    + Disallow: /latam/search/
    + Disallow: /faq/search/
    + Disallow: /tc/search/
    + Disallow: /en/news/thp
    + Disallow: /subscriptions/group/manage/
    + Disallow: /preview/lineup
    + Disallow: /preview/articles
    + Disallow: /explore/
    …
    Disallow: /apps/news
    - Disallow: /search
    …
    + User-agent: Twitterbot
    + Allow: /en/news/thp
    +
    + User-agent: GPTBot
    + Disallow: /
    +
    + User-agent: CCBot
    + Disallow: /
    +
    + User-agent: Google-Extended
    + Disallow: /
    +
    Sitemap: https://www.bloomberg.com/sitemap.xml

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.