All sites / News and publishing
BBC
bbc.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 14 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Claude-SearchBot and Claude-User can reach most of it.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
7 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
4 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
3 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 153 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 132 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 163 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 150 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 120 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 182 | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 141 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 135 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 168 | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 173 | Suggests yes |
| AmazonbotAmazon | Blocked | Named, line 114 | Says yes |
| Amzn-SearchBotAmazon | Blocked | Named, line 224 | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 158 | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 176 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Blocked | Named, line 227 | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| omgili | Blocked | line 123 | Not in our verified list |
| omgilibot | Blocked | line 126 | Not in our verified list |
| Claude-Web | Blocked | line 129 | Not in our verified list |
| cohere-ai | Blocked | line 138 | Not in our verified list |
| PetalBot | Blocked | line 144 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 179 | Documented by its operator |
| Diffbot | Blocked | line 203 | Documented by its operator |
| Diffbot-User | Blocked | line 206 | Documented by its operator |
| YouBot | Blocked | line 218 | Not in our verified list |
| FirecrawlAgent | Blocked | line 221 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# version: 20b91abf30af3b3b076745396ba368978384f4d7# The BBC's Terms of Use: https://www.bbc.co.uk/terms# - Explain the rules for using our services# - Tell you what you can do with our content## In short: Please use our site like a human, not a robot.# That means:# - No scraping, crawling, or systematic extraction of content # - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content# - No creating datasets from BBC content# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market# - No using BBC content to create summaries for your own use# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law # TL;DR: Browse, read, watch, enjoy - like a human.# # HTTPS www.bbc.com User-agent: *Sitemap: https://www.bbc.com/afrique/sitemap.xmlSitemap: https://www.bbc.com/arabic/sitemap.xmlSitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xmlSitemap: https://www.bbc.com/bbcx/content_index_sitemap.xmlSitemap: https://www.bbc.com/bbcx/index_sitemap.xmlSitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xmlSitemap: https://www.bbc.com/bengali/sitemap.xmlSitemap: https://www.bbc.com/burmese/sitemap.xmlSitemap: https://www.bbc.com/gahuza/sitemap.xmlSitemap: https://www.bbc.com/hausa/sitemap.xmlSitemap: https://www.bbc.com/hindi/sitemap.xmlSitemap: https://www.bbc.com/indonesia/sitemap.xmlSitemap: https://www.bbc.com/mundo/sitemap.xmlSitemap: https://www.bbc.com/pashto/sitemap.xmlSitemap: https://www.bbc.com/persian/sitemap.xmlSitemap: https://www.bbc.com/portuguese/sitemap.xmlSitemap: https://www.bbc.com/russian/sitemap.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml Sitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-video.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-news.xmlSitemap: https://www.bbc.com/sitemaps/https-index-com-video.xmlSitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xmlSitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xmlSitemap: https://www.bbc.com/sitemaps/world-service-homepages.xmlSitemap: https://www.bbc.com/sport/sitemap.xmlSitemap: https://www.bbc.com/swahili/sitemap.xmlSitemap: https://www.bbc.com/tajik/sitemap.xmlSitemap: https://www.bbc.com/turkce/sitemap.xmlSitemap: https://www.bbc.com/ukchina/simp/sitemap.xmlSitemap: https://www.bbc.com/ukrainian/sitemap.xmlSitemap: https://www.bbc.com/urdu/sitemap.xmlSitemap: https://www.bbc.com/uzbek/sitemap.xmlSitemap: https://www.bbc.com/vietnamese/sitemap.xmlSitemap: https://www.bbc.com/zhongwen/simp/sitemap.xmlSitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml Disallow: /asset/Disallow: /backstage/bbc-login-help/Disallow: /backstage/bbc-login-help$Disallow: /bitesize/search$Disallow: /bitesize/search/Disallow: /bitesize/search?Disallow: /cbbc/search/Disallow: /cbbc/search$Disallow: /cbbc/search?Disallow: /cbeebies/search/Disallow: /cbeebies/search$Disallow: /cbeebies/search?Disallow: /chwilio/Disallow: /chwilio$Disallow: /chwilio?Disallow: /education/blocks$Disallow: /education/blocks/Disallow: /newsroundDisallow: /search/Disallow: /search$Disallow: /search?Disallow: /food/favouritesDisallow: /food/search*?*Disallow: /food/recipes/search*?*Disallow: /education/my$Disallow: /education/my/Disallow: /bitesize/my$Disallow: /bitesize/my/Disallow: /food/recipes/*/shopping-listDisallow: /food/menus/*/shopping-listDisallow: /news/0Disallow: /sport/alpha/Disallow: /ugc$Disallow: /ugc/Disallow: /ugcsupport$Disallow: /ugcsupport/Disallow: /userinfo/Disallow: /userinfoDisallow: /u5llnop$Disallow: /u5llnop/Disallow: /sounds/search$Disallow: /sounds/search/Disallow: /sounds/search?Disallow: /ws/includesDisallow: /radio/imdaDisallow: /storyworks/preview/*Disallow: /rd/search$Disallow: /rd/search/Disallow: /rd/search?… 112 lines not about AI bots User-agent: AmazonbotDisallow: / User-agent: magpie-crawlerDisallow: /… 2 lines not about AI bots User-agent: CCBotDisallow: / User-Agent: omgiliDisallow: / User-Agent: omgilibotDisallow: / User-agent: Claude-WebDisallow: / User-agent: ClaudeBotDisallow: / User-agent: anthropic-aiDisallow: / User-agent: cohere-aiDisallow: / User-agent: BytespiderDisallow: / User-agent: PetalBotDisallow: / User-agent: ScrapyDisallow: /… 2 lines not about AI bots User-agent: Applebot-ExtendedDisallow: / User-agent: GPTBotDisallow: /Allow: /storyworksAllow: /storyworks/ User-agent: ChatGPT-UserDisallow: /Allow: /storyworksAllow: /storyworks/ User-agent: Google-ExtendedDisallow: /Allow: /storyworksAllow: /storyworks/ User-agent: OAI-SearchBotDisallow: /Allow: /storyworksAllow: /storyworks/ User-Agent: PerplexityBotDisallow: / User-agent: Perplexity-UserDisallow: / User-agent: Google-CloudVertexBotDisallow: / User-agent: meta-externalagentDisallow: / User-agent: YandexAdditionalDisallow: / User-agent: YandexAdditionalBotDisallow: / User-agent: TurnitinBotDisallow: / User-agent: BrightbotDisallow: / User-agent: ApifyBotDisallow: / User-agent: ApifyWebsiteContentCrawlerDisallow: /… 17 lines not about AI bots User-agent: DiffbotDisallow: / User-agent: Diffbot-UserDisallow: / User-agent: ExaBotDisallow: / User-agent: TavilyBotDisallow: / User-agent: ShapBotDisallow: /… 8 lines not about AI bots User-agent: YouBotDisallow: / User-agent: FirecrawlAgentDisallow: / User-agent: Amzn-SearchBotDisallow: / User-agent: Amzn-UserDisallow: / User-agent: ProRataIncDisallow: / User-agent: CloudflareBrowserRenderingCrawlerDisallow: / User-agent: AhrefsBotDisallow: / … 9 lines not about AI bots
Notes about AI written in the file: "- No scraping, crawling, or systematic extraction of content" · "- No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)" · "- No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content"
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.bbc.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Page tags | <meta name="robots" content="NOODP, NOYDIR"> X-Robots-Tag: bingbot: noarchive |
| Anti-bot software | Served through Fastly.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: WebPage, NewsMediaOrganization. Most of the page is script; an agent that does not run JavaScript sees little text. Built on Next.js.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | www.bbc.com/pages/terms-of-use (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context.
|
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Jan 2026More blocking Now blocked: Amzn-SearchBot and Amzn-User.Found in the Internet Archive's copies: changed between 1 Dec 2025 and 1 Jan 2026. Evidence: before · after
Show exactly what changed
- # version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f+ # version: 8c1c7053723d6bd586586053ecdd261591364cfe…Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml…+ User-agent: Amzn-SearchBot+ Disallow: /++ User-agent: Amzn-User+ Disallow: / - Apr to Aug 2025More blocking Now blocked: Amazonbot and Perplexity-User.Found in the Internet Archive's copies: changed between 1 Apr 2025 and 1 Aug 2025. Evidence: before · after
Show exactly what changed
- # version: ca16af0d01ecf92c8730f124f789053bfbf7293b-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml- Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml-+ # version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search- Disallow: /bengali/search- Disallow: /burmese/search- Disallow: /cbbc/search- Disallow: /cbeebies/search- Disallow: /cgi-bin- Disallow: /cgi-perl- Disallow: /cgi-perlx- Disallow: /cgi-store- Disallow: /corporate2/bbctrust/search- Disallow: /corporate2/insidethebbc/search- Disallow: /corporate2/mediacentre/proginfo/search- Disallow: /corporate2/mediacentre/search- Disallow: /corporate2/safety/search- Disallow: /eastenders/updates/- Disallow: /education$- Disallow: /education/- Disallow: /education/bitesize- Disallow: /education/dev- Disallow: /education/images- Disallow: /education/nav… 745 more lines - Mar 2025More blocking Now blocked: Meta-ExternalAgent and OAI-SearchBot.Found in the Internet Archive's copies: changed between 1 Feb 2025 and 2 Mar 2025. Evidence: before · after
Show exactly what changed
- # version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68+ # version: ca16af0d01ecf92c8730f124f789053bfbf7293b- # HTTPS www.bbc.com+ # HTTP www.bbc.com- User-agent: *- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/sport/sitemap.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/afrique/sitemap.xml- Sitemap: https://www.bbc.com/arabic/sitemap.xml- Sitemap: https://www.bbc.com/bengali/sitemap.xml- Sitemap: https://www.bbc.com/burmese/sitemap.xml- Sitemap: https://www.bbc.com/gahuza/sitemap.xml- Sitemap: https://www.bbc.com/hausa/sitemap.xml- Sitemap: https://www.bbc.com/hindi/sitemap.xml- Sitemap: https://www.bbc.com/indonesia/sitemap.xml- Sitemap: https://www.bbc.com/mundo/sitemap.xml- Sitemap: https://www.bbc.com/pashto/sitemap.xml- Sitemap: https://www.bbc.com/persian/sitemap.xml- Sitemap: https://www.bbc.com/portuguese/sitemap.xml- Sitemap: https://www.bbc.com/russian/sitemap.xml- Sitemap: https://www.bbc.com/swahili/sitemap.xml- Sitemap: https://www.bbc.com/tajik/sitemap.xml- Sitemap: https://www.bbc.com/turkce/sitemap.xml- Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml- Sitemap: https://www.bbc.com/ukrainian/sitemap.xml- Sitemap: https://www.bbc.com/urdu/sitemap.xml- Sitemap: https://www.bbc.com/uzbek/sitemap.xml- Sitemap: https://www.bbc.com/vietnamese/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml- Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml+ User-agent: Googlebot+ Sitemap: http://www.bbc.com/sitemap.xml+ Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml+ Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml+ Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml- Disallow: /bitesize/search$- Disallow: /bitesize/search/- Disallow: /bitesize/search?- Disallow: /cbbc/search/- Disallow: /cbbc/search$- Disallow: /cbbc/search?- Disallow: /cbeebies/search/- Disallow: /cbeebies/search$- Disallow: /cbeebies/search?- Disallow: /chwilio/- Disallow: /chwilio$- Disallow: /chwilio?++ Disallow: /iplayer/episode/*?from=r*+ Disallow: /iplayer/cy/episode/*?from=r*… 785 more lines - Oct 2024More blocking Now blocked: ClaudeBot and Applebot-Extended.Found in the Internet Archive's copies: changed between 1 Aug 2024 and 1 Oct 2024. Evidence: before · after
Show exactly what changed
- # version: 03e3d0d3861e30b21826aa11558f45235a4d4143-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml- Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml-+ # version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search- Disallow: /bengali/search- Disallow: /burmese/search- Disallow: /cbbc/search- Disallow: /cbeebies/search- Disallow: /cgi-bin- Disallow: /cgi-perl- Disallow: /cgi-perlx- Disallow: /cgi-store- Disallow: /corporate2/bbctrust/search- Disallow: /corporate2/insidethebbc/search- Disallow: /corporate2/mediacentre/proginfo/search- Disallow: /corporate2/mediacentre/search- Disallow: /corporate2/safety/search- Disallow: /eastenders/updates/- Disallow: /education$- Disallow: /education/- Disallow: /education/bitesize- Disallow: /education/dev- Disallow: /education/images- Disallow: /education/nav… 733 more lines - May 2024More blocking Now blocked: Bytespider and PerplexityBot.Found in the Internet Archive's copies: changed between 1 Apr 2024 and 1 May 2024. Evidence: before · after
Show exactly what changed
- # version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053+ # version: 03e3d0d3861e30b21826aa11558f45235a4d4143- # HTTPS www.bbc.com+ # HTTP www.bbc.com- User-agent: *- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/sport/sitemap.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/afrique/sitemap.xml- Sitemap: https://www.bbc.com/arabic/sitemap.xml- Sitemap: https://www.bbc.com/bengali/sitemap.xml- Sitemap: https://www.bbc.com/burmese/sitemap.xml- Sitemap: https://www.bbc.com/gahuza/sitemap.xml- Sitemap: https://www.bbc.com/hausa/sitemap.xml- Sitemap: https://www.bbc.com/hindi/sitemap.xml- Sitemap: https://www.bbc.com/indonesia/sitemap.xml- Sitemap: https://www.bbc.com/mundo/sitemap.xml- Sitemap: https://www.bbc.com/pashto/sitemap.xml- Sitemap: https://www.bbc.com/persian/sitemap.xml- Sitemap: https://www.bbc.com/portuguese/sitemap.xml- Sitemap: https://www.bbc.com/russian/sitemap.xml- Sitemap: https://www.bbc.com/swahili/sitemap.xml- Sitemap: https://www.bbc.com/tajik/sitemap.xml- Sitemap: https://www.bbc.com/turkce/sitemap.xml- Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml- Sitemap: https://www.bbc.com/ukrainian/sitemap.xml- Sitemap: https://www.bbc.com/urdu/sitemap.xml- Sitemap: https://www.bbc.com/uzbek/sitemap.xml- Sitemap: https://www.bbc.com/vietnamese/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml- Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml+ User-agent: Googlebot+ Sitemap: http://www.bbc.com/sitemap.xml+ Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml+ Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml+ Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml- Disallow: /bitesize/search$- Disallow: /bitesize/search/- Disallow: /bitesize/search?- Disallow: /cbbc/search/- Disallow: /cbbc/search$- Disallow: /cbbc/search?- Disallow: /cbeebies/search/- Disallow: /cbeebies/search$- Disallow: /cbeebies/search?- Disallow: /chwilio/- Disallow: /chwilio$- Disallow: /chwilio?++ Disallow: /iplayer/episode/*?from=r*+ Disallow: /iplayer/cy/episode/*?from=r*… 730 more lines - Feb 2024More blocking Now blocked: Google-Extended and anthropic-ai.Found in the Internet Archive's copies: changed between 1 Dec 2023 and 1 Feb 2024. Evidence: before · after
Show exactly what changed
- # version: f6f959638af29cbc598b7adab27fd84e19d0d062+ # version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053…Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml…Disallow: /news/0+ Disallow: /sport/alpha/Disallow: /ugc$…Disallow: /radio/imda+ Disallow: /storyworks/preview/*…Disallow: /++ User-Agent: omgili+ Disallow: /++ User-Agent: omgilibot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Google-Extended+ Disallow: / - Nov 2023More blocking Now blocked: GPTBot, CCBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after
Show exactly what changed
- # version: 587aced11b074061603bab7e2b21f94ae7f96821-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml--- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search- Disallow: /bengali/search- Disallow: /burmese/search- Disallow: /cbbc/search- Disallow: /cbeebies/search- Disallow: /cgi-bin- Disallow: /cgi-perl- Disallow: /cgi-perlx- Disallow: /cgi-store- Disallow: /corporate2/bbctrust/search- Disallow: /corporate2/insidethebbc/search- Disallow: /corporate2/mediacentre/proginfo/search- Disallow: /corporate2/mediacentre/search- Disallow: /corporate2/safety/search- Disallow: /eastenders/updates/- Disallow: /education$- Disallow: /education/- Disallow: /education/bitesize- Disallow: /education/dev- Disallow: /education/images- Disallow: /education/nav- Disallow: /education/navigation- Disallow: /education/ximages… 706 more lines
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.