{"domain":"bbc.com","name":"BBC","category":"news","subcategory":"broadcaster","country":"GB","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 14 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Claude-SearchBot and Claude-User can reach most of it.","computed_at":"2026-10-20T03:43:15.177Z","last_checked":"2026-10-19T13:34:17.699Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.bbc.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.bbc.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.bbc.com/robots.txt","hops":[{"from":"https://bbc.com/robots.txt","to":"https://www.bbc.com/robots.txt","status":301}],"blob":"37a8567b049d9a34ca7b7dcb420ed3b8cbd929f1720a1c57c03ad28a1efec0b7","text":"\n# version: 20b91abf30af3b3b076745396ba368978384f4d7\n# The BBC's Terms of Use: https://www.bbc.co.uk/terms\n# - Explain the rules for using our services\n# - Tell you what you can do with our content\n#\n# In short: Please use our site like a human, not a robot.\n# That means:\n# - No scraping, crawling, or systematic extraction of content \n# - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)\n# - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content\n# - No creating datasets from BBC content\n# - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market\n# - No using BBC content to create summaries for your own use\n# - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/)\n# - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law\n \n# TL;DR: Browse, read, watch, enjoy - like a human.\n#\n\n# HTTPS www.bbc.com\n\nUser-agent: *\nSitemap: https://www.bbc.com/afrique/sitemap.xml\nSitemap: https://www.bbc.com/arabic/sitemap.xml\nSitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml\nSitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml\nSitemap: https://www.bbc.com/bbcx/index_sitemap.xml\nSitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml\nSitemap: https://www.bbc.com/bengali/sitemap.xml\nSitemap: https://www.bbc.com/burmese/sitemap.xml\nSitemap: https://www.bbc.com/gahuza/sitemap.xml\nSitemap: https://www.bbc.com/hausa/sitemap.xml\nSitemap: https://www.bbc.com/hindi/sitemap.xml\nSitemap: https://www.bbc.com/indonesia/sitemap.xml\nSitemap: https://www.bbc.com/mundo/sitemap.xml\nSitemap: https://www.bbc.com/pashto/sitemap.xml\nSitemap: https://www.bbc.com/persian/sitemap.xml\nSitemap: https://www.bbc.com/portuguese/sitemap.xml\nSitemap: https://www.bbc.com/russian/sitemap.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml \nSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml \nSitemap: https://www.bbc.com/sitemaps/https-index-com-mundo-video.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml\nSitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml\nSitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml\nSitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml\nSitemap: https://www.bbc.com/sitemaps/world-service-homepages.xml\nSitemap: https://www.bbc.com/sport/sitemap.xml\nSitemap: https://www.bbc.com/swahili/sitemap.xml\nSitemap: https://www.bbc.com/tajik/sitemap.xml\nSitemap: https://www.bbc.com/turkce/sitemap.xml\nSitemap: https://www.bbc.com/ukchina/simp/sitemap.xml\nSitemap: https://www.bbc.com/ukrainian/sitemap.xml\nSitemap: https://www.bbc.com/urdu/sitemap.xml\nSitemap: https://www.bbc.com/uzbek/sitemap.xml\nSitemap: https://www.bbc.com/vietnamese/sitemap.xml\nSitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml\nSitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml\n\nDisallow: /asset/\nDisallow: /backstage/bbc-login-help/\nDisallow: /backstage/bbc-login-help$\nDisallow: /bitesize/search$\nDisallow: /bitesize/search/\nDisallow: /bitesize/search?\nDisallow: /cbbc/search/\nDisallow: /cbbc/search$\nDisallow: /cbbc/search?\nDisallow: /cbeebies/search/\nDisallow: /cbeebies/search$\nDisallow: /cbeebies/search?\nDisallow: /chwilio/\nDisallow: /chwilio$\nDisallow: /chwilio?\nDisallow: /education/blocks$\nDisallow: /education/blocks/\nDisallow: /newsround\nDisallow: /search/\nDisallow: /search$\nDisallow: /search?\nDisallow: /food/favourites\nDisallow: /food/search*?*\nDisallow: /food/recipes/search*?*\nDisallow: /education/my$\nDisallow: /education/my/\nDisallow: /bitesize/my$\nDisallow: /bitesize/my/\nDisallow: /food/recipes/*/shopping-list\nDisallow: /food/menus/*/shopping-list\nDisallow: /news/0\nDisallow: /sport/alpha/\nDisallow: /ugc$\nDisallow: /ugc/\nDisallow: /ugcsupport$\nDisallow: /ugcsupport/\nDisallow: /userinfo/\nDisallow: /userinfo\nDisallow: /u5llnop$\nDisallow: /u5llnop/\nDisallow: /sounds/search$\nDisallow: /sounds/search/\nDisallow: /sounds/search?\nDisallow: /ws/includes\nDisallow: /radio/imda\nDisallow: /storyworks/preview/*\nDisallow: /rd/search$\nDisallow: /rd/search/\nDisallow: /rd/search?\n\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-Agent: omgili\nDisallow: /\n \nUser-Agent: omgilibot\nDisallow: /\n \nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: anthropic-ai\nDisallow: /\n \nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\nAllow: /storyworks\nAllow: /storyworks/\n\nUser-agent: ChatGPT-User\nDisallow: /\nAllow: /storyworks\nAllow: /storyworks/\n\nUser-agent: Google-Extended\nDisallow: /\nAllow: /storyworks\nAllow: /storyworks/\n\nUser-agent: OAI-SearchBot\nDisallow: /\nAllow: /storyworks\nAllow: /storyworks/\n\nUser-Agent: PerplexityBot\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: Google-CloudVertexBot\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: YandexAdditional\nDisallow: /\n\nUser-agent: YandexAdditionalBot\nDisallow: /\n\nUser-agent: TurnitinBot\nDisallow: /\n\nUser-agent: Brightbot\nDisallow: /\n\nUser-agent: ApifyBot\nDisallow: /\n\nUser-agent: ApifyWebsiteContentCrawler\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: Diffbot-User\nDisallow: /\n\nUser-agent: ExaBot\nDisallow: /\n\nUser-agent: TavilyBot\nDisallow: /\n\nUser-agent: ShapBot\nDisallow: /\n\nUser-agent: YouBot\nDisallow: /\n\nUser-agent: FirecrawlAgent\nDisallow: /\n\nUser-agent: Amzn-SearchBot\nDisallow: /\n\nUser-agent: Amzn-User\nDisallow: /\n\nUser-agent: ProRataInc\nDisallow: /\n\nUser-agent: CloudflareBrowserRenderingCrawler\nDisallow: /\n\nUser-agent: AhrefsBot\nDisallow: /\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":9,"text":"- No scraping, crawling, or systematic extraction of content"},{"line":10,"text":"- No use of BBC content for training or fine-tuning AI models, including large language models (LLMs)"},{"line":11,"text":"- No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Amazonbot","magpie-crawler","CCBot","omgili","omgilibot","Claude-Web","ClaudeBot","anthropic-ai","cohere-ai","Bytespider","PetalBot","Scrapy","Applebot-Extended","GPTBot","ChatGPT-User","Google-Extended","OAI-SearchBot","PerplexityBot","Perplexity-User","Google-CloudVertexBot","meta-externalagent","YandexAdditional","YandexAdditionalBot","TurnitinBot","Brightbot","ApifyBot","ApifyWebsiteContentCrawler","Diffbot","Diffbot-User","ExaBot","TavilyBot","ShapBot","YouBot","FirecrawlAgent","Amzn-SearchBot","Amzn-User","ProRataInc","CloudflareBrowserRenderingCrawler","AhrefsBot"],"lines":238},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":154},"any":{"allowed":false,"line":154},"search":{"allowed":false,"line":154},"cart":{"allowed":false,"line":154},"checkout":{"allowed":false,"line":154},"account":{"allowed":false,"line":154}},"disallows":1,"lines":[153,156],"groupRanges":[[153,156]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":133},"any":{"allowed":false,"line":133},"search":{"allowed":false,"line":133},"cart":{"allowed":false,"line":133},"checkout":{"allowed":false,"line":133},"account":{"allowed":false,"line":133}},"disallows":1,"lines":[132,133],"groupRanges":[[132,133]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":164},"any":{"allowed":false,"line":164},"search":{"allowed":false,"line":164},"cart":{"allowed":false,"line":164},"checkout":{"allowed":false,"line":164},"account":{"allowed":false,"line":164}},"disallows":1,"lines":[163,166],"groupRanges":[[163,166]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":151},"any":{"allowed":false,"line":151},"search":{"allowed":false,"line":151},"cart":{"allowed":false,"line":151},"checkout":{"allowed":false,"line":151},"account":{"allowed":false,"line":151}},"disallows":1,"lines":[150,151],"groupRanges":[[150,151]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":121},"any":{"allowed":false,"line":121},"search":{"allowed":false,"line":121},"cart":{"allowed":false,"line":121},"checkout":{"allowed":false,"line":121},"account":{"allowed":false,"line":121}},"disallows":1,"lines":[120,121],"groupRanges":[[120,121]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":183},"any":{"allowed":false,"line":183},"search":{"allowed":false,"line":183},"cart":{"allowed":false,"line":183},"checkout":{"allowed":false,"line":183},"account":{"allowed":false,"line":183}},"disallows":1,"lines":[182,183],"groupRanges":[[182,183]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":142},"any":{"allowed":false,"line":142},"search":{"allowed":false,"line":142},"cart":{"allowed":false,"line":142},"checkout":{"allowed":false,"line":142},"account":{"allowed":false,"line":142}},"disallows":1,"lines":[141,142],"groupRanges":[[141,142]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":136},"any":{"allowed":false,"line":136},"search":{"allowed":false,"line":136},"cart":{"allowed":false,"line":136},"checkout":{"allowed":false,"line":136},"account":{"allowed":false,"line":136}},"disallows":1,"lines":[135,136],"groupRanges":[[135,136]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":169},"any":{"allowed":false,"line":169},"search":{"allowed":false,"line":169},"cart":{"allowed":false,"line":169},"checkout":{"allowed":false,"line":169},"account":{"allowed":false,"line":169}},"disallows":1,"lines":[168,171],"groupRanges":[[168,171]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":174},"any":{"allowed":false,"line":174},"search":{"allowed":false,"line":174},"cart":{"allowed":false,"line":174},"checkout":{"allowed":false,"line":174},"account":{"allowed":false,"line":174}},"disallows":1,"lines":[173,174],"groupRanges":[[173,174]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":115},"any":{"allowed":false,"line":115},"search":{"allowed":false,"line":115},"cart":{"allowed":false,"line":115},"checkout":{"allowed":false,"line":115},"account":{"allowed":false,"line":115}},"disallows":1,"lines":[114,115],"groupRanges":[[114,115]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":225},"any":{"allowed":false,"line":225},"search":{"allowed":false,"line":225},"cart":{"allowed":false,"line":225},"checkout":{"allowed":false,"line":225},"account":{"allowed":false,"line":225}},"disallows":1,"lines":[224,225],"groupRanges":[[224,225]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":159},"any":{"allowed":false,"line":159},"search":{"allowed":false,"line":159},"cart":{"allowed":false,"line":159},"checkout":{"allowed":false,"line":159},"account":{"allowed":false,"line":159}},"disallows":1,"lines":[158,161],"groupRanges":[[158,161]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":177},"any":{"allowed":false,"line":177},"search":{"allowed":false,"line":177},"cart":{"allowed":false,"line":177},"checkout":{"allowed":false,"line":177},"account":{"allowed":false,"line":177}},"disallows":1,"lines":[176,177],"groupRanges":[[176,177]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"Amzn-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":228},"any":{"allowed":false,"line":228},"search":{"allowed":false,"line":228},"cart":{"allowed":false,"line":228},"checkout":{"allowed":false,"line":228},"account":{"allowed":false,"line":228}},"disallows":1,"lines":[227,228],"groupRanges":[[227,228]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":83},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":49,"lines":[23,112],"groupRanges":[[23,112]]}],"lanes":{"training":{"level":"mixed","none":7,"full":2,"restricted":0,"n":9},"search":{"level":"mixed","none":4,"full":4,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":3,"full":4,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":["Fastly"],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.bbc.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.bbc.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.bbc.com/.well-known/tdmrep.json"},"meta":{"title":"BBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video & Audio","robotsMeta":[{"name":"robots","content":"NOODP, NOYDIR"}],"noai":[],"noindex":false,"jsonLdTypes":["WebPage","NewsMediaOrganization"],"platform":"Next.js","textRatio":0.027,"llmsLink":false},"x_robots_tag":"bingbot: noarchive","tos":{"url":"https://www.bbc.com/pages/terms-of-use","source":"discovered","http":200,"hash":"03a21dc287fa4a5b2a7375c28aff04f2541988b018afa5b91b7c12ce3b466af9","chars":33595,"thin":false,"clauses":[{"keyword":"artificial intelligence","prohibition":true,"excerpt":"… us, in connection with the development, training, fine tuning, or grounding of any software program, model, algorithm, artificial intelligence system, or machine learning or generative artificial intelligence tool."},{"keyword":"agents","prohibition":true,"excerpt":"…o protect ourselves, the Affiliated Entities, and our or their respective employees, officers, directors, shareholders, agents, and representatives, and the Services’ users and visitors, to comply with legal obligations or governmental requests, to enforce these Terms, or for any other reason or pur…"},{"keyword":"agents","prohibition":true,"excerpt":"These Terms do not apply to waive your right to make statements regarding BBC.com or its employees or agents, or concerning any of BBC.com’s Content and Services."},{"keyword":"agent","prohibition":true,"excerpt":"… good faith belief that use of the material in the manner complained of is not authorised by the owner of the work, its agent, or the law;"},{"keyword":"robots","prohibition":false,"excerpt":"Hacking our Services or use robots, spiders, scripts, or any tool or process designed to data mine or scrape the Content or collect the Content, data or information from the Services using automated means"},{"keyword":"artificial intelligence","prohibition":false,"excerpt":"Anything taken from our Services to develop or train artificial intelligence or to do computer analysis"},{"keyword":"agent","prohibition":false,"excerpt":"BBC’s designated copyright agent to receive notifications of claimed infringement is:"}],"door":{"door":"open","http":200}},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":3},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":7},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":4},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":8},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":5},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":14}]},"events":[{"domain":"bbc.com","name":"BBC","category":"news","date":"2026-01","precision":"month","source":"archive","provisional":false,"between":["2025-12-01","2026-01-01"],"summary":"Now blocked: Amzn-SearchBot and Amzn-User.","dir":"up","up":["Amzn-SearchBot","Amzn-User"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20251201150544/https://www.bbc.com/robots.txt","https://web.archive.org/web/20260101012945/https://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f"},{"t":"add","s":"# version: 8c1c7053723d6bd586586053ecdd261591364cfe"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml"},{"t":"add","s":"Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml"},{"t":"add","s":"Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Amzn-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amzn-User"},{"t":"add","s":"Disallow: /"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-04-01","2025-08-01"],"summary":"Now blocked: Amazonbot and Perplexity-User.","dir":"up","up":["Amazonbot","Perplexity-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250401011704/https://www.bbc.com/robots.txt","https://web.archive.org/web/20250801014616/https://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: ca16af0d01ecf92c8730f124f789053bfbf7293b"},{"t":"del","s":""},{"t":"del","s":"# HTTP www.bbc.com"},{"t":"del","s":""},{"t":"del","s":"User-agent: Googlebot"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemap.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"del","s":""},{"t":"add","s":"# version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f"},{"t":"ctx","s":""},{"t":"del","s":"Disallow: /iplayer/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/cy/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/gd/episode/*?from=r*"},{"t":"del","s":"Disallow: /_programmes"},{"t":"del","s":"Disallow: /606/"},{"t":"del","s":"Disallow: /aboutthebbc/insidethebbc/search"},{"t":"del","s":"Disallow: /academy/chinese-trad/search"},{"t":"del","s":"Disallow: /afrique/search"},{"t":"del","s":"Disallow: /apps/cbbc"},{"t":"del","s":"Disallow: /apps/flash"},{"t":"del","s":"Disallow: /apps/ide"},{"t":"del","s":"Disallow: /apps/ids"},{"t":"del","s":"Disallow: /apps/ifl"},{"t":"del","s":"Disallow: /apps/nr"},{"t":"del","s":"Disallow: /apps/query"},{"t":"del","s":"Disallow: /apps/t"},{"t":"del","s":"Disallow: /arabic/search"},{"t":"del","s":"Disallow: /arts/yourpaintings/artdetective/"},{"t":"del","s":"Disallow: /arts/yourpaintings/feedback/sent/"},{"t":"del","s":"Disallow: /arts/yourpaintings/images/guidedtours/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/error/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/success/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/unsubscribe/"},{"t":"del","s":"Disallow: /arts/yourpaintings/paintings/search*"},{"t":"del","s":"Disallow: /arts/yourpaintings/search/"},{"t":"del","s":"Disallow: /azeri/search"},{"t":"del","s":"Disallow: /bbctrust/search"},{"t":"del","s":"Disallow: /bengali/search"},{"t":"del","s":"Disallow: /burmese/search"},{"t":"del","s":"Disallow: /cbbc/search"},{"t":"del","s":"Disallow: /cbeebies/search"},{"t":"del","s":"Disallow: /cgi-bin"},{"t":"del","s":"Disallow: /cgi-perl"},{"t":"del","s":"Disallow: /cgi-perlx"},{"t":"del","s":"Disallow: /cgi-store"},{"t":"del","s":"Disallow: /corporate2/bbctrust/search"},{"t":"del","s":"Disallow: /corporate2/insidethebbc/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/proginfo/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/search"},{"t":"del","s":"Disallow: /corporate2/safety/search"},{"t":"del","s":"Disallow: /eastenders/updates/"},{"t":"del","s":"Disallow: /education$"},{"t":"del","s":"Disallow: /education/"},{"t":"del","s":"Disallow: /education/bitesize"},{"t":"del","s":"Disallow: /education/dev"},{"t":"del","s":"Disallow: /education/images"},{"t":"del","s":"Disallow: /education/nav"},{"t":"gap","s":"… 745 more lines"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2025-03","precision":"month","source":"archive","provisional":false,"between":["2025-02-01","2025-03-02"],"summary":"Now blocked: Meta-ExternalAgent and OAI-SearchBot.","dir":"up","up":["Meta-ExternalAgent","OAI-SearchBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250201035627/https://www.bbc.com/robots.txt","https://web.archive.org/web/20250302094514/http://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68"},{"t":"add","s":"# version: ca16af0d01ecf92c8730f124f789053bfbf7293b"},{"t":"ctx","s":""},{"t":"del","s":"# HTTPS www.bbc.com"},{"t":"add","s":"# HTTP www.bbc.com"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: *"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sport/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/afrique/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/arabic/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/bengali/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/burmese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/gahuza/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/hausa/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/hindi/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/indonesia/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/mundo/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/pashto/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/persian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/portuguese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/russian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/swahili/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/tajik/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/turkce/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/ukrainian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/urdu/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/uzbek/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/vietnamese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"add","s":"User-agent: Googlebot"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemap.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"ctx","s":""},{"t":"del","s":"Disallow: /bitesize/search$"},{"t":"del","s":"Disallow: /bitesize/search/"},{"t":"del","s":"Disallow: /bitesize/search?"},{"t":"del","s":"Disallow: /cbbc/search/"},{"t":"del","s":"Disallow: /cbbc/search$"},{"t":"del","s":"Disallow: /cbbc/search?"},{"t":"del","s":"Disallow: /cbeebies/search/"},{"t":"del","s":"Disallow: /cbeebies/search$"},{"t":"del","s":"Disallow: /cbeebies/search?"},{"t":"del","s":"Disallow: /chwilio/"},{"t":"del","s":"Disallow: /chwilio$"},{"t":"del","s":"Disallow: /chwilio?"},{"t":"add","s":""},{"t":"add","s":"Disallow: /iplayer/episode/*?from=r*"},{"t":"add","s":"Disallow: /iplayer/cy/episode/*?from=r*"},{"t":"gap","s":"… 785 more lines"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2024-10","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-10-01"],"summary":"Now blocked: ClaudeBot and Applebot-Extended.","dir":"up","up":["ClaudeBot","Applebot-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801003016/https://www.bbc.com/robots.txt","https://web.archive.org/web/20241001000744/https://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: 03e3d0d3861e30b21826aa11558f45235a4d4143"},{"t":"del","s":""},{"t":"del","s":"# HTTP www.bbc.com"},{"t":"del","s":""},{"t":"del","s":"User-agent: Googlebot"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemap.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"del","s":""},{"t":"add","s":"# version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68"},{"t":"ctx","s":""},{"t":"del","s":"Disallow: /iplayer/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/cy/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/gd/episode/*?from=r*"},{"t":"del","s":"Disallow: /_programmes"},{"t":"del","s":"Disallow: /606/"},{"t":"del","s":"Disallow: /aboutthebbc/insidethebbc/search"},{"t":"del","s":"Disallow: /academy/chinese-trad/search"},{"t":"del","s":"Disallow: /afrique/search"},{"t":"del","s":"Disallow: /apps/cbbc"},{"t":"del","s":"Disallow: /apps/flash"},{"t":"del","s":"Disallow: /apps/ide"},{"t":"del","s":"Disallow: /apps/ids"},{"t":"del","s":"Disallow: /apps/ifl"},{"t":"del","s":"Disallow: /apps/nr"},{"t":"del","s":"Disallow: /apps/query"},{"t":"del","s":"Disallow: /apps/t"},{"t":"del","s":"Disallow: /arabic/search"},{"t":"del","s":"Disallow: /arts/yourpaintings/artdetective/"},{"t":"del","s":"Disallow: /arts/yourpaintings/feedback/sent/"},{"t":"del","s":"Disallow: /arts/yourpaintings/images/guidedtours/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/error/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/success/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/unsubscribe/"},{"t":"del","s":"Disallow: /arts/yourpaintings/paintings/search*"},{"t":"del","s":"Disallow: /arts/yourpaintings/search/"},{"t":"del","s":"Disallow: /azeri/search"},{"t":"del","s":"Disallow: /bbctrust/search"},{"t":"del","s":"Disallow: /bengali/search"},{"t":"del","s":"Disallow: /burmese/search"},{"t":"del","s":"Disallow: /cbbc/search"},{"t":"del","s":"Disallow: /cbeebies/search"},{"t":"del","s":"Disallow: /cgi-bin"},{"t":"del","s":"Disallow: /cgi-perl"},{"t":"del","s":"Disallow: /cgi-perlx"},{"t":"del","s":"Disallow: /cgi-store"},{"t":"del","s":"Disallow: /corporate2/bbctrust/search"},{"t":"del","s":"Disallow: /corporate2/insidethebbc/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/proginfo/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/search"},{"t":"del","s":"Disallow: /corporate2/safety/search"},{"t":"del","s":"Disallow: /eastenders/updates/"},{"t":"del","s":"Disallow: /education$"},{"t":"del","s":"Disallow: /education/"},{"t":"del","s":"Disallow: /education/bitesize"},{"t":"del","s":"Disallow: /education/dev"},{"t":"del","s":"Disallow: /education/images"},{"t":"del","s":"Disallow: /education/nav"},{"t":"gap","s":"… 733 more lines"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2024-05","precision":"month","source":"archive","provisional":false,"between":["2024-04-01","2024-05-01"],"summary":"Now blocked: Bytespider and PerplexityBot.","dir":"up","up":["Bytespider","PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240401000803/https://www.bbc.com/robots.txt","https://web.archive.org/web/20240501002309/http://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053"},{"t":"add","s":"# version: 03e3d0d3861e30b21826aa11558f45235a4d4143"},{"t":"ctx","s":""},{"t":"del","s":"# HTTPS www.bbc.com"},{"t":"add","s":"# HTTP www.bbc.com"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: *"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sport/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/afrique/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/arabic/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/bengali/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/burmese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/gahuza/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/hausa/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/hindi/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/indonesia/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/mundo/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/pashto/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/persian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/portuguese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/russian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/swahili/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/tajik/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/turkce/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/ukrainian/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/urdu/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/uzbek/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/vietnamese/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml"},{"t":"del","s":"Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"add","s":"User-agent: Googlebot"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemap.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml"},{"t":"add","s":"Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"ctx","s":""},{"t":"del","s":"Disallow: /bitesize/search$"},{"t":"del","s":"Disallow: /bitesize/search/"},{"t":"del","s":"Disallow: /bitesize/search?"},{"t":"del","s":"Disallow: /cbbc/search/"},{"t":"del","s":"Disallow: /cbbc/search$"},{"t":"del","s":"Disallow: /cbbc/search?"},{"t":"del","s":"Disallow: /cbeebies/search/"},{"t":"del","s":"Disallow: /cbeebies/search$"},{"t":"del","s":"Disallow: /cbeebies/search?"},{"t":"del","s":"Disallow: /chwilio/"},{"t":"del","s":"Disallow: /chwilio$"},{"t":"del","s":"Disallow: /chwilio?"},{"t":"add","s":""},{"t":"add","s":"Disallow: /iplayer/episode/*?from=r*"},{"t":"add","s":"Disallow: /iplayer/cy/episode/*?from=r*"},{"t":"gap","s":"… 730 more lines"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2024-02","precision":"month","source":"archive","provisional":false,"between":["2023-12-01","2024-02-01"],"summary":"Now blocked: Google-Extended and anthropic-ai.","dir":"up","up":["Google-Extended","anthropic-ai"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231201000213/http://www.bbc.com/robots.txt","https://web.archive.org/web/20240201000336/https://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: f6f959638af29cbc598b7adab27fd84e19d0d062"},{"t":"add","s":"# version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml"},{"t":"add","s":"Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /news/0"},{"t":"add","s":"Disallow: /sport/alpha/"},{"t":"ctx","s":"Disallow: /ugc$"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /radio/imda"},{"t":"add","s":"Disallow: /storyworks/preview/*"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-Agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":" "},{"t":"add","s":"User-Agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":" "},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":" "},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":" "},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"}]},{"domain":"bbc.com","name":"BBC","category":"news","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: GPTBot, CCBot and ChatGPT-User.","dir":"up","up":["GPTBot","CCBot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20231001002857/https://www.bbc.com/robots.txt","https://web.archive.org/web/20231101000948/https://www.bbc.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# version: 587aced11b074061603bab7e2b21f94ae7f96821"},{"t":"del","s":""},{"t":"del","s":"# HTTP www.bbc.com"},{"t":"del","s":""},{"t":"del","s":"User-agent: Googlebot"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemap.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml"},{"t":"del","s":"Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml"},{"t":"del","s":""},{"t":"del","s":""},{"t":"del","s":"Disallow: /iplayer/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/cy/episode/*?from=r*"},{"t":"del","s":"Disallow: /iplayer/gd/episode/*?from=r*"},{"t":"del","s":"Disallow: /_programmes"},{"t":"del","s":"Disallow: /606/"},{"t":"del","s":"Disallow: /aboutthebbc/insidethebbc/search"},{"t":"del","s":"Disallow: /academy/chinese-trad/search"},{"t":"del","s":"Disallow: /afrique/search"},{"t":"del","s":"Disallow: /apps/cbbc"},{"t":"del","s":"Disallow: /apps/flash"},{"t":"del","s":"Disallow: /apps/ide"},{"t":"del","s":"Disallow: /apps/ids"},{"t":"del","s":"Disallow: /apps/ifl"},{"t":"del","s":"Disallow: /apps/nr"},{"t":"del","s":"Disallow: /apps/query"},{"t":"del","s":"Disallow: /apps/t"},{"t":"del","s":"Disallow: /arabic/search"},{"t":"del","s":"Disallow: /arts/yourpaintings/artdetective/"},{"t":"del","s":"Disallow: /arts/yourpaintings/feedback/sent/"},{"t":"del","s":"Disallow: /arts/yourpaintings/images/guidedtours/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/error/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/success/"},{"t":"del","s":"Disallow: /arts/yourpaintings/newsletter/unsubscribe/"},{"t":"del","s":"Disallow: /arts/yourpaintings/paintings/search*"},{"t":"del","s":"Disallow: /arts/yourpaintings/search/"},{"t":"del","s":"Disallow: /azeri/search"},{"t":"del","s":"Disallow: /bbctrust/search"},{"t":"del","s":"Disallow: /bengali/search"},{"t":"del","s":"Disallow: /burmese/search"},{"t":"del","s":"Disallow: /cbbc/search"},{"t":"del","s":"Disallow: /cbeebies/search"},{"t":"del","s":"Disallow: /cgi-bin"},{"t":"del","s":"Disallow: /cgi-perl"},{"t":"del","s":"Disallow: /cgi-perlx"},{"t":"del","s":"Disallow: /cgi-store"},{"t":"del","s":"Disallow: /corporate2/bbctrust/search"},{"t":"del","s":"Disallow: /corporate2/insidethebbc/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/proginfo/search"},{"t":"del","s":"Disallow: /corporate2/mediacentre/search"},{"t":"del","s":"Disallow: /corporate2/safety/search"},{"t":"del","s":"Disallow: /eastenders/updates/"},{"t":"del","s":"Disallow: /education$"},{"t":"del","s":"Disallow: /education/"},{"t":"del","s":"Disallow: /education/bitesize"},{"t":"del","s":"Disallow: /education/dev"},{"t":"del","s":"Disallow: /education/images"},{"t":"del","s":"Disallow: /education/nav"},{"t":"del","s":"Disallow: /education/navigation"},{"t":"del","s":"Disallow: /education/ximages"},{"t":"gap","s":"… 706 more lines"}]}]}