{"domain":"theguardian.com","name":"The Guardian","category":"news","subcategory":"national","country":"GB","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent and Claude-SearchBot. GPTBot, Google-Extended, OAI-SearchBot, ChatGPT-User and 1 more can reach most of it.","computed_at":"2026-10-20T03:43:15.145Z","last_checked":"2026-10-19T13:34:20.753Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.theguardian.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.theguardian.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.theguardian.com/robots.txt","hops":[{"from":"https://theguardian.com/robots.txt","to":"https://www.theguardian.com/robots.txt","status":301}],"blob":"3387cd8d393a0c257e687ce63fcf30558ada1eb1fc167bfc51cb216e29005db2","text":"# This is the robots.txt file for theguardian.com\n\n# Guardian content is made available under our terms and conditions of use.\n# Any other uses are not permitted, incl. but not limited to: for large language\n# models (LLMs), machine learning and/or artificial intelligence-related\n# purposes; with any of the aforementioned technologies; and/or for any\n# commercial purposes. Contact licensing@theguardian.com for assistance\n\nUser-agent: *\nDisallow: /sendarticle/\nDisallow: /Users/\nDisallow: /users/\nDisallow: /*/print$\nDisallow: /email/\nDisallow: /contactus/\nDisallow: /share/\nDisallow: /websearch\nDisallow: /*?commentpage=\nDisallow: /whsmiths/\nDisallow: /external/overture/\nDisallow: /discussion/report-abuse/*\nDisallow: /discussion/report-abuse-ajax/*\nDisallow: /discussion/comment-permalink/*\nDisallow: /discussion/report-abuse/*\nDisallow: /discussion/user-report-abuse/*\nDisallow: /discussion/handlers/*\nDisallow: /discussion/your-profile\nDisallow: /discussion/your-comments\nDisallow: /discussion/edit-profile\nDisallow: /discussion/search/comments\nDisallow: /discussion/*\nDisallow: /search\nDisallow: /music/artist/*\nDisallow: /music/album/*\nDisallow: /books/data/*\nDisallow: /settings/\nDisallow: /embed/\nDisallow: /*styles/js-on.css$\nDisallow: /sport/olympics/2008/events/*\nDisallow: /sport/olympics/2008/medals/*\nDisallow: /f/healthcheck\nDisallow: /sections\nDisallow: /top-stories\nDisallow: /most-read/sport\nDisallow: /articles\nDisallow: /global$\nDisallow: /*/feedarticle/*\nDisallow: /travel/2013/aug/22/been-there-readers-competition?*\nDisallow: /preference/*\nDisallow: /59666047/\nDisallow: /print/\nDisallow: /info/tech-feedback\nDisallow: /production-monitoring/\nDisallow: *.emailjson\nDisallow: *.emailtxt\nDisallow: /headline.txt\nDisallow: *?*dcr=apps*\n\nUser-agent: Mediapartners-Google\nDisallow:\n\nSitemap: http://www.theguardian.com/sitemaps/news.xml\nSitemap: http://www.theguardian.com/sitemaps/video.xml\n\nUser-agent: NewsNow\nUser-agent: CCBot\nUser-agent: TurnitinBot\nUser-agent: PetalBot\nUser-agent: MoodleBot\nUser-agent: FacebookBot\nUser-agent: Bytespider\nUser-agent: Mojeek\nUser-agent: JenkersBot\nUser-agent: Seekr\nUser-agent: YouBot\nUser-agent: Arquivo-web-crawler\nUser-agent: coccocbot-web\nUser-agent: SeznamBot\nUser-Agent: PerplexityBot\nUser-Agent: yacy\nUser-agent: anthropic-ai\nUser-agent: ClaudeBot\nUser-agent: Claude-SearchBot\nUser-agent: Claude-User\nUser-agent: AwarioRssBot\nUser-agent: AwarioSmartBot\nUser-agent: SentiOne\nUser-agent: ImageSift\nUser-agent: Applebot-Extended\nUser-agent: YandexAdditional\nUser-agent: YandexAdditionalBot\nUser-agent: scalepostAI\nUser-agent: Buck\nUser-agent: meta-externalagent\nUser-agent: Amazonbot\nUser-agent: amazon-QBusiness\nUser-agent: DuckAssistBot\nUser-agent: Google-CloudVertexBot\nUser-agent: Amzn-SearchBot\nUser-agent: AhrefsBot\nUser-agent: AhrefsSiteAudit\nDisallow: /\n\nLicense: https://theguardian.com/license.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[{"url":"https://theguardian.com/license.xml","line":104,"agents":["newsnow","ccbot","turnitinbot","petalbot","moodlebot","facebookbot","bytespider","mojeek","jenkersbot","seekr","youbot","arquivo-web-crawler","coccocbot-web","seznambot","perplexitybot","yacy","anthropic-ai","claudebot","claude-searchbot","claude-user","awariorssbot","awariosmartbot","sentione","imagesift","applebot-extended","yandexadditional","yandexadditionalbot","scalepostai","buck","meta-externalagent","amazonbot","amazon-qbusiness","duckassistbot","google-cloudvertexbot","amzn-searchbot","ahrefsbot","ahrefssiteaudit"]}],"aiComments":[{"line":5,"text":"models (LLMs), machine learning and/or artificial intelligence-related"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Mediapartners-Google","NewsNow","CCBot","TurnitinBot","PetalBot","MoodleBot","FacebookBot","Bytespider","Mojeek","JenkersBot","Seekr","YouBot","Arquivo-web-crawler","coccocbot-web","SeznamBot","PerplexityBot","yacy","anthropic-ai","ClaudeBot","Claude-SearchBot","Claude-User","AwarioRssBot","AwarioSmartBot","SentiOne","ImageSift","Applebot-Extended","YandexAdditional","YandexAdditionalBot","scalepostAI","Buck","meta-externalagent","Amazonbot","amazon-QBusiness","DuckAssistBot","Google-CloudVertexBot","Amzn-SearchBot","AhrefsBot","AhrefsSiteAudit"],"lines":105},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":102},"any":{"allowed":false,"line":102},"search":{"allowed":false,"line":102},"cart":{"allowed":false,"line":102},"checkout":{"allowed":false,"line":102},"account":{"allowed":false,"line":102}},"disallows":1,"lines":[65,104],"groupRanges":[[65,104]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":32},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":48,"lines":[9,57],"groupRanges":[[9,57]]}],"lanes":{"training":{"level":"mixed","none":5,"full":4,"restricted":0,"n":9},"search":{"level":"mixed","none":5,"full":3,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":1,"full":6,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.theguardian.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.theguardian.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.theguardian.com/.well-known/tdmrep.json"},"meta":{"title":"Latest news, sport and opinion from the Guardian","robotsMeta":[{"name":"robots","content":"max-image-preview:large"}],"noai":[],"noindex":false,"jsonLdTypes":[],"platform":null,"textRatio":0.015,"llmsLink":false},"x_robots_tag":"bingbot: noarchive","tos":{"url":"https://www.theguardian.com/help/terms-of-service","source":"discovered","http":200,"hash":"5350af78cf041eb80e51e3722b9039da46b5137482e2550abe90a4caeb75836e","chars":35403,"thin":false,"clauses":[{"keyword":"scrape","prohibition":true,"excerpt":"…any other purpose without our prior written approval - this includes, without limitation, that you shall not use, copy, scrape, reproduce, alter, modify, collect, mine and/or extract the Guardian Content:"},{"keyword":"robot","prohibition":true,"excerpt":"…se approved in writing by the Guardian, you shall also not use, or facilitate, authorise or permit the use of, any (i) “robot”, “bot”, “spider”, “scraper”, “crawler” or (ii) other automated device, program, technique, tool, process, algorithm or method (whether for data gathering, mining, collection…"},{"keyword":"robots","prohibition":true,"excerpt":"…ing bypassing or circumventing any access controls or use limits of the Guardian Content (including, by way of example, robots.txt internet protocols or similar or successor protocols)."},{"keyword":"robots","prohibition":true,"excerpt":"Our robots.txt notice does not, and shall not, constitute the Guardian’s permission or approval for the purposes of these terms and conditions."},{"keyword":"text and data mining","prohibition":true,"excerpt":"…s’ attention to specific sections, to make clear that reproduction and/or extraction of Guardian content for commercial text and data mining is not permitted and that the Guardian is the sole seller of advertising on the Guardian Site."},{"keyword":"scraping","prohibition":true,"excerpt":"Paragraph 3 of Section 3 amended to begin with “ Without limitation” and to replace the wording “ any scraping of the Guardian Content or reproduction, copying…or mining purposes or for any commercial use ” with “ that you shall not use, copy, scrape, reproduce, alter, modify, collect, mine and/or e…"},{"keyword":"robots","prohibition":true,"excerpt":"New paragraph 8 to Section 3 beginning “ Our robots.txt notice does not, and shall not, constitute…”"},{"keyword":"agent","prohibition":true,"excerpt":"… good faith belief that use of the material in the manner complained of is not authorised by the owner of the work, its agent, or the law;"}],"door":{"door":"open","http":200}},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":1,"first":"2026-10-19","archive_points":0,"series":[{"m":"2023-01","s":null},{"m":"2023-02","s":null},{"m":"2023-03","s":null},{"m":"2023-04","s":null},{"m":"2023-05","s":null},{"m":"2023-06","s":null},{"m":"2023-07","s":null},{"m":"2023-08","s":null},{"m":"2023-09","s":null},{"m":"2023-10","s":null},{"m":"2023-11","s":null},{"m":"2023-12","s":null},{"m":"2024-01","s":null},{"m":"2024-02","s":null},{"m":"2024-03","s":null},{"m":"2024-04","s":null},{"m":"2024-05","s":null},{"m":"2024-06","s":null},{"m":"2024-07","s":null},{"m":"2024-08","s":null},{"m":"2024-09","s":null},{"m":"2024-10","s":null},{"m":"2024-11","s":null},{"m":"2024-12","s":null},{"m":"2025-01","s":null},{"m":"2025-02","s":null},{"m":"2025-03","s":null},{"m":"2025-04","s":null},{"m":"2025-05","s":null},{"m":"2025-06","s":null},{"m":"2025-07","s":null},{"m":"2025-08","s":null},{"m":"2025-09","s":null},{"m":"2025-10","s":null},{"m":"2025-11","s":null},{"m":"2025-12","s":null},{"m":"2026-01","s":null},{"m":"2026-02","s":null},{"m":"2026-03","s":null},{"m":"2026-04","s":null},{"m":"2026-05","s":null},{"m":"2026-06","s":null},{"m":"2026-07","s":null},{"m":"2026-08","s":null},{"m":"2026-09","s":null},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11}]},"events":[]}