The Guardian

theguardian.com

Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 11 of the 24 AI bots we track from the whole site, including ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent and Claude-SearchBot. GPTBot, Google-Extended, OAI-SearchBot, ChatGPT-User and 1 more can reach most of it.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

5 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

5 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Some blocked

1 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
ClaudeBotAnthropic	Blocked	Named, line 65	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Blocked	Named, line 65	Says yes
CCBotCommon Crawl	Blocked	Named, line 65	Says yes
Meta-ExternalAgentMeta	Blocked	Named, line 65	Suggests yes
BytespiderByteDance	Blocked	Named, line 65	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Allowed	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Blocked	Named, line 65	Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Blocked	Named, line 65	Says yes
PerplexityBotPerplexity	Blocked	Named, line 65	Suggests yes
AmazonbotAmazon	Blocked	Named, line 65	Says yes
Amzn-SearchBotAmazon	Blocked	Named, line 65	Says yes
DuckAssistBotDuckDuckGo	Blocked	Named, line 65	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Some blocked
ChatGPT-UserOpenAI	Allowed	Not named; the catch-all rule applies	Says not always
Claude-UserAnthropic	Blocked	Named, line 65	Says yes
Perplexity-UserPerplexity	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Allowed	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

PetalBot	Blocked	line 68	Not in our verified list
FacebookBot	Blocked	line 70	Not in our verified list
YouBot	Blocked	line 75	Not in our verified list
ImageSift	Blocked	line 88	Not in our verified list
Google-CloudVertexBot	Blocked	line 98	Documented by its operator

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 105 lines · 40 lines about AI bots Source

# This is the robots.txt file for theguardian.com # Guardian content is made available under our terms and conditions of use.# Any other uses are not permitted, incl. but not limited to: for large language# models (LLMs), machine learning and/or artificial intelligence-related# purposes; with any of the aforementioned technologies; and/or for any# commercial purposes. Contact licensing@theguardian.com for assistance User-agent: *Disallow: /sendarticle/Disallow: /Users/Disallow: /users/Disallow: /*/print$Disallow: /email/Disallow: /contactus/Disallow: /share/Disallow: /websearchDisallow: /*?commentpage=Disallow: /whsmiths/Disallow: /external/overture/Disallow: /discussion/report-abuse/*Disallow: /discussion/report-abuse-ajax/*Disallow: /discussion/comment-permalink/*Disallow: /discussion/report-abuse/*Disallow: /discussion/user-report-abuse/*Disallow: /discussion/handlers/*Disallow: /discussion/your-profileDisallow: /discussion/your-commentsDisallow: /discussion/edit-profileDisallow: /discussion/search/commentsDisallow: /discussion/*Disallow: /searchDisallow: /music/artist/*Disallow: /music/album/*Disallow: /books/data/*Disallow: /settings/Disallow: /embed/Disallow: /*styles/js-on.css$Disallow: /sport/olympics/2008/events/*Disallow: /sport/olympics/2008/medals/*Disallow: /f/healthcheckDisallow: /sectionsDisallow: /top-storiesDisallow: /most-read/sportDisallow: /articlesDisallow: /global$Disallow: /*/feedarticle/*Disallow: /travel/2013/aug/22/been-there-readers-competition?*Disallow: /preference/*Disallow: /59666047/Disallow: /print/Disallow: /info/tech-feedbackDisallow: /production-monitoring/Disallow: *.emailjsonDisallow: *.emailtxtDisallow: /headline.txtDisallow: *?*dcr=apps* User-agent: Mediapartners-GoogleDisallow: Sitemap: http://www.theguardian.com/sitemaps/news.xmlSitemap: http://www.theguardian.com/sitemaps/video.xml… 63 lines not about AI bots User-agent: NewsNowUser-agent: CCBotUser-agent: TurnitinBotUser-agent: PetalBotUser-agent: MoodleBotUser-agent: FacebookBotUser-agent: BytespiderUser-agent: MojeekUser-agent: JenkersBotUser-agent: SeekrUser-agent: YouBotUser-agent: Arquivo-web-crawlerUser-agent: coccocbot-webUser-agent: SeznamBotUser-Agent: PerplexityBotUser-Agent: yacyUser-agent: anthropic-aiUser-agent: ClaudeBotUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: AwarioRssBotUser-agent: AwarioSmartBotUser-agent: SentiOneUser-agent: ImageSiftUser-agent: Applebot-ExtendedUser-agent: YandexAdditionalUser-agent: YandexAdditionalBotUser-agent: scalepostAIUser-agent: BuckUser-agent: meta-externalagentUser-agent: AmazonbotUser-agent: amazon-QBusinessUser-agent: DuckAssistBotUser-agent: Google-CloudVertexBotUser-agent: Amzn-SearchBotUser-agent: AhrefsBotUser-agent: AhrefsSiteAuditDisallow: / License: https://theguardian.com/license.xml

Notes about AI written in the file: "models (LLMs), machine learning and/or artificial intelligence-related"

Everything else we checked

Front door	Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.theguardian.com/ as WalledWebBot, from a residential connection in the United States
Licensing	RSL `License:` https://theguardian.com/license.xml
Page tags	`<meta name="robots" content="max-image-preview:large">` `X-Robots-Tag: bingbot: noarchive`
Anti-bot software	No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	No schema.org structured data on the homepage. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result.
Terms of Service	www.theguardian.com/help/terms-of-service (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context. …any other purpose without our prior written approval - this includes, without limitation, that you shall not use, copy, scrape, reproduce, alter, modify, collect, mine and/or extract the Guardian Content:matched "scrape" · reads like a restriction …se approved in writing by the Guardian, you shall also not use, or facilitate, authorise or permit the use of, any (i) “robot”, “bot”, “spider”, “scraper”, “crawler” or (ii) other automated device, program, technique, tool, process, algorithm or method (whether for data gathering, mining, collection…matched "robot" · reads like a restriction …ing bypassing or circumventing any access controls or use limits of the Guardian Content (including, by way of example, robots.txt internet protocols or similar or successor protocols).matched "robots" · reads like a restriction Our robots.txt notice does not, and shall not, constitute the Guardian’s permission or approval for the purposes of these terms and conditions.matched "robots" · reads like a restriction …s’ attention to specific sections, to make clear that reproduction and/or extraction of Guardian content for commercial text and data mining is not permitted and that the Guardian is the sole seller of advertising on the Guardian Site.matched "text and data mining" · reads like a restriction
Not found	No llms.txt, ai.txt, tdmrep.json or Content-Signal or Content-Usage lines.

History

History starts with our first check

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

One version on record so far (19 Oct 2026).

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.