All sites / Developer and docs
Cloudflare Docs
developers.cloudflare.comAllows AI
Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (User-agent: *), which lets them into the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Allowed
0 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Allowed | |||
| GPTBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Allowed | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Allowed | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# As a condition of accessing this website, you agree to# abide by the following content signals: # (a) If a content-signal = yes, you may collect content# for the corresponding use.# (b) If a content-signal = no, you may not collect content# for the corresponding use.# (c) If the website operator does not include a content# signal for a corresponding use, the website operator# neither grants nor restricts permission via content signal# with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search# results (e.g., returning hyperlinks and short excerpts# from your website's contents). Search does not include# providing AI-generated search summaries.# ai-input: inputting content into one or more AI models# (e.g., retrieval augmented generation, grounding, or other# real-time taking of content for generative AI search# answers).# ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS# RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN# UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS# IN THE DIGITAL SINGLE MARKET. User-agent: *Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: /Disallow: /client-ip-geolocationDisallow: /constellationDisallow: /cdn-cgi/Disallow: /email-security/ Sitemap: https://developers.cloudflare.com/sitemap-index.xml
Notes about AI written in the file: "providing AI-generated search summaries." · "ai-input: inputting content into one or more AI models" · "real-time taking of content for generative AI search" · "ai-train: training or fine-tuning AI models."
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://developers.cloudflare.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| llms.txt | Present · 16.4 KB · "Cloudflare Developer Documentation" · 112 linkshttps://developers.cloudflare.com/llms.txt |
| Content signals | Content-Signal: ai-train=yes, search=yes, ai-input=yes line 31Content-Signal is Cloudflare's CC0 policy (search, ai-input, ai-train). Content-Usage is the IETF AI-preferences draft. Both state preferences; neither blocks anything by itself. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: WebPage, Organization, ImageObject, PostalAddress, ContactPoint, WebSite. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | https://www.cloudflare.com/policies/terms/ (ToS hosted on another domain; not fetched in v0.1) |
| Not found | No ai.txt, tdmrep.json, licensing signals (RSL License, HTTP 402) or robots meta tags. |
History
44 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Oct 2025Signals Added
Content-Signal: ai-train=yes, search=yes, ai-input=yes.Found in the Internet Archive's copies: changed between 1 Aug 2025 and 1 Oct 2025. Evidence: before · afterShow exactly what changed
+ # As a condition of accessing this website, you agree to+ # abide by the following content signals:++ # (a) If a content-signal = yes, you may collect content+ # for the corresponding use.+ # (b) If a content-signal = no, you may not collect content+ # for the corresponding use.+ # (c) If the website operator does not include a content+ # signal for a corresponding use, the website operator+ # neither grants nor restricts permission via content signal+ # with respect to the corresponding use.++ # The content signals and their meanings are:++ # search: building a search index and providing search+ # results (e.g., returning hyperlinks and short excerpts+ # from your website's contents). Search does not include+ # providing AI-generated search summaries.+ # ai-input: inputting content into one or more AI models+ # (e.g., retrieval augmented generation, grounding, or other+ # real-time taking of content for generative AI search+ # answers).+ # ai-train: training or fine-tuning AI models.++ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS+ # RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN+ # UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS+ # IN THE DIGITAL SINGLE MARKET.+User-agent: *+ Content-Signal: ai-train=yes, search=yes, ai-input=yes++ Allow: /Disallow: /client-ip-geolocationDisallow: /plans/+ Disallow: /constellation+ Disallow: /cdn-cgi/+ Disallow: /email-security/- Sitemap: https://developers.cloudflare.com/sitemap.xml+ Sitemap: https://developers.cloudflare.com/sitemap-index.xml
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.