Stack Overflow

stackoverflow.com

Blocks some

Its robots.txt (the public rules file for bots) mentions GPTBot, Google-Extended, Amazonbot and ChatGPT-User, but still lets all 12 major AI bots into the site. (We used the Internet Archive's copy from 10 Sep 2026, because the site would not give our bot its robots.txt: HTTP 418.) But the homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site. The written rules say come in; the front door disagrees.

Last checked 19 Oct 2026. Rules file from the Internet Archive copy of 10 Sep 2026. Everything we checked · History

Bots that train AI Allowed

0 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Allowed

0 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Allowed
GPTBotOpenAI	Allowed	Named, line 113	Says yes
ClaudeBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Allowed	Named, line 113	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
CCBotCommon Crawl	Allowed	Not named; the catch-all rule applies	Says yes
Meta-ExternalAgentMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
BytespiderByteDance	Allowed	Not named; the catch-all rule applies	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Allowed	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Allowed	Not named; the catch-all rule applies	Unknown
Bots for AI search · Allowed
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
PerplexityBotPerplexity	Allowed	Not named; the catch-all rule applies	Suggests yes
AmazonbotAmazon	Allowed	Named, line 113	Says yes
Amzn-SearchBotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Allowed	Not named; the catch-all rule applies	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI	Allowed	Named, line 113	Says not always
Claude-UserAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Perplexity-UserPerplexity	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Allowed	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Named
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 192 lines · 70 lines about AI bots Source

User-Agent: *Content-Signal: search=yesDisallow: /posts/Disallow: /posts?Disallow: /amzn/click/Disallow: /questions/ask/Disallow: /questions/ask?Disallow: /search/Disallow: /search?Disallow: /feeds/Disallow: /feeds?Disallow: /users/login/Disallow: /users/login?Disallow: /users/logout/Disallow: /users/logout?Disallow: /users/filter/Disallow: /users/filter?Disallow: /users/signupDisallow: /users/signup/Disallow: /users/signup?Disallow: /users/authenticate/Disallow: /users/authenticate?Disallow: /users/oauth/*Disallow: /users/flag-summary/Disallow: /users/flair/Disallow: /users/flair?Disallow: /users/activity/Disallow: /users/activity/?Disallow: /users/activity/show?*Disallow: /users/stats/Disallow: /users/*?tab=accountsDisallow: /users/*?tab=activityDisallow: /users/rep/showDisallow: /users/rep/show?Disallow: /unanswered/Disallow: /unanswered?Disallow: /u/Disallow: /messages/Disallow: /api/*Disallow: /review/*Disallow: /*/ivc/*Disallow: /*?*lastactivityDisallow: /users/login/global/request/Disallow: /users/login/global/request?Disallow: /questions/*answertab=*Disallow: /questions/tagged/*+*Disallow: /questions/tagged/*%20*Disallow: /questions/*/answer/submitDisallow: /questions/*?*rq=*Disallow: /tags/*+*Disallow: /tags/*%20*Disallow: /suggested-edits/Disallow: /suggested-edits?Disallow: /ajax/Disallow: /plugins/Disallow: /errorDisallow: /gps/*Disallow: /10mDisallow: /jobs/cv/sign-up-and-createDisallow: /jobs/n/*Disallow: /jobs/a/*Disallow: /jobs/apply/*Disallow: /jobs/companies/n/*Disallow: /jobs/companies/a/*Disallow: /jobs/*/more-jobs-countDisallow: /jobs/email-jobDisallow: /emails/*Disallow: /_/*Disallow: /click?*Disallow: /filter/tags?*Disallow: /landing/r/digest?*Disallow: /admin/*Disallow: /impression.gif?*md=*Disallow: /markup_single.js?*Disallow: /snippet.js?*Disallow: /misc/wp-adminDisallow: /misc/piiDisallow: /misc/usertableDisallow: /misc/questionsdumpDisallow: /misc/answersdumpDisallow: /?*Disallow: /questions?*tagnames=*Disallow: /questions?*tagMode=*Disallow: /questions?*pagesize=*Disallow: /questions?*edited=*Disallow: /questions?*days=*Disallow: /questions/tagged/*?tagnames=*Disallow: /questions/tagged/*?tagMode=*Disallow: /questions/tagged/*?pagesize=*Disallow: /questions/tagged/*?days=*Disallow: /questions/tagged/*?edited=*Disallow: /questions/tagged/*?r=*Disallow: /tags/*?tagnames=*Disallow: /tags/*?tagMode=*Disallow: /tags/*?pagesize=*Disallow: /tags/*?days=*Disallow: /tags/*?edited=*Disallow: /tags/*?r=*Disallow: /users?*tab=*Disallow: /users?*sort=*Disallow: /users?*startdate=*Disallow: /users?*StartDate=*Disallow: /users?*showDeleted=*Disallow: /users?*showdeleted=*Disallow: /users?*filter=*Disallow: /users/rank?*Disallow: /users/auth/gcp*Disallow: /posts/markdown-help*Disallow: /posts/comment-help*Disallow: /tags/*/popup?*origin=*Allow: /… 108 lines not about AI bots User-agent: GooglebotUser-agent: Google-ExtendedUser-agent: amazonbotUser-agent: GPTBotUser-agent: ChatGPT-UserDisallow: /posts/Disallow: /posts?Disallow: /amzn/click/Disallow: /questions/ask/Disallow: /questions/ask?Disallow: /search/Disallow: /search?Disallow: /feeds/Disallow: /feeds?Disallow: /users/login/Disallow: /users/login?Disallow: /users/logout/Disallow: /users/logout?Disallow: /users/filter/Disallow: /users/filter?Disallow: /users/signupDisallow: /users/signup/Disallow: /users/signup?Disallow: /users/authenticate/Disallow: /users/authenticate?Disallow: /users/oauth/*Disallow: /users/flag-summary/Disallow: /users/flair/Disallow: /users/flair?Disallow: /users/activity/Disallow: /users/activity/?Disallow: /users/stats/Disallow: /users/*?tab=accountsDisallow: /users/*?tab=activityDisallow: /users/rep/showDisallow: /users/rep/show?Disallow: /unanswered/Disallow: /unanswered?Disallow: /u/Disallow: /messages/Disallow: /api/*Disallow: /review/*Disallow: /*/ivc/*Disallow: /*?lastactivityDisallow: /users/login/global/request/Disallow: /users/login/global/request?Disallow: /questions/*answertab=*Disallow: /questions/tagged/*+*Disallow: /questions/tagged/*%20*Disallow: /questions/*/answer/submitDisallow: /tags/*+*Disallow: /tags/*%20*Disallow: /suggested-edits/Disallow: /suggested-edits?Disallow: /ajax/Disallow: /plugins/Disallow: /errorDisallow: /gps/*Disallow: /10mDisallow: /jobs/cv/sign-up-and-createDisallow: /jobs/n/*Disallow: /jobs/a/*Disallow: /jobs/apply/*Disallow: /jobs/companies/n/*Disallow: /jobs/companies/a/*Disallow: /jobs/*/more-jobs-countDisallow: /jobs/email-jobDisallow: /emails/*Disallow: /_/*Allow: / User-agent: Googlebot-ImageDisallow: /*/ivc/*Disallow: /users/flair/Disallow: /jobs/n/*Disallow: /jobs/a/*Disallow: /jobs/companies/n/*Disallow: /jobs/companies/a/*Sitemap: https://stackoverflow.com/sitemap.xml … 9 lines not about AI bots

Everything else we checked

Front door	Human check The homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site (HTTP 429).GET https://stackoverflow.com/ as WalledWebBot, from a residential connection in the United States
Content signals	`Content-Signal: search=yes` line 2Content-Signal is Cloudflare's CC0 policy (search, ai-input, ai-train). Content-Usage is the IETF AI-preferences draft. Both state preferences; neither blocks anything by itself.
Page tags	Homepage not read.
Anti-bot software	Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	Homepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of Service	No Terms of Service page found.
Not found	No llms.txt, ai.txt, tdmrep.json or licensing signals (RSL License, HTTP 402).

History

39 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Feb 2026
Signals Named GPTBot, Google-Extended, Amazonbot and ChatGPT-User explicitly, without changing what they may reach. Removed its RSL License: line.
Found in the Internet Archive's copies: changed between 10 Jan 2026 and 13 Feb 2026. Evidence: before · after

Show exactly what changed
- License: https://stackoverflow.com/license.xml
-
User-Agent: *
…

+ User-agent: Googlebot
+ User-agent: Google-Extended
+ User-agent: amazonbot
+ User-agent: GPTBot
+ User-agent: ChatGPT-User
+ Disallow: /posts/
+ Disallow: /posts?
+ Disallow: /amzn/click/
+ Disallow: /questions/ask/
+ Disallow: /questions/ask?
+ Disallow: /search/
+ Disallow: /search?
+ Disallow: /feeds/
+ Disallow: /feeds?
+ Disallow: /users/login/
+ Disallow: /users/login?
+ Disallow: /users/logout/
+ Disallow: /users/logout?
+ Disallow: /users/filter/
+ Disallow: /users/filter?
+ Disallow: /users/signup
+ Disallow: /users/signup/
+ Disallow: /users/signup?
+ Disallow: /users/authenticate/
+ Disallow: /users/authenticate?
+ Disallow: /users/oauth/*
+ Disallow: /users/flag-summary/
+ Disallow: /users/flair/
+ Disallow: /users/flair?
+ Disallow: /users/activity/
+ Disallow: /users/activity/?
+ Disallow: /users/stats/
+ Disallow: /users/*?tab=accounts
+ Disallow: /users/*?tab=activity
+ Disallow: /users/rep/show
+ Disallow: /users/rep/show?
+ Disallow: /unanswered/
+ Disallow: /unanswered?
+ Disallow: /u/
+ Disallow: /messages/
+ Disallow: /api/*
+ Disallow: /review/*
+ Disallow: /*/ivc/*
+ Disallow: /*?lastactivity
+ Disallow: /users/login/global/request/
+ Disallow: /users/login/global/request?
+ Disallow: /questions/*answertab=*
+ Disallow: /questions/tagged/*+*
+ Disallow: /questions/tagged/*%20*
+ Disallow: /questions/*/answer/submit
+ Disallow: /tags/*+*
+ Disallow: /tags/*%20*
+ Disallow: /suggested-edits/
+ Disallow: /suggested-edits?
+ Disallow: /ajax/
… 17 more lines
Aug to Dec 2025
Signals Added Content-Signal: search=yes. Added an RSL License: line pointing to licensing terms.
Found in the Internet Archive's copies: changed between 1 Aug 2025 and 13 Dec 2025. Evidence: before · after

Show exactly what changed
+ License: https://stackoverflow.com/license.xml
+
User-Agent: *
+ Content-Signal: search=yes
Disallow: /posts/
…
Disallow: /users/activity/?
+ Disallow: /users/activity/show?*
Disallow: /users/stats/
…
Disallow: /*/ivc/*
- Disallow: /*?lastactivity
+ Disallow: /*?*lastactivity
Disallow: /users/login/global/request/
…
Disallow: /questions/*/answer/submit
+ Disallow: /questions/*?*rq=*
Disallow: /tags/*+*
…
Disallow: /_/*
- Allow: /
+ Disallow: /click?*
+ Disallow: /filter/tags?*
+ Disallow: /landing/r/digest?*
+ Disallow: /admin/*
+ Disallow: /impression.gif?*md=*
+ Disallow: /markup_single.js?*
+ Disallow: /snippet.js?*
Disallow: /misc/wp-admin
…
Disallow: /misc/answersdump
+ Disallow: /?*
+ Disallow: /questions?*tagnames=*
+ Disallow: /questions?*tagMode=*
+ Disallow: /questions?*pagesize=*
+ Disallow: /questions?*edited=*
+ Disallow: /questions?*days=*
+ Disallow: /questions/tagged/*?tagnames=*
+ Disallow: /questions/tagged/*?tagMode=*
+ Disallow: /questions/tagged/*?pagesize=*
+ Disallow: /questions/tagged/*?days=*
+ Disallow: /questions/tagged/*?edited=*
+ Disallow: /questions/tagged/*?r=*
+ Disallow: /tags/*?tagnames=*
+ Disallow: /tags/*?tagMode=*
+ Disallow: /tags/*?pagesize=*
+ Disallow: /tags/*?days=*
+ Disallow: /tags/*?edited=*
+ Disallow: /tags/*?r=*
+ Disallow: /users?*tab=*
+ Disallow: /users?*sort=*
+ Disallow: /users?*startdate=*
+ Disallow: /users?*StartDate=*
+ Disallow: /users?*showDeleted=*
+ Disallow: /users?*showdeleted=*
+ Disallow: /users?*filter=*
+ Disallow: /users/rank?*
+ Disallow: /users/auth/gcp*
+ Disallow: /posts/markdown-help*
+ Disallow: /posts/comment-help*
… 3 more lines
May 2025
Less blocking Now allowed: ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and PerplexityBot.
Found in the Internet Archive's copies: changed between 7 Apr 2025 and 4 May 2025. Evidence: before · after

Show exactly what changed
Allow: /
- User-agent: Yahoo Pipes 1.0
- Disallow: /
- User-agent: 008
- Disallow: /
- User-agent: voltron
- Disallow: /
- User-agent: Bytespider
- Disallow: /
- User-Agent: PerplexityBot
- Disallow: /
- User-agent: meta-externalagent
- Disallow: /
- User-agent: Applebot-Extended
- Disallow: /
- User-agent: cohere-ai
- Disallow: /
- User-agent: Omgilibot
- Disallow: /
- User-agent: Omgili
- Disallow: /
- User-agent: magpie-crawler
- Disallow: /
- User-agent: TurnitinBot
- Disallow: /
- User-agent: DataForSeoBot
- Disallow: /
- User-agent: Twitterbot
- Disallow: /
- User-agent: peer39_crawler
- Disallow: /
- User-agent: scrapy
- Disallow: /
- User-agent: ClaudeBot
- Disallow: /
- User-agent: Claude-Web
- Disallow: /
- User-agent: CCBot
- Disallow: /
- User-agent: Diffbot
- Disallow: /
- User-agent: FacebookBot
- Disallow: /
- User-agent: YouBot
- Disallow: /
- User-agent: AiZBot
- Disallow: /
- User-agent: AiZBot-Dolma
- Disallow: /
+ Disallow: /misc/wp-admin
+ Disallow: /misc/pii
+ Disallow: /misc/usertable
+ Disallow: /misc/questionsdump
+ Disallow: /misc/answersdump
+
User-agent: Googlebot-Image
Dec 2024
Less blocking Now allowed: Amazonbot.
Found in the Internet Archive's copies: changed between 1 Nov 2024 and 1 Dec 2024. Evidence: before · after

Show exactly what changed
Disallow: /
- User-agent: Amazonbot
- Disallow: /
User-Agent: PerplexityBot
…
Disallow: /
- User-agent: Applebot
- Disallow: /
User-agent: Applebot-Extended
Oct 2024
More blocking Now blocked: ClaudeBot, Applebot-Extended, CCBot and Meta-ExternalAgent.
Found in the Internet Archive's copies: changed between 1 Aug 2024 and 1 Oct 2024. Evidence: before · after

Show exactly what changed
Disallow: /
+ User-agent: meta-externalagent
+ Disallow: /
+ User-agent: Applebot
+ Disallow: /
+ User-agent: Applebot-Extended
+ Disallow: /
+ User-agent: cohere-ai
+ Disallow: /
+ User-agent: Omgilibot
+ Disallow: /
+ User-agent: Omgili
+ Disallow: /
+ User-agent: magpie-crawler
+ Disallow: /
+ User-agent: TurnitinBot
+ Disallow: /
+ User-agent: DataForSeoBot
+ Disallow: /
+ User-agent: Twitterbot
+ Disallow: /
+ User-agent: peer39_crawler
+ Disallow: /
+ User-agent: scrapy
+ Disallow: /
+ User-agent: ClaudeBot
+ Disallow: /
+ User-agent: Claude-Web
+ Disallow: /
+ User-agent: CCBot
+ Disallow: /
+ User-agent: Diffbot
+ Disallow: /
+ User-agent: FacebookBot
+ Disallow: /
+ User-agent: YouBot
+ Disallow: /
+ User-agent: AiZBot
+ Disallow: /
+ User-agent: AiZBot-Dolma
+ Disallow: /
User-agent: Googlebot-Image
Apr 2024
Mixed Now blocked: PerplexityBot. Now allowed: GPTBot.
Found in the Internet Archive's copies: changed between 1 Mar 2024 and 1 Apr 2024. Evidence: before · after

Show exactly what changed
Disallow: /users/rep/show?
- Disallow: /users/prediction-data
- Disallow: /users/prediction-data/
- Disallow: /users/prediction-data?
Disallow: /unanswered/
…
Disallow: /
- User-agent: GPTBot
- Disallow: /
User-agent: Amazonbot
Disallow: /
+ User-Agent: PerplexityBot
+ Disallow: /
User-agent: Googlebot-Image
Feb 2024
Less blocking Now allowed: Google-Extended.
Found in the Internet Archive's copies: changed between 1 Jan 2024 and 1 Feb 2024. Evidence: before · after

Show exactly what changed
Disallow: /
- User-agent: Google-Extended
- Disallow: /
User-agent: Googlebot-Image
Nov 2023
More blocking Now blocked: Google-Extended and Amazonbot.
Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after

Show exactly what changed
Disallow: /
+ User-agent: Amazonbot
+ Disallow: /
+ User-agent: Google-Extended
+ Disallow: /
User-agent: Googlebot-Image
Sep 2023
More blocking Now blocked: GPTBot.
Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after

Show exactly what changed
Disallow: /
+ User-agent: GPTBot
+ Disallow: /
User-agent: Googlebot-Image

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.