{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","subcategory":"forum","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) mentions GPTBot, Google-Extended, Amazonbot and ChatGPT-User, but still lets all 12 major AI bots into the site. (We used the Internet Archive's copy from 10 Sep 2026, because the site would not give our bot its robots.txt: HTTP 418.) But the homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site. The written rules say come in; the front door disagrees.","computed_at":"2026-10-20T03:43:15.556Z","last_checked":"2026-10-19T13:37:20.930Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://stackoverflow.com"},"robots":{"via":{"type":"archive","date":"2026-09-10","url":"https://web.archive.org/web/20260910002727/https://stackoverflow.com/robots.txt","blob":"fe2b3430825eb33ca89e9ad3a2cfa448265c81c0410a40b0682f6f5ffd66b838"},"kind":"refused","http":418,"error":null,"url":"https://stackoverflow.com/robots.txt","hops":[],"blob":"fe2b3430825eb33ca89e9ad3a2cfa448265c81c0410a40b0682f6f5ffd66b838","text":"User-Agent: *\nContent-Signal: search=yes\nDisallow: /posts/\nDisallow: /posts?\nDisallow: /amzn/click/\nDisallow: /questions/ask/\nDisallow: /questions/ask?\nDisallow: /search/\nDisallow: /search?\nDisallow: /feeds/\nDisallow: /feeds?\nDisallow: /users/login/\nDisallow: /users/login?\nDisallow: /users/logout/\nDisallow: /users/logout?\nDisallow: /users/filter/\nDisallow: /users/filter?\nDisallow: /users/signup\nDisallow: /users/signup/\nDisallow: /users/signup?\nDisallow: /users/authenticate/\nDisallow: /users/authenticate?\nDisallow: /users/oauth/*\nDisallow: /users/flag-summary/\nDisallow: /users/flair/\nDisallow: /users/flair?\nDisallow: /users/activity/\nDisallow: /users/activity/?\nDisallow: /users/activity/show?*\nDisallow: /users/stats/\nDisallow: /users/*?tab=accounts\nDisallow: /users/*?tab=activity\nDisallow: /users/rep/show\nDisallow: /users/rep/show?\nDisallow: /unanswered/\nDisallow: /unanswered?\nDisallow: /u/\nDisallow: /messages/\nDisallow: /api/*\nDisallow: /review/*\nDisallow: /*/ivc/*\nDisallow: /*?*lastactivity\nDisallow: /users/login/global/request/\nDisallow: /users/login/global/request?\nDisallow: /questions/*answertab=*\nDisallow: /questions/tagged/*+*\nDisallow: /questions/tagged/*%20*\nDisallow: /questions/*/answer/submit\nDisallow: /questions/*?*rq=*\nDisallow: /tags/*+*\nDisallow: /tags/*%20*\nDisallow: /suggested-edits/\nDisallow: /suggested-edits?\nDisallow: /ajax/\nDisallow: /plugins/\nDisallow: /error\nDisallow: /gps/*\nDisallow: /10m\nDisallow: /jobs/cv/sign-up-and-create\nDisallow: /jobs/n/*\nDisallow: /jobs/a/*\nDisallow: /jobs/apply/*\nDisallow: /jobs/companies/n/*\nDisallow: /jobs/companies/a/*\nDisallow: /jobs/*/more-jobs-count\nDisallow: /jobs/email-job\nDisallow: /emails/*\nDisallow: /_/*\nDisallow: /click?*\nDisallow: /filter/tags?*\nDisallow: /landing/r/digest?*\nDisallow: /admin/*\nDisallow: /impression.gif?*md=*\nDisallow: /markup_single.js?*\nDisallow: /snippet.js?*\nDisallow: /misc/wp-admin\nDisallow: /misc/pii\nDisallow: /misc/usertable\nDisallow: /misc/questionsdump\nDisallow: /misc/answersdump\nDisallow: /?*\nDisallow: /questions?*tagnames=*\nDisallow: /questions?*tagMode=*\nDisallow: /questions?*pagesize=*\nDisallow: /questions?*edited=*\nDisallow: /questions?*days=*\nDisallow: /questions/tagged/*?tagnames=*\nDisallow: /questions/tagged/*?tagMode=*\nDisallow: /questions/tagged/*?pagesize=*\nDisallow: /questions/tagged/*?days=*\nDisallow: /questions/tagged/*?edited=*\nDisallow: /questions/tagged/*?r=*\nDisallow: /tags/*?tagnames=*\nDisallow: /tags/*?tagMode=*\nDisallow: /tags/*?pagesize=*\nDisallow: /tags/*?days=*\nDisallow: /tags/*?edited=*\nDisallow: /tags/*?r=*\nDisallow: /users?*tab=*\nDisallow: /users?*sort=*\nDisallow: /users?*startdate=*\nDisallow: /users?*StartDate=*\nDisallow: /users?*showDeleted=*\nDisallow: /users?*showdeleted=*\nDisallow: /users?*filter=*\nDisallow: /users/rank?*\nDisallow: /users/auth/gcp*\nDisallow: /posts/markdown-help*\nDisallow: /posts/comment-help*\nDisallow: /tags/*/popup?*origin=*\nAllow: /\n\nUser-agent: Googlebot\nUser-agent: Google-Extended\nUser-agent: amazonbot\nUser-agent: GPTBot\nUser-agent: ChatGPT-User\nDisallow: /posts/\nDisallow: /posts?\nDisallow: /amzn/click/\nDisallow: /questions/ask/\nDisallow: /questions/ask?\nDisallow: /search/\nDisallow: /search?\nDisallow: /feeds/\nDisallow: /feeds?\nDisallow: /users/login/\nDisallow: /users/login?\nDisallow: /users/logout/\nDisallow: /users/logout?\nDisallow: /users/filter/\nDisallow: /users/filter?\nDisallow: /users/signup\nDisallow: /users/signup/\nDisallow: /users/signup?\nDisallow: /users/authenticate/\nDisallow: /users/authenticate?\nDisallow: /users/oauth/*\nDisallow: /users/flag-summary/\nDisallow: /users/flair/\nDisallow: /users/flair?\nDisallow: /users/activity/\nDisallow: /users/activity/?\nDisallow: /users/stats/\nDisallow: /users/*?tab=accounts\nDisallow: /users/*?tab=activity\nDisallow: /users/rep/show\nDisallow: /users/rep/show?\nDisallow: /unanswered/\nDisallow: /unanswered?\nDisallow: /u/\nDisallow: /messages/\nDisallow: /api/*\nDisallow: /review/*\nDisallow: /*/ivc/*\nDisallow: /*?lastactivity\nDisallow: /users/login/global/request/\nDisallow: /users/login/global/request?\nDisallow: /questions/*answertab=*\nDisallow: /questions/tagged/*+*\nDisallow: /questions/tagged/*%20*\nDisallow: /questions/*/answer/submit\nDisallow: /tags/*+*\nDisallow: /tags/*%20*\nDisallow: /suggested-edits/\nDisallow: /suggested-edits?\nDisallow: /ajax/\nDisallow: /plugins/\nDisallow: /error\nDisallow: /gps/*\nDisallow: /10m\nDisallow: /jobs/cv/sign-up-and-create\nDisallow: /jobs/n/*\nDisallow: /jobs/a/*\nDisallow: /jobs/apply/*\nDisallow: /jobs/companies/n/*\nDisallow: /jobs/companies/a/*\nDisallow: /jobs/*/more-jobs-count\nDisallow: /jobs/email-job\nDisallow: /emails/*\nDisallow: /_/*\nAllow: /\n\nUser-agent: Googlebot-Image\nDisallow: /*/ivc/*\nDisallow: /users/flair/\nDisallow: /jobs/n/*\nDisallow: /jobs/a/*\nDisallow: /jobs/companies/n/*\nDisallow: /jobs/companies/a/*\nSitemap: https://stackoverflow.com/sitemap.xml\n","refused":{"door":"refused","http":418},"warnings":[],"contentSignals":[{"value":"search=yes","signals":{"search":"yes"},"line":2,"agents":["*"]}],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Googlebot","Google-Extended","amazonbot","GPTBot","ChatGPT-User","Googlebot-Image"],"lines":192},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":182},"any":{"allowed":true,"line":182},"search":{"allowed":true,"line":182},"cart":{"allowed":true,"line":182},"checkout":{"allowed":true,"line":182},"account":{"allowed":true,"line":182}},"disallows":64,"lines":[113,182],"groupRanges":[[113,182]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":182},"any":{"allowed":true,"line":182},"search":{"allowed":true,"line":182},"cart":{"allowed":true,"line":182},"checkout":{"allowed":true,"line":182},"account":{"allowed":true,"line":182}},"disallows":64,"lines":[113,182],"groupRanges":[[113,182]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":182},"any":{"allowed":true,"line":182},"search":{"allowed":true,"line":182},"cart":{"allowed":true,"line":182},"checkout":{"allowed":true,"line":182},"account":{"allowed":true,"line":182}},"disallows":64,"lines":[113,182],"groupRanges":[[113,182]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":182},"any":{"allowed":true,"line":182},"search":{"allowed":true,"line":182},"cart":{"allowed":true,"line":182},"checkout":{"allowed":true,"line":182},"account":{"allowed":true,"line":182}},"disallows":64,"lines":[113,182],"groupRanges":[[113,182]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":182},"any":{"allowed":true,"line":182},"search":{"allowed":true,"line":182},"cart":{"allowed":true,"line":182},"checkout":{"allowed":true,"line":182},"account":{"allowed":true,"line":182}},"disallows":64,"lines":[113,182],"groupRanges":[[113,182]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":111},"any":{"allowed":true,"line":111},"search":{"allowed":true,"line":111},"cart":{"allowed":true,"line":111},"checkout":{"allowed":true,"line":111},"account":{"allowed":true,"line":111}},"disallows":108,"lines":[1,111],"groupRanges":[[1,111]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"challenged","http":429,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"absent","http":404,"url":"https://stackoverflow.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://stackoverflow.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://stackoverflow.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":39,"first":"2023-01-01","archive_points":39,"series":[{"m":"2023-01","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-02","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-03","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-04","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-05","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-06","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-07","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-08","s":"allowed","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2026-02","precision":"month","source":"archive","provisional":false,"between":["2026-01-10","2026-02-13"],"summary":"Named GPTBot, Google-Extended, Amazonbot and ChatGPT-User explicitly, without changing what they may reach. Removed its RSL <code>License:</code> line.","dir":"other","up":[],"down":[],"major":false,"signals":true,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20260110004931/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20260213180442/https://stackoverflow.com/robots.txt"],"diff":[{"t":"del","s":"License: https://stackoverflow.com/license.xml"},{"t":"del","s":""},{"t":"ctx","s":"User-Agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Googlebot"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: amazonbot"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /posts/"},{"t":"add","s":"Disallow: /posts?"},{"t":"add","s":"Disallow: /amzn/click/"},{"t":"add","s":"Disallow: /questions/ask/"},{"t":"add","s":"Disallow: /questions/ask?"},{"t":"add","s":"Disallow: /search/"},{"t":"add","s":"Disallow: /search?"},{"t":"add","s":"Disallow: /feeds/"},{"t":"add","s":"Disallow: /feeds?"},{"t":"add","s":"Disallow: /users/login/"},{"t":"add","s":"Disallow: /users/login?"},{"t":"add","s":"Disallow: /users/logout/"},{"t":"add","s":"Disallow: /users/logout?"},{"t":"add","s":"Disallow: /users/filter/"},{"t":"add","s":"Disallow: /users/filter?"},{"t":"add","s":"Disallow: /users/signup"},{"t":"add","s":"Disallow: /users/signup/"},{"t":"add","s":"Disallow: /users/signup?"},{"t":"add","s":"Disallow: /users/authenticate/"},{"t":"add","s":"Disallow: /users/authenticate?"},{"t":"add","s":"Disallow: /users/oauth/*"},{"t":"add","s":"Disallow: /users/flag-summary/"},{"t":"add","s":"Disallow: /users/flair/"},{"t":"add","s":"Disallow: /users/flair?"},{"t":"add","s":"Disallow: /users/activity/"},{"t":"add","s":"Disallow: /users/activity/?"},{"t":"add","s":"Disallow: /users/stats/"},{"t":"add","s":"Disallow: /users/*?tab=accounts"},{"t":"add","s":"Disallow: /users/*?tab=activity"},{"t":"add","s":"Disallow: /users/rep/show"},{"t":"add","s":"Disallow: /users/rep/show?"},{"t":"add","s":"Disallow: /unanswered/"},{"t":"add","s":"Disallow: /unanswered?"},{"t":"add","s":"Disallow: /u/"},{"t":"add","s":"Disallow: /messages/"},{"t":"add","s":"Disallow: /api/*"},{"t":"add","s":"Disallow: /review/*"},{"t":"add","s":"Disallow: /*/ivc/*"},{"t":"add","s":"Disallow: /*?lastactivity"},{"t":"add","s":"Disallow: /users/login/global/request/"},{"t":"add","s":"Disallow: /users/login/global/request?"},{"t":"add","s":"Disallow: /questions/*answertab=*"},{"t":"add","s":"Disallow: /questions/tagged/*+*"},{"t":"add","s":"Disallow: /questions/tagged/*%20*"},{"t":"add","s":"Disallow: /questions/*/answer/submit"},{"t":"add","s":"Disallow: /tags/*+*"},{"t":"add","s":"Disallow: /tags/*%20*"},{"t":"add","s":"Disallow: /suggested-edits/"},{"t":"add","s":"Disallow: /suggested-edits?"},{"t":"add","s":"Disallow: /ajax/"},{"t":"gap","s":"… 17 more lines"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2025-12","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-12-13"],"summary":"Added <code>Content-Signal: search=yes</code>. Added an RSL <code>License:</code> line pointing to licensing terms.","dir":"other","up":[],"down":[],"major":false,"signals":true,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20250801163142/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20251213134544/https://stackoverflow.com/robots.txt"],"diff":[{"t":"add","s":"License: https://stackoverflow.com/license.xml"},{"t":"add","s":""},{"t":"ctx","s":"User-Agent: *"},{"t":"add","s":"Content-Signal: search=yes"},{"t":"ctx","s":"Disallow: /posts/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /users/activity/?"},{"t":"add","s":"Disallow: /users/activity/show?*"},{"t":"ctx","s":"Disallow: /users/stats/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*/ivc/*"},{"t":"del","s":"Disallow: /*?lastactivity"},{"t":"add","s":"Disallow: /*?*lastactivity"},{"t":"ctx","s":"Disallow: /users/login/global/request/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /questions/*/answer/submit"},{"t":"add","s":"Disallow: /questions/*?*rq=*"},{"t":"ctx","s":"Disallow: /tags/*+*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /_/*"},{"t":"del","s":"Allow: /"},{"t":"add","s":"Disallow: /click?*"},{"t":"add","s":"Disallow: /filter/tags?*"},{"t":"add","s":"Disallow: /landing/r/digest?*"},{"t":"add","s":"Disallow: /admin/*"},{"t":"add","s":"Disallow: /impression.gif?*md=*"},{"t":"add","s":"Disallow: /markup_single.js?*"},{"t":"add","s":"Disallow: /snippet.js?*"},{"t":"ctx","s":"Disallow: /misc/wp-admin"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /misc/answersdump"},{"t":"add","s":"Disallow: /?*"},{"t":"add","s":"Disallow: /questions?*tagnames=*"},{"t":"add","s":"Disallow: /questions?*tagMode=*"},{"t":"add","s":"Disallow: /questions?*pagesize=*"},{"t":"add","s":"Disallow: /questions?*edited=*"},{"t":"add","s":"Disallow: /questions?*days=*"},{"t":"add","s":"Disallow: /questions/tagged/*?tagnames=*"},{"t":"add","s":"Disallow: /questions/tagged/*?tagMode=*"},{"t":"add","s":"Disallow: /questions/tagged/*?pagesize=*"},{"t":"add","s":"Disallow: /questions/tagged/*?days=*"},{"t":"add","s":"Disallow: /questions/tagged/*?edited=*"},{"t":"add","s":"Disallow: /questions/tagged/*?r=*"},{"t":"add","s":"Disallow: /tags/*?tagnames=*"},{"t":"add","s":"Disallow: /tags/*?tagMode=*"},{"t":"add","s":"Disallow: /tags/*?pagesize=*"},{"t":"add","s":"Disallow: /tags/*?days=*"},{"t":"add","s":"Disallow: /tags/*?edited=*"},{"t":"add","s":"Disallow: /tags/*?r=*"},{"t":"add","s":"Disallow: /users?*tab=*"},{"t":"add","s":"Disallow: /users?*sort=*"},{"t":"add","s":"Disallow: /users?*startdate=*"},{"t":"add","s":"Disallow: /users?*StartDate=*"},{"t":"add","s":"Disallow: /users?*showDeleted=*"},{"t":"add","s":"Disallow: /users?*showdeleted=*"},{"t":"add","s":"Disallow: /users?*filter=*"},{"t":"add","s":"Disallow: /users/rank?*"},{"t":"add","s":"Disallow: /users/auth/gcp*"},{"t":"add","s":"Disallow: /posts/markdown-help*"},{"t":"add","s":"Disallow: /posts/comment-help*"},{"t":"gap","s":"… 3 more lines"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2025-05","precision":"month","source":"archive","provisional":false,"between":["2025-04-07","2025-05-04"],"summary":"Now allowed: ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and PerplexityBot.","dir":"down","up":[],"down":["ClaudeBot","Applebot-Extended","CCBot","Meta-ExternalAgent","Bytespider","PerplexityBot"],"major":true,"signals":false,"status_from":"partial","status_to":"allowed","evidence":["https://web.archive.org/web/20250407014700/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20250504231302/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Allow: /"},{"t":"del","s":"User-agent: Yahoo Pipes 1.0"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: 008"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: voltron"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Bytespider"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-Agent: PerplexityBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: meta-externalagent"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Applebot-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: cohere-ai"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Omgilibot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Omgili"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: magpie-crawler"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: TurnitinBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: DataForSeoBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Twitterbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: peer39_crawler"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: scrapy"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: ClaudeBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Claude-Web"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: CCBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: Diffbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: FacebookBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: YouBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: AiZBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":"User-agent: AiZBot-Dolma"},{"t":"del","s":"Disallow: /"},{"t":"add","s":"Disallow: /misc/wp-admin"},{"t":"add","s":"Disallow: /misc/pii"},{"t":"add","s":"Disallow: /misc/usertable"},{"t":"add","s":"Disallow: /misc/questionsdump"},{"t":"add","s":"Disallow: /misc/answersdump"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Googlebot-Image"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2024-12","precision":"month","source":"archive","provisional":false,"between":["2024-11-01","2024-12-01"],"summary":"Now allowed: Amazonbot.","dir":"down","up":[],"down":["Amazonbot"],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241101012522/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20241201113654/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"del","s":"User-agent: Amazonbot"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":"User-Agent: PerplexityBot"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"del","s":"User-agent: Applebot"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Applebot-Extended"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2024-10","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-10-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, CCBot and Meta-ExternalAgent.","dir":"up","up":["ClaudeBot","Applebot-Extended","CCBot","Meta-ExternalAgent"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801011925/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20241001000547/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: TurnitinBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: DataForSeoBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Twitterbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: peer39_crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: scrapy"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: YouBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: AiZBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: AiZBot-Dolma"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Googlebot-Image"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2024-04","precision":"month","source":"archive","provisional":false,"between":["2024-03-01","2024-04-01"],"summary":"Now blocked: PerplexityBot. Now allowed: GPTBot.","dir":"mixed","up":["PerplexityBot"],"down":["GPTBot"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240301034635/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20240401024304/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /users/rep/show?"},{"t":"del","s":"Disallow: /users/prediction-data"},{"t":"del","s":"Disallow: /users/prediction-data/"},{"t":"del","s":"Disallow: /users/prediction-data?"},{"t":"ctx","s":"Disallow: /unanswered/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"User-Agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Googlebot-Image"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2024-02","precision":"month","source":"archive","provisional":false,"between":["2024-01-01","2024-02-01"],"summary":"Now allowed: Google-Extended.","dir":"down","up":[],"down":["Google-Extended"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240101000541/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20240201001637/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Googlebot-Image"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended and Amazonbot.","dir":"up","up":["Google-Extended","Amazonbot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001034346/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20231101001314/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Googlebot-Image"}]},{"domain":"stackoverflow.com","name":"Stack Overflow","category":"social","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801012935/https://stackoverflow.com/robots.txt","https://web.archive.org/web/20230901004203/https://stackoverflow.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":"User-agent: Googlebot-Image"}]}]}