Wikipedia
wikipedia.orgAllows AI
Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (User-agent: *), which lets them into the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Allowed
0 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Allowed | |||
| GPTBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Allowed | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Allowed | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# robots.txt for http://www.wikipedia.org/ and friends## Please note: There are a lot of pages on this site, and there are# some misbehaved spiders out there that go _way_ too fast. If you're# irresponsible, your access to the site may be blocked.# # Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN# and ignoring 429 ratelimit responses, claims to respect robots:# http://mj12bot.com/User-agent: MJ12botDisallow: / # advertising-related bots:User-agent: Mediapartners-Google*Disallow: / # Wikipedia work bots:User-agent: IsraBotDisallow: User-agent: OrthogaffeDisallow: # Crawlers that are kind enough to obey, but which we'd rather not have# unless they're feeding search engines.User-agent: UbiCrawlerDisallow: / User-agent: DOCDisallow: / User-agent: ZaoDisallow: / # Some bots are known to be trouble, particularly those designed to copy# entire sites. Please obey robots.txt.User-agent: sitecheck.internetseer.comDisallow: / User-agent: ZealbotDisallow: / User-agent: MSIECrawlerDisallow: / User-agent: SiteSnaggerDisallow: / User-agent: WebStripperDisallow: / User-agent: WebCopierDisallow: / User-agent: FetchDisallow: / User-agent: Offline ExplorerDisallow: / User-agent: TeleportDisallow: / User-agent: TeleportProDisallow: / User-agent: WebZIPDisallow: / User-agent: linkoDisallow: / User-agent: HTTrackDisallow: / User-agent: Microsoft.URL.ControlDisallow: / User-agent: XenuDisallow: / User-agent: larbinDisallow: / User-agent: libwwwDisallow: / User-agent: ZyBORGDisallow: / User-agent: Download NinjaDisallow: / # Misbehaving, requests much too fastUser-agent: fastDisallow: / # Sorry, wget in its recursive mode is a frequent problem.# Please read the man page and use it properly; there is a# --wait option you can use to set the delay between hits,# for instance.User-agent: wgetDisallow: / # The 'grub' distributed client has been *very* poorly behaved.User-agent: grub-clientDisallow: / # Doesn't follow robots.txt anyway, but...User-agent: k2spiderDisallow: / # Hits many times per second, not acceptable# http://www.nameprotect.com/botinfo.htmlUser-agent: NPBotDisallow: / # A capture bot, downloads gazillions of pages with no public benefit# http://www.webreaper.net/User-agent: WebReaperDisallow: / # Per their statement, semrushbot respects crawl-delay directives# We want them to overall stay within reasonable request rates to# the backend (20 rps); keeping in mind that the crawl-delay will# be applied by site and not globally by the bot, 5 seconds seem# like a reasonable approximationUser-agent: SemrushBotCrawl-delay: 5 ## Friendly, low-speed bots are welcome viewing article pages, but not# dynamically-generated pages please.## Inktomi's "Slurp" can read a minimum delay between hits; if your# bot supports such a thing using the 'Crawl-delay' or another# instruction, please let us know.## There is a special exception for API mobileview to allow dynamic# mobile web & app views to load section content.# These views aren't HTTP-cached but use parser cache aggressively# and don't expose Special-pages and other uncached content.## Another exception is for REST API documentation, located at# /api/rest_v1/?doc.#User-agent: *Allow: /w/api.php?action=mobileview&Allow: /w/load.php?Allow: /api/rest_v1/?docAllow: /w/rest.php/site/v1/sitemapDisallow: /w/Disallow: /api/Disallow: /trap/Disallow: /wiki/Special:Disallow: /wiki/Spezial:Disallow: /wiki/Spesial:Disallow: /wiki/Special%3ADisallow: /wiki/Spezial%3ADisallow: /wiki/Spesial%3A## ar:Disallow: /wiki/%D8%AE%D8%A7%D8%B5:SearchDisallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch## dewiki:# T6937# sensible deletion and meta user discussion pages:Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/Disallow: /wiki/Wikipedia:Löschkandidaten/Disallow: /wiki/Wikipedia:Vandalensperrung/Disallow: /wiki/Wikipedia:Benutzersperrung/Disallow: /wiki/Wikipedia:Vermittlungsausschuss/Disallow: /wiki/Wikipedia:Administratoren/Probleme/Disallow: /wiki/Wikipedia:Adminkandidaturen/Disallow: /wiki/Wikipedia:Qualitätssicherung/Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/# 4937#5Disallow: /wiki/Wikipedia:Vandalismusmeldung/Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/Disallow: /wiki/Wikipedia:Löschprüfung/Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/Disallow: /wiki/Wikipedia:Administratoren/Notizen/Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/# T14111Disallow: /wiki/Wikipedia:Checkuser/Disallow: /wiki/Wikipedia_Diskussion:Checkuser/Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/# T15961Disallow: /wiki/Wikipedia:Spam-Blacklist-LogDisallow: /wiki/Wikipedia%3ASpam-Blacklist-LogDisallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-LogDisallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log## enwiki:# Folks get annoyed when VfD discussions end up the number 1 google hit for# their name. See T6776Disallow: /wiki/Wikipedia:Articles_for_deletion/Disallow: /wiki/Wikipedia%3AArticles_for_deletion/Disallow: /wiki/Wikipedia:Votes_for_deletion/Disallow: /wiki/Wikipedia%3AVotes_for_deletion/Disallow: /wiki/Wikipedia:Pages_for_deletion/Disallow: /wiki/Wikipedia%3APages_for_deletion/Disallow: /wiki/Wikipedia:Miscellany_for_deletion/Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/Disallow: /wiki/Wikipedia:Miscellaneous_deletion/Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/Disallow: /wiki/Wikipedia:Copyright_problemsDisallow: /wiki/Wikipedia%3ACopyright_problemsDisallow: /wiki/Wikipedia:Protected_titles/Disallow: /wiki/Wikipedia%3AProtected_titles/# T15398Disallow: /wiki/Wikipedia:WikiProject_Spam/Disallow: /wiki/Wikipedia%3AWikiProject_Spam/# T16075Disallow: /wiki/MediaWiki:Spam-blacklistDisallow: /wiki/MediaWiki%3ASpam-blacklistDisallow: /wiki/MediaWiki_talk:Spam-blacklistDisallow: /wiki/MediaWiki_talk%3ASpam-blacklist# T13261Disallow: /wiki/Wikipedia:Requests_for_arbitration/Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/Disallow: /wiki/Wikipedia:Requests_for_comment/Disallow: /wiki/Wikipedia%3ARequests_for_comment/Disallow: /wiki/Wikipedia:Requests_for_adminship/Disallow: /wiki/Wikipedia%3ARequests_for_adminship/# T12288Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/# T16793Disallow: /wiki/Wikipedia:Changing_usernameDisallow: /wiki/Wikipedia%3AChanging_usernameDisallow: /wiki/Wikipedia:Changing_username/Disallow: /wiki/Wikipedia%3AChanging_username/Disallow: /wiki/Wikipedia_talk:Changing_usernameDisallow: /wiki/Wikipedia_talk%3AChanging_usernameDisallow: /wiki/Wikipedia_talk:Changing_username/Disallow: /wiki/Wikipedia_talk%3AChanging_username/## eswiki:# T8746Disallow: /wiki/Wikipedia:Consultas_de_borrado/Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/## fiwiki:# T10695Disallow: /wiki/Wikipedia:Poistettavat_sivutDisallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:Disallow: /wiki/Käyttäjä:Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:Disallow: /wiki/Keskustelu_käyttäjästä:Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/Disallow: /wiki/Wikipedia:Ylläpitäjät/## hewiki:Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:SearchDisallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch#T11517Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגניםDisallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגניםDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9DDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9DDisallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגניםDisallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגניםDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9DDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D## huwiki:Disallow: /wiki/Speci%C3%A1lis:SearchDisallow: /wiki/Speci%C3%A1lis%3ASearch## itwiki:# T7545Disallow: /wiki/Wikipedia:Pagine_da_cancellareDisallow: /wiki/Wikipedia%3APagine_da_cancellareDisallow: /wiki/Wikipedia:Utenti_problematiciDisallow: /wiki/Wikipedia%3AUtenti_problematiciDisallow: /wiki/Wikipedia:Vandalismi_in_corsoDisallow: /wiki/Wikipedia%3AVandalismi_in_corsoDisallow: /wiki/Wikipedia:AmministratoriDisallow: /wiki/Wikipedia%3AAmministratoriDisallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificataDisallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificataDisallow: /wiki/Categoria:Da_cancellare_subitoDisallow: /wiki/Categoria%3ADa_cancellare_subitoDisallow: /wiki/Wikipedia:Sospette_violazioni_di_copyrightDisallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyrightDisallow: /wiki/Categoria:Da_controllare_per_copyrightDisallow: /wiki/Categoria%3ADa_controllare_per_copyrightDisallow: /wiki/Progetto:Rimozione_contributi_sospettiDisallow: /wiki/Progetto%3ARimozione_contributi_sospettiDisallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyrightDisallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyrightDisallow: /wiki/Progetto:CococoDisallow: /wiki/Progetto%3ACococoDisallow: /wiki/Discussioni_progetto:CococoDisallow: /wiki/Discussioni_progetto%3ACococo## jawikiDisallow: /wiki/%E7%89%B9%E5%88%A5:SearchDisallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch# T7239Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BCDisallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC# nowiki# T13432Disallow: /wiki/Bruker:Disallow: /wiki/Bruker%3ADisallow: /wiki/BrukerdiskusjonDisallow: /wiki/Wikipedia:AdministratorerDisallow: /wiki/Wikipedia%3AAdministratorerDisallow: /wiki/Wikipedia-diskusjon:AdministratorerDisallow: /wiki/Wikipedia-diskusjon%3AAdministratorerDisallow: /wiki/Wikipedia:SlettingDisallow: /wiki/Wikipedia%3ASlettingDisallow: /wiki/Wikipedia-diskusjon:SlettingDisallow: /wiki/Wikipedia-diskusjon%3ASletting## plwiki# T10067Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99ciaDisallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99ciaDisallow: /wiki/Wikipedia:Do_usuni%C4%99ciaDisallow: /wiki/Wikipedia%3ADo_usuni%C4%99ciaDisallow: /wiki/Wikipedia:SDU/Disallow: /wiki/Wikipedia%3ASDU/Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskichDisallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich## ptwiki:# T7394Disallow: /wiki/Wikipedia:Páginas_para_eliminar/Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/## rowiki:# T14546Disallow: /wiki/Wikipedia:Pagini_de_%C5%9FtersDisallow: /wiki/Wikipedia%3APagini_de_%C5%9FtersDisallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9FtersDisallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters## ruwiki:Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:SearchDisallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch## svwiki:# T12229Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_raderingDisallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_raderingDisallow: /wiki/Wikipedia:Sidor_föreslagna_för_raderingDisallow: /wiki/AnvändareDisallow: /wiki/Anv%C3%A4ndareDisallow: /wiki/AnvändardiskussionDisallow: /wiki/Anv%C3%A4ndardiskussionDisallow: /wiki/Wikipedia:Skyddade_sidnamnDisallow: /wiki/Wikipedia%3ASkyddade_sidnamn# T13291Disallow: /wiki/Wikipedia:Sidor_som_bör_raderasDisallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderasDisallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas## zhwiki:# T7104Disallow: /wiki/Wikipedia:删除投票/侵权Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83Disallow: /wiki/Wikipedia:删除投票和请求Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82Disallow: /wiki/Category:快速删除候选Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89Disallow: /wiki/Category:维基百科需要翻译的文章Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0## sister projects## enwikinews:# T7340Disallow: /wiki/Portal:Prepared_stories/Disallow: /wiki/Portal%3APrepared_stories/## itwikinews# T11138Disallow: /wiki/Wikinotizie:Richieste_di_cancellazioneDisallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyrightDisallow: /wiki/Categoria:Da_cancellare_subitoDisallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyrightDisallow: /wiki/Wikinotizie:Storie_in_preparazione## enwikiquote:# T17095Disallow: /wiki/Wikiquote:Votes_for_deletion/Disallow: /wiki/Wikiquote%3AVotes_for_deletion/Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/## enwikibooksDisallow: /wiki/Wikibooks:Votes_for_deletion# Sitemap: https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0 ###----------------------------------------------------------###### Localisable part of robots.txt for en.wikipedia.org## Edit at https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit# Don't add newlines here. All rules set here are active for every user-agent.## Please check any changes using a syntax validator# Enter https://en.wikipedia.org/robots.txt as the URL to check.## https://phabricator.wikimedia.org/T16075Disallow: /wiki/MediaWiki:Spam-blacklistDisallow: /wiki/MediaWiki%3ASpam-blacklistDisallow: /wiki/MediaWiki_talk:Spam-blacklistDisallow: /wiki/MediaWiki_talk%3ASpam-blacklistDisallow: /wiki/Wikipedia:WikiProject_SpamDisallow: /wiki/Wikipedia_talk:WikiProject_Spam## Folks get annoyed when XfD discussions end up the number 1 google hit for# their name. # https://phabricator.wikimedia.org/T16075Disallow: /wiki/Wikipedia:Articles_for_deletionDisallow: /wiki/Wikipedia%3AArticles_for_deletionDisallow: /wiki/Wikipedia:Votes_for_deletionDisallow: /wiki/Wikipedia%3AVotes_for_deletionDisallow: /wiki/Wikipedia:Pages_for_deletionDisallow: /wiki/Wikipedia%3APages_for_deletionDisallow: /wiki/Wikipedia:Miscellany_for_deletionDisallow: /wiki/Wikipedia%3AMiscellany_for_deletionDisallow: /wiki/Wikipedia:Miscellaneous_deletionDisallow: /wiki/Wikipedia%3AMiscellaneous_deletionDisallow: /wiki/Wikipedia:Categories_for_discussionDisallow: /wiki/Wikipedia%3ACategories_for_discussionDisallow: /wiki/Wikipedia:Templates_for_deletionDisallow: /wiki/Wikipedia%3ATemplates_for_deletionDisallow: /wiki/Wikipedia:Redirects_for_discussionDisallow: /wiki/Wikipedia%3ARedirects_for_discussionDisallow: /wiki/Wikipedia:Deletion_reviewDisallow: /wiki/Wikipedia%3ADeletion_reviewDisallow: /wiki/Wikipedia:WikiProject_Deletion_sortingDisallow: /wiki/Wikipedia%3AWikiProject_Deletion_sortingDisallow: /wiki/Wikipedia:Files_for_deletionDisallow: /wiki/Wikipedia%3AFiles_for_deletionDisallow: /wiki/Wikipedia:Files_for_discussionDisallow: /wiki/Wikipedia%3AFiles_for_discussionDisallow: /wiki/Wikipedia:Possibly_unfree_filesDisallow: /wiki/Wikipedia%3APossibly_unfree_files## https://phabricator.wikimedia.org/T12288Disallow: /wiki/Wikipedia_talk:Articles_for_deletionDisallow: /wiki/Wikipedia_talk%3AArticles_for_deletionDisallow: /wiki/Wikipedia_talk:Votes_for_deletionDisallow: /wiki/Wikipedia_talk%3AVotes_for_deletionDisallow: /wiki/Wikipedia_talk:Pages_for_deletionDisallow: /wiki/Wikipedia_talk%3APages_for_deletionDisallow: /wiki/Wikipedia_talk:Miscellany_for_deletionDisallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletionDisallow: /wiki/Wikipedia_talk:Miscellaneous_deletionDisallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletionDisallow: /wiki/Wikipedia_talk:Templates_for_deletionDisallow: /wiki/Wikipedia_talk%3ATemplates_for_deletionDisallow: /wiki/Wikipedia_talk:Categories_for_discussionDisallow: /wiki/Wikipedia_talk%3ACategories_for_discussionDisallow: /wiki/Wikipedia_talk:Deletion_reviewDisallow: /wiki/Wikipedia_talk%3ADeletion_reviewDisallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sortingDisallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sortingDisallow: /wiki/Wikipedia_talk:Files_for_deletionDisallow: /wiki/Wikipedia_talk%3AFiles_for_deletionDisallow: /wiki/Wikipedia_talk:Files_for_discussionDisallow: /wiki/Wikipedia_talk%3AFiles_for_discussionDisallow: /wiki/Wikipedia_talk:Possibly_unfree_filesDisallow: /wiki/Wikipedia_talk%3APossibly_unfree_files#Disallow: /wiki/Wikipedia:Copyright_problemsDisallow: /wiki/Wikipedia%3ACopyright_problemsDisallow: /wiki/Wikipedia_talk:Copyright_problemsDisallow: /wiki/Wikipedia_talk%3ACopyright_problemsDisallow: /wiki/Wikipedia:Suspected_copyright_violationsDisallow: /wiki/Wikipedia%3ASuspected_copyright_violationsDisallow: /wiki/Wikipedia_talk:Suspected_copyright_violationsDisallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violationsDisallow: /wiki/Wikipedia:Contributor_copyright_investigationsDisallow: /wiki/Wikipedia%3AContributor_copyright_investigationsDisallow: /wiki/Wikipedia:Contributor_copyright_investigationsDisallow: /wiki/Wikipedia%3AContributor_copyright_investigationsDisallow: /wiki/Wikipedia_talk:Contributor_copyright_investigationsDisallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigationsDisallow: /wiki/Wikipedia_talk:Contributor_copyright_investigationsDisallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigationsDisallow: /wiki/Wikipedia:Protected_titlesDisallow: /wiki/Wikipedia%3AProtected_titlesDisallow: /wiki/Wikipedia_talk:Protected_titlesDisallow: /wiki/Wikipedia_talk%3AProtected_titlesDisallow: /wiki/Wikipedia:Articles_for_creationDisallow: /wiki/Wikipedia%3AArticles_for_creationDisallow: /wiki/Wikipedia_talk:Articles_for_creationDisallow: /wiki/Wikipedia_talk%3AArticles_for_creationDisallow: /wiki/Wikipedia_talk:Article_wizardDisallow: /wiki/Wikipedia_talk%3AArticle_wizard## https://phabricator.wikimedia.org/T13261Disallow: /wiki/Wikipedia:Requests_for_arbitrationDisallow: /wiki/Wikipedia%3ARequests_for_arbitrationDisallow: /wiki/Wikipedia_talk:Requests_for_arbitrationDisallow: /wiki/Wikipedia_talk%3ARequests_for_arbitrationDisallow: /wiki/Wikipedia:Requests_for_commentDisallow: /wiki/Wikipedia%3ARequests_for_commentDisallow: /wiki/Wikipedia_talk:Requests_for_commentDisallow: /wiki/Wikipedia_talk%3ARequests_for_commentDisallow: /wiki/Wikipedia:Requests_for_adminshipDisallow: /wiki/Wikipedia%3ARequests_for_adminshipDisallow: /wiki/Wikipedia_talk:Requests_for_adminshipDisallow: /wiki/Wikipedia_talk%3ARequests_for_adminship## https://phabricator.wikimedia.org/T14111Disallow: /wiki/Wikipedia:Requests_for_checkuserDisallow: /wiki/Wikipedia%3ARequests_for_checkuserDisallow: /wiki/Wikipedia_talk:Requests_for_checkuserDisallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser## https://phabricator.wikimedia.org/T15398Disallow: /wiki/Wikipedia:WikiProject_SpamDisallow: /wiki/Wikipedia%3AWikiProject_Spam## https://phabricator.wikimedia.org/T16793Disallow: /wiki/Wikipedia:Changing_usernameDisallow: /wiki/Wikipedia%3AChanging_usernameDisallow: /wiki/Wikipedia:Changing_usernameDisallow: /wiki/Wikipedia%3AChanging_usernameDisallow: /wiki/Wikipedia_talk:Changing_usernameDisallow: /wiki/Wikipedia_talk%3AChanging_usernameDisallow: /wiki/Wikipedia_talk:Changing_usernameDisallow: /wiki/Wikipedia_talk%3AChanging_username#Disallow: /wiki/Wikipedia:Administrators%27_noticeboardDisallow: /wiki/Wikipedia%3AAdministrators%27_noticeboardDisallow: /wiki/Wikipedia_talk:Administrators%27_noticeboardDisallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboardDisallow: /wiki/Wikipedia:Community_sanction_noticeboardDisallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboardDisallow: /wiki/Wikipedia_talk:Community_sanction_noticeboardDisallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboardDisallow: /wiki/Wikipedia:Bureaucrats%27_noticeboardDisallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboardDisallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboardDisallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard#Disallow: /wiki/Wikipedia:Sockpuppet_investigationsDisallow: /wiki/Wikipedia%3ASockpuppet_investigationsDisallow: /wiki/Wikipedia_talk:Sockpuppet_investigationsDisallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations#Disallow: /wiki/Wikipedia:Neutral_point_of_view/NoticeboardDisallow: /wiki/Wikipedia%3ANeutral_point_of_view/NoticeboardDisallow: /wiki/Wikipedia_talk:Neutral_point_of_view/NoticeboardDisallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard#Disallow: /wiki/Wikipedia:No_original_research/noticeboardDisallow: /wiki/Wikipedia%3ANo_original_research/noticeboardDisallow: /wiki/Wikipedia_talk:No_original_research/noticeboardDisallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard#Disallow: /wiki/Wikipedia:Fringe_theories/NoticeboardDisallow: /wiki/Wikipedia%3AFringe_theories/NoticeboardDisallow: /wiki/Wikipedia_talk:Fringe_theories/NoticeboardDisallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard#Disallow: /wiki/Wikipedia:Conflict_of_interest/NoticeboardDisallow: /wiki/Wikipedia%3AConflict_of_interest/NoticeboardDisallow: /wiki/Wikipedia_talk:Conflict_of_interest/NoticeboardDisallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard#Disallow: /wiki/Wikipedia:Long-term_abuseDisallow: /wiki/Wikipedia%3ALong-term_abuseDisallow: /wiki/Wikipedia_talk:Long-term_abuseDisallow: /wiki/Wikipedia_talk%3ALong-term_abuseDisallow: /wiki/Wikipedia:Long_term_abuseDisallow: /wiki/Wikipedia%3ALong_term_abuseDisallow: /wiki/Wikipedia_talk:Long_term_abuseDisallow: /wiki/Wikipedia_talk%3ALong_term_abuse#Disallow: /wiki/Wikipedia:Wikiquette_assistanceDisallow: /wiki/Wikipedia%3AWikiquette_assistance#Disallow: /wiki/Wikipedia:Abuse_reportsDisallow: /wiki/Wikipedia%3AAbuse_reportsDisallow: /wiki/Wikipedia_talk:Abuse_reportsDisallow: /wiki/Wikipedia_talk%3AAbuse_reportsDisallow: /wiki/Wikipedia:Abuse_responseDisallow: /wiki/Wikipedia%3AAbuse_responseDisallow: /wiki/Wikipedia_talk:Abuse_responseDisallow: /wiki/Wikipedia_talk%3AAbuse_response#Disallow: /wiki/Wikipedia:Reliable_sources/NoticeboardDisallow: /wiki/Wikipedia%3AReliable_sources/NoticeboardDisallow: /wiki/Wikipedia_talk:Reliable_sources/NoticeboardDisallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard#Disallow: /wiki/Wikipedia:Suspected_sock_puppetsDisallow: /wiki/Wikipedia%3ASuspected_sock_puppetsDisallow: /wiki/Wikipedia_talk:Suspected_sock_puppetsDisallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets#Disallow: /wiki/Wikipedia:Biographies_of_living_persons/NoticeboardDisallow: /wiki/Wikipedia%3ABiographies_of_living_persons/NoticeboardDisallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/NoticeboardDisallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/NoticeboardDisallow: /wiki/Wikipedia:Biographies_of_living_persons%2FNoticeboardDisallow: /wiki/Wikipedia%3ABiographies_of_living_persons%2FNoticeboardDisallow: /wiki/Wikipedia_talk:Biographies_of_living_persons%2FNoticeboardDisallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons%2FNoticeboard#Disallow: /wiki/Wikipedia:Content_noticeboardDisallow: /wiki/Wikipedia%3AContent_noticeboardDisallow: /wiki/Wikipedia_talk:Content_noticeboardDisallow: /wiki/Wikipedia_talk%3AContent_noticeboard#Disallow: /wiki/Template:EditnoticesDisallow: /wiki/Template%3AEditnotices#Disallow: /wiki/Wikipedia:ArbitrationDisallow: /wiki/Wikipedia%3AArbitrationDisallow: /wiki/Wikipedia_talk:ArbitrationDisallow: /wiki/Wikipedia_talk%3AArbitration#Disallow: /wiki/Wikipedia:Arbitration_CommitteeDisallow: /wiki/Wikipedia%3AArbitration_CommitteeDisallow: /wiki/Wikipedia_talk:Arbitration_CommitteeDisallow: /wiki/Wikipedia_talk%3AArbitration_Committee#Disallow: /wiki/Wikipedia:Arbitration_Committee_ElectionsDisallow: /wiki/Wikipedia%3AArbitration_Committee_ElectionsDisallow: /wiki/Wikipedia_talk:Arbitration_Committee_ElectionsDisallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections#Disallow: /wiki/Wikipedia:Mediation_CommitteeDisallow: /wiki/Wikipedia%3AMediation_CommitteeDisallow: /wiki/Wikipedia_talk:Mediation_CommitteeDisallow: /wiki/Wikipedia_talk%3AMediation_Committee#Disallow: /wiki/Wikipedia:Mediation_Cabal/CasesDisallow: /wiki/Wikipedia%3AMediation_Cabal/Cases#Disallow: /wiki/Wikipedia:Requests_for_bureaucratshipDisallow: /wiki/Wikipedia%3ARequests_for_bureaucratshipDisallow: /wiki/Wikipedia_talk:Requests_for_bureaucratshipDisallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship#Disallow: /wiki/Wikipedia:Administrator_reviewDisallow: /wiki/Wikipedia%3AAdministrator_reviewDisallow: /wiki/Wikipedia_talk:Administrator_reviewDisallow: /wiki/Wikipedia_talk%3AAdministrator_review#Disallow: /wiki/Wikipedia:Editor_reviewDisallow: /wiki/Wikipedia%3AEditor_reviewDisallow: /wiki/Wikipedia_talk:Editor_reviewDisallow: /wiki/Wikipedia_talk%3AEditor_review#Disallow: /wiki/Wikipedia:Article_IncubatorDisallow: /wiki/Wikipedia%3AArticle_IncubatorDisallow: /wiki/Wikipedia_talk:Article_IncubatorDisallow: /wiki/Wikipedia_talk%3AArticle_Incubator#Disallow: /wiki/Category:Noindexed_pagesDisallow: /wiki/Category%3ANoindexed_pages## User sandboxes for modules and Template Styles are placed in these subpages for testing#Disallow: /wiki/Module:SandboxDisallow: /wiki/Module%3ASandboxDisallow: /wiki/Template:TemplateStyles_sandboxDisallow: /wiki/Template%3ATemplateStyles_sandbox# Adminship and arbitration discussions are blocked from search engines, so blocking newer adminship processes (as of 2024) as wellDisallow: /wiki/Wikipedia:Administrator_recallDisallow: /wiki/Wikipedia%3AAdministrator_recallDisallow: /wiki/Wikipedia_talk:Administrator_recallDisallow: /wiki/Wikipedia_talk%3AAdministrator_recallDisallow: /wiki/Wikipedia:Administrator_electionsDisallow: /wiki/Wikipedia%3AAdministrator_electionsDisallow: /wiki/Wikipedia_talk:Administrator_electionsDisallow: /wiki/Wikipedia_talk%3AAdministrator_elections
Notes about AI written in the file: "Don't add newlines here. All rules set here are active for every user-agent."
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://en.wikipedia.org/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Page tags | <meta name="robots" content="max-image-preview:standard"> |
| Anti-bot software | No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: Article, Organization, ImageObject. Readable text is present in the HTML.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Terms_of_Use (ToS hosted on another domain; not fetched in v0.1) |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
History starts with our first check
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
One version on record so far (19 Oct 2026).
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.