{"domain":"wikipedia.org","name":"Wikipedia","category":"reference","subcategory":"encyclopedia","country":"US","tags":[],"notes":"","status":"allowed","status_label":"Allows AI","rationale":"Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (<code>User-agent: *</code>), which lets them into the site.","computed_at":"2026-10-20T03:43:15.629Z","last_checked":"2026-10-19T13:39:46.816Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://en.wikipedia.org"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://en.wikipedia.org/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://en.wikipedia.org/robots.txt","hops":[{"from":"https://wikipedia.org/robots.txt","to":"https://www.wikipedia.org/robots.txt","status":301},{"from":"https://www.wikipedia.org/robots.txt","to":"https://en.wikipedia.org/robots.txt","status":301}],"blob":"48b98d80c8444d37cdf50cc9a4d2257b66039477e581066d2356062a4cbd678f","text":"# robots.txt for http://www.wikipedia.org/ and friends\n#\n# Please note: There are a lot of pages on this site, and there are\n# some misbehaved spiders out there that go _way_ too fast. If you're\n# irresponsible, your access to the site may be blocked.\n#\n\n# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN\n# and ignoring 429 ratelimit responses, claims to respect robots:\n# http://mj12bot.com/\nUser-agent: MJ12bot\nDisallow: /\n\n# advertising-related bots:\nUser-agent: Mediapartners-Google*\nDisallow: /\n\n# Wikipedia work bots:\nUser-agent: IsraBot\nDisallow:\n\nUser-agent: Orthogaffe\nDisallow:\n\n# Crawlers that are kind enough to obey, but which we'd rather not have\n# unless they're feeding search engines.\nUser-agent: UbiCrawler\nDisallow: /\n\nUser-agent: DOC\nDisallow: /\n\nUser-agent: Zao\nDisallow: /\n\n# Some bots are known to be trouble, particularly those designed to copy\n# entire sites. Please obey robots.txt.\nUser-agent: sitecheck.internetseer.com\nDisallow: /\n\nUser-agent: Zealbot\nDisallow: /\n\nUser-agent: MSIECrawler\nDisallow: /\n\nUser-agent: SiteSnagger\nDisallow: /\n\nUser-agent: WebStripper\nDisallow: /\n\nUser-agent: WebCopier\nDisallow: /\n\nUser-agent: Fetch\nDisallow: /\n\nUser-agent: Offline Explorer\nDisallow: /\n\nUser-agent: Teleport\nDisallow: /\n\nUser-agent: TeleportPro\nDisallow: /\n\nUser-agent: WebZIP\nDisallow: /\n\nUser-agent: linko\nDisallow: /\n\nUser-agent: HTTrack\nDisallow: /\n\nUser-agent: Microsoft.URL.Control\nDisallow: /\n\nUser-agent: Xenu\nDisallow: /\n\nUser-agent: larbin\nDisallow: /\n\nUser-agent: libwww\nDisallow: /\n\nUser-agent: ZyBORG\nDisallow: /\n\nUser-agent: Download Ninja\nDisallow: /\n\n# Misbehaving, requests much too fast\nUser-agent: fast\nDisallow: /\n\n# Sorry, wget in its recursive mode is a frequent problem.\n# Please read the man page and use it properly; there is a\n# --wait option you can use to set the delay between hits,\n# for instance.\nUser-agent: wget\nDisallow: /\n\n# The 'grub' distributed client has been *very* poorly behaved.\nUser-agent: grub-client\nDisallow: /\n\n# Doesn't follow robots.txt anyway, but...\nUser-agent: k2spider\nDisallow: /\n\n# Hits many times per second, not acceptable\n# http://www.nameprotect.com/botinfo.html\nUser-agent: NPBot\nDisallow: /\n\n# A capture bot, downloads gazillions of pages with no public benefit\n# http://www.webreaper.net/\nUser-agent: WebReaper\nDisallow: /\n\n# Per their statement, semrushbot respects crawl-delay directives\n# We want them to overall stay within reasonable request rates to\n# the backend (20 rps); keeping in mind that the crawl-delay will\n# be applied by site and not globally by the bot, 5 seconds seem\n# like a reasonable approximation\nUser-agent: SemrushBot\nCrawl-delay: 5\n\n#\n# Friendly, low-speed bots are welcome viewing article pages, but not\n# dynamically-generated pages please.\n#\n# Inktomi's \"Slurp\" can read a minimum delay between hits; if your\n# bot supports such a thing using the 'Crawl-delay' or another\n# instruction, please let us know.\n#\n# There is a special exception for API mobileview to allow dynamic\n# mobile web & app views to load section content.\n# These views aren't HTTP-cached but use parser cache aggressively\n# and don't expose Special-pages and other uncached content.\n#\n# Another exception is for REST API documentation, located at\n# /api/rest_v1/?doc.\n#\nUser-agent: *\nAllow: /w/api.php?action=mobileview&\nAllow: /w/load.php?\nAllow: /api/rest_v1/?doc\nAllow: /w/rest.php/site/v1/sitemap\nDisallow: /w/\nDisallow: /api/\nDisallow: /trap/\nDisallow: /wiki/Special:\nDisallow: /wiki/Spezial:\nDisallow: /wiki/Spesial:\nDisallow: /wiki/Special%3A\nDisallow: /wiki/Spezial%3A\nDisallow: /wiki/Spesial%3A\n#\n# ar:\nDisallow: /wiki/%D8%AE%D8%A7%D8%B5:Search\nDisallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch\n#\n# dewiki:\n# T6937\n# sensible deletion and meta user discussion pages:\nDisallow: /wiki/Wikipedia:L%C3%B6schkandidaten/\nDisallow: /wiki/Wikipedia:Löschkandidaten/\nDisallow: /wiki/Wikipedia:Vandalensperrung/\nDisallow: /wiki/Wikipedia:Benutzersperrung/\nDisallow: /wiki/Wikipedia:Vermittlungsausschuss/\nDisallow: /wiki/Wikipedia:Administratoren/Probleme/\nDisallow: /wiki/Wikipedia:Adminkandidaturen/\nDisallow: /wiki/Wikipedia:Qualitätssicherung/\nDisallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/\n# 4937#5\nDisallow: /wiki/Wikipedia:Vandalismusmeldung/\nDisallow: /wiki/Wikipedia:Gesperrte_Lemmata/\nDisallow: /wiki/Wikipedia:Löschprüfung/\nDisallow: /wiki/Wikipedia:L%C3%B6schprüfung/\nDisallow: /wiki/Wikipedia:Administratoren/Notizen/\nDisallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/\nDisallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/\n# T14111\nDisallow: /wiki/Wikipedia:Checkuser/\nDisallow: /wiki/Wikipedia_Diskussion:Checkuser/\nDisallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/\n# T15961\nDisallow: /wiki/Wikipedia:Spam-Blacklist-Log\nDisallow: /wiki/Wikipedia%3ASpam-Blacklist-Log\nDisallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log\nDisallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log\n#\n# enwiki:\n# Folks get annoyed when VfD discussions end up the number 1 google hit for\n# their name. See T6776\nDisallow: /wiki/Wikipedia:Articles_for_deletion/\nDisallow: /wiki/Wikipedia%3AArticles_for_deletion/\nDisallow: /wiki/Wikipedia:Votes_for_deletion/\nDisallow: /wiki/Wikipedia%3AVotes_for_deletion/\nDisallow: /wiki/Wikipedia:Pages_for_deletion/\nDisallow: /wiki/Wikipedia%3APages_for_deletion/\nDisallow: /wiki/Wikipedia:Miscellany_for_deletion/\nDisallow: /wiki/Wikipedia%3AMiscellany_for_deletion/\nDisallow: /wiki/Wikipedia:Miscellaneous_deletion/\nDisallow: /wiki/Wikipedia%3AMiscellaneous_deletion/\nDisallow: /wiki/Wikipedia:Copyright_problems\nDisallow: /wiki/Wikipedia%3ACopyright_problems\nDisallow: /wiki/Wikipedia:Protected_titles/\nDisallow: /wiki/Wikipedia%3AProtected_titles/\n# T15398\nDisallow: /wiki/Wikipedia:WikiProject_Spam/\nDisallow: /wiki/Wikipedia%3AWikiProject_Spam/\n# T16075\nDisallow: /wiki/MediaWiki:Spam-blacklist\nDisallow: /wiki/MediaWiki%3ASpam-blacklist\nDisallow: /wiki/MediaWiki_talk:Spam-blacklist\nDisallow: /wiki/MediaWiki_talk%3ASpam-blacklist\n# T13261\nDisallow: /wiki/Wikipedia:Requests_for_arbitration/\nDisallow: /wiki/Wikipedia%3ARequests_for_arbitration/\nDisallow: /wiki/Wikipedia:Requests_for_comment/\nDisallow: /wiki/Wikipedia%3ARequests_for_comment/\nDisallow: /wiki/Wikipedia:Requests_for_adminship/\nDisallow: /wiki/Wikipedia%3ARequests_for_adminship/\n# T12288\nDisallow: /wiki/Wikipedia_talk:Articles_for_deletion/\nDisallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/\nDisallow: /wiki/Wikipedia_talk:Votes_for_deletion/\nDisallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/\nDisallow: /wiki/Wikipedia_talk:Pages_for_deletion/\nDisallow: /wiki/Wikipedia_talk%3APages_for_deletion/\nDisallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/\nDisallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/\nDisallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/\nDisallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/\n# T16793\nDisallow: /wiki/Wikipedia:Changing_username\nDisallow: /wiki/Wikipedia%3AChanging_username\nDisallow: /wiki/Wikipedia:Changing_username/\nDisallow: /wiki/Wikipedia%3AChanging_username/\nDisallow: /wiki/Wikipedia_talk:Changing_username\nDisallow: /wiki/Wikipedia_talk%3AChanging_username\nDisallow: /wiki/Wikipedia_talk:Changing_username/\nDisallow: /wiki/Wikipedia_talk%3AChanging_username/\n#\n# eswiki:\n# T8746\nDisallow: /wiki/Wikipedia:Consultas_de_borrado/\nDisallow: /wiki/Wikipedia%3AConsultas_de_borrado/\n#\n# fiwiki:\n# T10695\nDisallow: /wiki/Wikipedia:Poistettavat_sivut\nDisallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:\nDisallow: /wiki/Käyttäjä:\nDisallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:\nDisallow: /wiki/Keskustelu_käyttäjästä:\nDisallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/\nDisallow: /wiki/Wikipedia:Ylläpitäjät/\n#\n# hewiki:\nDisallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search\nDisallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch\n#T11517\nDisallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/\nDisallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/\nDisallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים\nDisallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\nDisallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים\nDisallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\nDisallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\n#\n# huwiki:\nDisallow: /wiki/Speci%C3%A1lis:Search\nDisallow: /wiki/Speci%C3%A1lis%3ASearch\n#\n# itwiki:\n# T7545\nDisallow: /wiki/Wikipedia:Pagine_da_cancellare\nDisallow: /wiki/Wikipedia%3APagine_da_cancellare\nDisallow: /wiki/Wikipedia:Utenti_problematici\nDisallow: /wiki/Wikipedia%3AUtenti_problematici\nDisallow: /wiki/Wikipedia:Vandalismi_in_corso\nDisallow: /wiki/Wikipedia%3AVandalismi_in_corso\nDisallow: /wiki/Wikipedia:Amministratori\nDisallow: /wiki/Wikipedia%3AAmministratori\nDisallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata\nDisallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata\nDisallow: /wiki/Categoria:Da_cancellare_subito\nDisallow: /wiki/Categoria%3ADa_cancellare_subito\nDisallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright\nDisallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright\nDisallow: /wiki/Categoria:Da_controllare_per_copyright\nDisallow: /wiki/Categoria%3ADa_controllare_per_copyright\nDisallow: /wiki/Progetto:Rimozione_contributi_sospetti\nDisallow: /wiki/Progetto%3ARimozione_contributi_sospetti\nDisallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright\nDisallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright\nDisallow: /wiki/Progetto:Cococo\nDisallow: /wiki/Progetto%3ACococo\nDisallow: /wiki/Discussioni_progetto:Cococo\nDisallow: /wiki/Discussioni_progetto%3ACococo\n#\n# jawiki\nDisallow: /wiki/%E7%89%B9%E5%88%A5:Search\nDisallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch\n# T7239\nDisallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/\nDisallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/\nDisallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC\nDisallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC\n# nowiki\n# T13432\nDisallow: /wiki/Bruker:\nDisallow: /wiki/Bruker%3A\nDisallow: /wiki/Brukerdiskusjon\nDisallow: /wiki/Wikipedia:Administratorer\nDisallow: /wiki/Wikipedia%3AAdministratorer\nDisallow: /wiki/Wikipedia-diskusjon:Administratorer\nDisallow: /wiki/Wikipedia-diskusjon%3AAdministratorer\nDisallow: /wiki/Wikipedia:Sletting\nDisallow: /wiki/Wikipedia%3ASletting\nDisallow: /wiki/Wikipedia-diskusjon:Sletting\nDisallow: /wiki/Wikipedia-diskusjon%3ASletting\n#\n# plwiki\n# T10067\nDisallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia\nDisallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia\nDisallow: /wiki/Wikipedia:Do_usuni%C4%99cia\nDisallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia\nDisallow: /wiki/Wikipedia:SDU/\nDisallow: /wiki/Wikipedia%3ASDU/\nDisallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich\nDisallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich\n#\n# ptwiki:\n# T7394\nDisallow: /wiki/Wikipedia:Páginas_para_eliminar/\nDisallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/\nDisallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/\nDisallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/\nDisallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/\nDisallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/\n#\n# rowiki:\n# T14546\nDisallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters\nDisallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters\nDisallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters\nDisallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters\n#\n# ruwiki:\nDisallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search\nDisallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch\n#\n# svwiki:\n# T12229\nDisallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering\nDisallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering\nDisallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering\nDisallow: /wiki/Användare\nDisallow: /wiki/Anv%C3%A4ndare\nDisallow: /wiki/Användardiskussion\nDisallow: /wiki/Anv%C3%A4ndardiskussion\nDisallow: /wiki/Wikipedia:Skyddade_sidnamn\nDisallow: /wiki/Wikipedia%3ASkyddade_sidnamn\n# T13291\nDisallow: /wiki/Wikipedia:Sidor_som_bör_raderas\nDisallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas\nDisallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas\n#\n# zhwiki:\n# T7104\nDisallow: /wiki/Wikipedia:删除投票/侵权\nDisallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83\nDisallow: /wiki/Wikipedia:删除投票和请求\nDisallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82\nDisallow: /wiki/Category:快速删除候选\nDisallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89\nDisallow: /wiki/Category:维基百科需要翻译的文章\nDisallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0\n#\n# sister projects\n#\n# enwikinews:\n# T7340\nDisallow: /wiki/Portal:Prepared_stories/\nDisallow: /wiki/Portal%3APrepared_stories/\n#\n# itwikinews\n# T11138\nDisallow: /wiki/Wikinotizie:Richieste_di_cancellazione\nDisallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright\nDisallow: /wiki/Categoria:Da_cancellare_subito\nDisallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright\nDisallow: /wiki/Wikinotizie:Storie_in_preparazione\n#\n# enwikiquote:\n# T17095\nDisallow: /wiki/Wikiquote:Votes_for_deletion/\nDisallow: /wiki/Wikiquote%3AVotes_for_deletion/\nDisallow: /wiki/Wikiquote_talk:Votes_for_deletion/\nDisallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/\nDisallow: /wiki/Wikiquote:Votes_for_deletion_archive/\nDisallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/\nDisallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/\nDisallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/\n#\n# enwikibooks\nDisallow: /wiki/Wikibooks:Votes_for_deletion\n#\n\n\nSitemap: https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0\n\n#\n#\n#----------------------------------------------------------#\n#\n#\n#\n#\n# Localisable part of robots.txt for en.wikipedia.org\n#\n# Edit at https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit\n# Don't add newlines here. All rules set here are active for every user-agent.\n#\n# Please check any changes using a syntax validator\n# Enter https://en.wikipedia.org/robots.txt as the URL to check.\n#\n# https://phabricator.wikimedia.org/T16075\nDisallow: /wiki/MediaWiki:Spam-blacklist\nDisallow: /wiki/MediaWiki%3ASpam-blacklist\nDisallow: /wiki/MediaWiki_talk:Spam-blacklist\nDisallow: /wiki/MediaWiki_talk%3ASpam-blacklist\nDisallow: /wiki/Wikipedia:WikiProject_Spam\nDisallow: /wiki/Wikipedia_talk:WikiProject_Spam\n#\n# Folks get annoyed when XfD discussions end up the number 1 google hit for\n# their name. \n# https://phabricator.wikimedia.org/T16075\nDisallow: /wiki/Wikipedia:Articles_for_deletion\nDisallow: /wiki/Wikipedia%3AArticles_for_deletion\nDisallow: /wiki/Wikipedia:Votes_for_deletion\nDisallow: /wiki/Wikipedia%3AVotes_for_deletion\nDisallow: /wiki/Wikipedia:Pages_for_deletion\nDisallow: /wiki/Wikipedia%3APages_for_deletion\nDisallow: /wiki/Wikipedia:Miscellany_for_deletion\nDisallow: /wiki/Wikipedia%3AMiscellany_for_deletion\nDisallow: /wiki/Wikipedia:Miscellaneous_deletion\nDisallow: /wiki/Wikipedia%3AMiscellaneous_deletion\nDisallow: /wiki/Wikipedia:Categories_for_discussion\nDisallow: /wiki/Wikipedia%3ACategories_for_discussion\nDisallow: /wiki/Wikipedia:Templates_for_deletion\nDisallow: /wiki/Wikipedia%3ATemplates_for_deletion\nDisallow: /wiki/Wikipedia:Redirects_for_discussion\nDisallow: /wiki/Wikipedia%3ARedirects_for_discussion\nDisallow: /wiki/Wikipedia:Deletion_review\nDisallow: /wiki/Wikipedia%3ADeletion_review\nDisallow: /wiki/Wikipedia:WikiProject_Deletion_sorting\nDisallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting\nDisallow: /wiki/Wikipedia:Files_for_deletion\nDisallow: /wiki/Wikipedia%3AFiles_for_deletion\nDisallow: /wiki/Wikipedia:Files_for_discussion\nDisallow: /wiki/Wikipedia%3AFiles_for_discussion\nDisallow: /wiki/Wikipedia:Possibly_unfree_files\nDisallow: /wiki/Wikipedia%3APossibly_unfree_files\n#\n# https://phabricator.wikimedia.org/T12288\nDisallow: /wiki/Wikipedia_talk:Articles_for_deletion\nDisallow: /wiki/Wikipedia_talk%3AArticles_for_deletion\nDisallow: /wiki/Wikipedia_talk:Votes_for_deletion\nDisallow: /wiki/Wikipedia_talk%3AVotes_for_deletion\nDisallow: /wiki/Wikipedia_talk:Pages_for_deletion\nDisallow: /wiki/Wikipedia_talk%3APages_for_deletion\nDisallow: /wiki/Wikipedia_talk:Miscellany_for_deletion\nDisallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion\nDisallow: /wiki/Wikipedia_talk:Miscellaneous_deletion\nDisallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion\nDisallow: /wiki/Wikipedia_talk:Templates_for_deletion\nDisallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion\nDisallow: /wiki/Wikipedia_talk:Categories_for_discussion\nDisallow: /wiki/Wikipedia_talk%3ACategories_for_discussion\nDisallow: /wiki/Wikipedia_talk:Deletion_review\nDisallow: /wiki/Wikipedia_talk%3ADeletion_review\nDisallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting\nDisallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting\nDisallow: /wiki/Wikipedia_talk:Files_for_deletion\nDisallow: /wiki/Wikipedia_talk%3AFiles_for_deletion\nDisallow: /wiki/Wikipedia_talk:Files_for_discussion\nDisallow: /wiki/Wikipedia_talk%3AFiles_for_discussion\nDisallow: /wiki/Wikipedia_talk:Possibly_unfree_files\nDisallow: /wiki/Wikipedia_talk%3APossibly_unfree_files\n#\nDisallow: /wiki/Wikipedia:Copyright_problems\nDisallow: /wiki/Wikipedia%3ACopyright_problems\nDisallow: /wiki/Wikipedia_talk:Copyright_problems\nDisallow: /wiki/Wikipedia_talk%3ACopyright_problems\nDisallow: /wiki/Wikipedia:Suspected_copyright_violations\nDisallow: /wiki/Wikipedia%3ASuspected_copyright_violations\nDisallow: /wiki/Wikipedia_talk:Suspected_copyright_violations\nDisallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations\nDisallow: /wiki/Wikipedia:Contributor_copyright_investigations\nDisallow: /wiki/Wikipedia%3AContributor_copyright_investigations\nDisallow: /wiki/Wikipedia:Contributor_copyright_investigations\nDisallow: /wiki/Wikipedia%3AContributor_copyright_investigations\nDisallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations\nDisallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations\nDisallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations\nDisallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations\nDisallow: /wiki/Wikipedia:Protected_titles\nDisallow: /wiki/Wikipedia%3AProtected_titles\nDisallow: /wiki/Wikipedia_talk:Protected_titles\nDisallow: /wiki/Wikipedia_talk%3AProtected_titles\nDisallow: /wiki/Wikipedia:Articles_for_creation\nDisallow: /wiki/Wikipedia%3AArticles_for_creation\nDisallow: /wiki/Wikipedia_talk:Articles_for_creation\nDisallow: /wiki/Wikipedia_talk%3AArticles_for_creation\nDisallow: /wiki/Wikipedia_talk:Article_wizard\nDisallow: /wiki/Wikipedia_talk%3AArticle_wizard\n#\n# https://phabricator.wikimedia.org/T13261\nDisallow: /wiki/Wikipedia:Requests_for_arbitration\nDisallow: /wiki/Wikipedia%3ARequests_for_arbitration\nDisallow: /wiki/Wikipedia_talk:Requests_for_arbitration\nDisallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration\nDisallow: /wiki/Wikipedia:Requests_for_comment\nDisallow: /wiki/Wikipedia%3ARequests_for_comment\nDisallow: /wiki/Wikipedia_talk:Requests_for_comment\nDisallow: /wiki/Wikipedia_talk%3ARequests_for_comment\nDisallow: /wiki/Wikipedia:Requests_for_adminship\nDisallow: /wiki/Wikipedia%3ARequests_for_adminship\nDisallow: /wiki/Wikipedia_talk:Requests_for_adminship\nDisallow: /wiki/Wikipedia_talk%3ARequests_for_adminship\n#\n# https://phabricator.wikimedia.org/T14111\nDisallow: /wiki/Wikipedia:Requests_for_checkuser\nDisallow: /wiki/Wikipedia%3ARequests_for_checkuser\nDisallow: /wiki/Wikipedia_talk:Requests_for_checkuser\nDisallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser\n#\n# https://phabricator.wikimedia.org/T15398\nDisallow: /wiki/Wikipedia:WikiProject_Spam\nDisallow: /wiki/Wikipedia%3AWikiProject_Spam\n#\n# https://phabricator.wikimedia.org/T16793\nDisallow: /wiki/Wikipedia:Changing_username\nDisallow: /wiki/Wikipedia%3AChanging_username\nDisallow: /wiki/Wikipedia:Changing_username\nDisallow: /wiki/Wikipedia%3AChanging_username\nDisallow: /wiki/Wikipedia_talk:Changing_username\nDisallow: /wiki/Wikipedia_talk%3AChanging_username\nDisallow: /wiki/Wikipedia_talk:Changing_username\nDisallow: /wiki/Wikipedia_talk%3AChanging_username\n#\nDisallow: /wiki/Wikipedia:Administrators%27_noticeboard\nDisallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard\nDisallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard\nDisallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard\nDisallow: /wiki/Wikipedia:Community_sanction_noticeboard\nDisallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard\nDisallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard\nDisallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard\nDisallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard\nDisallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard\nDisallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard\nDisallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard\n#\nDisallow: /wiki/Wikipedia:Sockpuppet_investigations\nDisallow: /wiki/Wikipedia%3ASockpuppet_investigations\nDisallow: /wiki/Wikipedia_talk:Sockpuppet_investigations\nDisallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations\n#\nDisallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard\nDisallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard\nDisallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard\nDisallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard\n#\nDisallow: /wiki/Wikipedia:No_original_research/noticeboard\nDisallow: /wiki/Wikipedia%3ANo_original_research/noticeboard\nDisallow: /wiki/Wikipedia_talk:No_original_research/noticeboard\nDisallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard\n#\nDisallow: /wiki/Wikipedia:Fringe_theories/Noticeboard\nDisallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard\nDisallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard\nDisallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard\n#\nDisallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard\nDisallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard\nDisallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard\nDisallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard\n#\nDisallow: /wiki/Wikipedia:Long-term_abuse\nDisallow: /wiki/Wikipedia%3ALong-term_abuse\nDisallow: /wiki/Wikipedia_talk:Long-term_abuse\nDisallow: /wiki/Wikipedia_talk%3ALong-term_abuse\nDisallow: /wiki/Wikipedia:Long_term_abuse\nDisallow: /wiki/Wikipedia%3ALong_term_abuse\nDisallow: /wiki/Wikipedia_talk:Long_term_abuse\nDisallow: /wiki/Wikipedia_talk%3ALong_term_abuse\n#\nDisallow: /wiki/Wikipedia:Wikiquette_assistance\nDisallow: /wiki/Wikipedia%3AWikiquette_assistance\n#\nDisallow: /wiki/Wikipedia:Abuse_reports\nDisallow: /wiki/Wikipedia%3AAbuse_reports\nDisallow: /wiki/Wikipedia_talk:Abuse_reports\nDisallow: /wiki/Wikipedia_talk%3AAbuse_reports\nDisallow: /wiki/Wikipedia:Abuse_response\nDisallow: /wiki/Wikipedia%3AAbuse_response\nDisallow: /wiki/Wikipedia_talk:Abuse_response\nDisallow: /wiki/Wikipedia_talk%3AAbuse_response\n#\nDisallow: /wiki/Wikipedia:Reliable_sources/Noticeboard\nDisallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard\nDisallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard\nDisallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard\n#\nDisallow: /wiki/Wikipedia:Suspected_sock_puppets\nDisallow: /wiki/Wikipedia%3ASuspected_sock_puppets\nDisallow: /wiki/Wikipedia_talk:Suspected_sock_puppets\nDisallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets\n#\nDisallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard\nDisallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard\nDisallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard\nDisallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard\nDisallow: /wiki/Wikipedia:Biographies_of_living_persons%2FNoticeboard\nDisallow: /wiki/Wikipedia%3ABiographies_of_living_persons%2FNoticeboard\nDisallow: /wiki/Wikipedia_talk:Biographies_of_living_persons%2FNoticeboard\nDisallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons%2FNoticeboard\n#\nDisallow: /wiki/Wikipedia:Content_noticeboard\nDisallow: /wiki/Wikipedia%3AContent_noticeboard\nDisallow: /wiki/Wikipedia_talk:Content_noticeboard\nDisallow: /wiki/Wikipedia_talk%3AContent_noticeboard\n#\nDisallow: /wiki/Template:Editnotices\nDisallow: /wiki/Template%3AEditnotices\n#\nDisallow: /wiki/Wikipedia:Arbitration\nDisallow: /wiki/Wikipedia%3AArbitration\nDisallow: /wiki/Wikipedia_talk:Arbitration\nDisallow: /wiki/Wikipedia_talk%3AArbitration\n#\nDisallow: /wiki/Wikipedia:Arbitration_Committee\nDisallow: /wiki/Wikipedia%3AArbitration_Committee\nDisallow: /wiki/Wikipedia_talk:Arbitration_Committee\nDisallow: /wiki/Wikipedia_talk%3AArbitration_Committee\n#\nDisallow: /wiki/Wikipedia:Arbitration_Committee_Elections\nDisallow: /wiki/Wikipedia%3AArbitration_Committee_Elections\nDisallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections\nDisallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections\n#\nDisallow: /wiki/Wikipedia:Mediation_Committee\nDisallow: /wiki/Wikipedia%3AMediation_Committee\nDisallow: /wiki/Wikipedia_talk:Mediation_Committee\nDisallow: /wiki/Wikipedia_talk%3AMediation_Committee\n#\nDisallow: /wiki/Wikipedia:Mediation_Cabal/Cases\nDisallow: /wiki/Wikipedia%3AMediation_Cabal/Cases\n#\nDisallow: /wiki/Wikipedia:Requests_for_bureaucratship\nDisallow: /wiki/Wikipedia%3ARequests_for_bureaucratship\nDisallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship\nDisallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship\n#\nDisallow: /wiki/Wikipedia:Administrator_review\nDisallow: /wiki/Wikipedia%3AAdministrator_review\nDisallow: /wiki/Wikipedia_talk:Administrator_review\nDisallow: /wiki/Wikipedia_talk%3AAdministrator_review\n#\nDisallow: /wiki/Wikipedia:Editor_review\nDisallow: /wiki/Wikipedia%3AEditor_review\nDisallow: /wiki/Wikipedia_talk:Editor_review\nDisallow: /wiki/Wikipedia_talk%3AEditor_review\n#\nDisallow: /wiki/Wikipedia:Article_Incubator\nDisallow: /wiki/Wikipedia%3AArticle_Incubator\nDisallow: /wiki/Wikipedia_talk:Article_Incubator\nDisallow: /wiki/Wikipedia_talk%3AArticle_Incubator\n#\nDisallow: /wiki/Category:Noindexed_pages\nDisallow: /wiki/Category%3ANoindexed_pages\n#\n# User sandboxes for modules and Template Styles are placed in these subpages for testing\n#\nDisallow: /wiki/Module:Sandbox\nDisallow: /wiki/Module%3ASandbox\nDisallow: /wiki/Template:TemplateStyles_sandbox\nDisallow: /wiki/Template%3ATemplateStyles_sandbox\n# Adminship and arbitration discussions are blocked from search engines, so blocking newer adminship processes (as of 2024) as well\nDisallow: /wiki/Wikipedia:Administrator_recall\nDisallow: /wiki/Wikipedia%3AAdministrator_recall\nDisallow: /wiki/Wikipedia_talk:Administrator_recall\nDisallow: /wiki/Wikipedia_talk%3AAdministrator_recall\nDisallow: /wiki/Wikipedia:Administrator_elections\nDisallow: /wiki/Wikipedia%3AAdministrator_elections\nDisallow: /wiki/Wikipedia_talk:Administrator_elections\nDisallow: /wiki/Wikipedia_talk%3AAdministrator_elections","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":436,"text":"Don't add newlines here. All rules set here are active for every user-agent."}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["MJ12bot","Mediapartners-Google*","IsraBot","Orthogaffe","UbiCrawler","DOC","Zao","sitecheck.internetseer.com","Zealbot","MSIECrawler","SiteSnagger","WebStripper","WebCopier","Fetch","Offline Explorer","Teleport","TeleportPro","WebZIP","linko","HTTrack","Microsoft.URL.Control","Xenu","larbin","libwww","ZyBORG","Download Ninja","fast","wget","grub-client","k2spider","NPBot","WebReaper","SemrushBot","*"],"lines":712},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":428,"lines":[129,712],"groupRanges":[[129,712]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://en.wikipedia.org/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://en.wikipedia.org/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://en.wikipedia.org/.well-known/tdmrep.json"},"meta":{"title":"Wikipedia, the free encyclopedia","robotsMeta":[{"name":"robots","content":"max-image-preview:standard"}],"noai":[],"noindex":false,"jsonLdTypes":["Article","Organization","ImageObject"],"platform":null,"textRatio":0.058,"llmsLink":false},"x_robots_tag":null,"tos":{"url":"https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Terms_of_Use","source":"discovered","skipped":"ToS hosted on another domain; not fetched in v0.1"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":1,"first":"2026-10-19","archive_points":0,"series":[{"m":"2023-01","s":null},{"m":"2023-02","s":null},{"m":"2023-03","s":null},{"m":"2023-04","s":null},{"m":"2023-05","s":null},{"m":"2023-06","s":null},{"m":"2023-07","s":null},{"m":"2023-08","s":null},{"m":"2023-09","s":null},{"m":"2023-10","s":null},{"m":"2023-11","s":null},{"m":"2023-12","s":null},{"m":"2024-01","s":null},{"m":"2024-02","s":null},{"m":"2024-03","s":null},{"m":"2024-04","s":null},{"m":"2024-05","s":null},{"m":"2024-06","s":null},{"m":"2024-07","s":null},{"m":"2024-08","s":null},{"m":"2024-09","s":null},{"m":"2024-10","s":null},{"m":"2024-11","s":null},{"m":"2024-12","s":null},{"m":"2025-01","s":null},{"m":"2025-02","s":null},{"m":"2025-03","s":null},{"m":"2025-04","s":null},{"m":"2025-05","s":null},{"m":"2025-06","s":null},{"m":"2025-07","s":null},{"m":"2025-08","s":null},{"m":"2025-09","s":null},{"m":"2025-10","s":null},{"m":"2025-11","s":null},{"m":"2025-12","s":null},{"m":"2026-01","s":null},{"m":"2026-02","s":null},{"m":"2026-03","s":null},{"m":"2026-04","s":null},{"m":"2026-05","s":null},{"m":"2026-06","s":null},{"m":"2026-07","s":null},{"m":"2026-08","s":null},{"m":"2026-09","s":null},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[]}