{"domain":"quora.com","name":"Quora","category":"social","subcategory":"forum","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 9 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Applebot-Extended, OAI-SearchBot and Claude-SearchBot. Its catch-all rule keeps 13 other bots out of parts of it. Google-Extended can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.515Z","last_checked":"2026-10-19T13:36:45.158Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.quora.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.quora.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.quora.com/robots.txt","hops":[{"from":"https://quora.com/robots.txt","to":"https://www.quora.com/robots.txt","status":308}],"blob":"a3755528cbd265c9d927ad5b486f6a4eda99a1c642daf60929b2dfbcceb7db59","text":"# NOTICE: All crawlers and bots, regardless of whether or not they are\n# specified below, are strictly prohibited from using Quora platform content \n# for the purposes of training AI models or similar machine learning systems, \n# except where explicit prior permission has been granted by Quora through a\n# contractual licensing agreement. Any use of Quora platform content for such \n# purposes without our explicit contractual permission is a violation of our \n# terms of service and intellectual property rights.\n\n# If you operate a search engine and would like to crawl Quora,\n# you can email robotstxt@quora.com. Thanks.\n\nUser-agent: AASA-Bot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Amazonbot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Googlebot\nAllow: */answer/ix_/\nAllow: /sitemap/collections/answer/\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\nDisallow: */unanswered/*\nDisallow: */answer/\nDisallow: */answers/\n\nUser-agent: Google-Extended\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Facebot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Applebot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Twitterbot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: LinkedInBot\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Feedly\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\nUser-agent: Pingdom\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n\n# exclusive for AI training\n\nUser-agent: Applebot-Extended\nDisallow: /\n\n# another archiving service\n\nUser-agent: Arquivo-web-crawler\nDisallow: /\n\n\nUser-agent: Bytespider\nDisallow: /\n\n\nUser-agent: Diffbot\nDisallow: /\n\n\nUser-agent: GPTBot\nDisallow: /\n\n# People share a lot of sensitive material on Quora - controversial political\n# views, workplace gossip and compensation, and negative opinions held of\n# companies. Over many years, as they change jobs or change their views, it is\n# important that they can delete or anonymize their previously-written answers.\n# \n# We opt out of the wayback machine because inclusion would allow people to\n# discover the identity of authors who had written sensitive answers publicly and\n# later had made them anonymous, and because it would prevent authors from being\n# able to remove their content from the internet if they change their mind about\n# publishing it. As far as we can tell, there is no way for sites to selectively\n# programmatically remove content from the archive and so this is the only way\n# for us to protect writers. If they open up an API where we can remove content\n# from the archive when authors remove it from Quora, but leave the rest of the\n# content archived, we would be happy to opt back in. See the page here:\n# \n# https://archive.org/about/exclude.php\n# \n# Meanwhile, if you are looking for an older version of any content on Quora, we\n# have full edit history tracked and accessible in product (with the exception of\n# content that has been removed by the author). You can generally access this by\n# clicking on timestamps, or by appending \"/log\" to the URL of any content page.\n# \n# For any questions or feedback about this please visit our contact page\n# https://help.quora.com/hc/en-us/requests/new\n\nUser-agent: ia_archiver\nDisallow: /\n\n\nUser-agent: ImagesiftBot\nDisallow: /\n\n# Blocked since a lot of bad requests were made by this crawler.\n\nUser-agent: JamesBOT\nDisallow: /\n\n\nUser-agent: OAI-SearchBot\nDisallow: /\n# drinking water\nAllow: /*drinking-water\nAllow: /*Drinking-water\nAllow: /*Drinking-Water\nAllow: /*drinking-Water\n\n\nUser-agent: PerplexityBot\nDisallow: /\n\n\nUser-agent: PetalBot\nDisallow: /\n\n\nUser-agent: Scrapy\nDisallow: /\n\n\nUser-agent: ChatGPT-User\nDisallow: /\n\n\nUser-agent: Bravebot\nDisallow: /\n\n\nUser-agent: CriteoBot/0.1\nDisallow: /\n\n\nUser-agent: ClaudeBot\nDisallow: /\n\n\nUser-agent: Claude-User\nDisallow: /\n\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: *\nAllow: /$\nAllow: /about$\nAllow: /about/\nAllow: /challenges$\nAllow: /press$\nAllow: /login/\nAllow: /signup$\nDisallow: /\nDisallow: /ajax/\nDisallow: /*_POST$\nDisallow: /*_POST/\nDisallow: /@async\nDisallow: /*/@async\nDisallow: /log/\nDisallow: /*/log\nDisallow: /*/about\nDisallow: /*/action\nDisallow: /*/activity\nDisallow: /*/all_questions\nDisallow: /*/all_posts$\nDisallow: /*/all_posts/\nDisallow: /*/blogs$\nDisallow: /*/blogs/\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/link/\nDisallow: /*/manage\nDisallow: /*/mentions\nDisallow: /*/merged\nDisallow: /*/open_questions\nDisallow: /*/posts$\nDisallow: /*/posts/\nDisallow: /*/questions\nDisallow: /*/related\nDisallow: /*/reviews$\nDisallow: /*/reviews/\nDisallow: /*/share\nDisallow: /*/top_questions\nDisallow: /*/topic-bio\nDisallow: /*/topic_bio\nDisallow: /*/topics\nDisallow: /*/comment\nDisallow: /comment/\nDisallow: /*/comments\nDisallow: /*/all_comments\nDisallow: /*/answer_comments\nDisallow: /*/mobile_collapsed\nDisallow: /*/mobile_expanded_voter_list\nDisallow: /home/global_feed\nDisallow: /search?q=\nDisallow: /search/?q=\nDisallow: /digest/\nDisallow: /email_optout/\nDisallow: /qemail/\nDisallow: /invite/\nDisallow: /widgets/content_iframe/\nDisallow: /widgets/content_js/\nDisallow: /_/\nDisallow: /webnode2/\nDisallow: /anonymous/\nDisallow: /q/*/admin_log\nDisallow: /q/*/stats\nDisallow: /q/*/settings\nDisallow: /q/*/queue\nDisallow: /q/*/suggestions\nDisallow: /q/*/submissions\nDisallow: /q/*/quality\nDisallow: /q/*/earnings\nDisallow: /profile/*/Rss\nDisallow: /topic/*/Rss\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":3,"text":"for the purposes of training AI models or similar machine learning systems,"},{"line":657,"text":"exclusive for AI training"}],"platform":null,"wildcard":{"exists":true,"access":"restricted"},"named":["AASA-Bot","Amazonbot","Googlebot","Google-Extended","Facebot","Applebot","Twitterbot","LinkedInBot","Feedly","Pingdom","Applebot-Extended","Arquivo-web-crawler","Bytespider","Diffbot","GPTBot","ia_archiver","ImagesiftBot","JamesBOT","OAI-SearchBot","PerplexityBot","PetalBot","Scrapy","ChatGPT-User","Bravebot","CriteoBot/0.1","ClaudeBot","Claude-User","Claude-SearchBot","*"],"lines":832},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":677},"any":{"allowed":false,"line":677},"search":{"allowed":false,"line":677},"cart":{"allowed":false,"line":677},"checkout":{"allowed":false,"line":677},"account":{"allowed":false,"line":677}},"disallows":1,"lines":[676,677],"groupRanges":[[676,677]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":751},"any":{"allowed":false,"line":751},"search":{"allowed":false,"line":751},"cart":{"allowed":false,"line":751},"checkout":{"allowed":false,"line":751},"account":{"allowed":false,"line":751}},"disallows":1,"lines":[750,751],"groupRanges":[[750,751]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":62,"lines":[209,271],"groupRanges":[[209,271]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":660},"any":{"allowed":false,"line":660},"search":{"allowed":false,"line":660},"cart":{"allowed":false,"line":660},"checkout":{"allowed":false,"line":660},"account":{"allowed":false,"line":660}},"disallows":1,"lines":[659,660],"groupRanges":[[659,660]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":669},"any":{"allowed":false,"line":669},"search":{"allowed":false,"line":669},"cart":{"allowed":false,"line":669},"checkout":{"allowed":false,"line":669},"account":{"allowed":false,"line":669}},"disallows":1,"lines":[668,669],"groupRanges":[[668,669]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":718},"any":{"allowed":false,"line":718},"search":{"allowed":false,"line":718},"cart":{"allowed":false,"line":718},"checkout":{"allowed":false,"line":718},"account":{"allowed":false,"line":718}},"disallows":1,"lines":[717,723],"groupRanges":[[717,723]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":759},"any":{"allowed":false,"line":759},"search":{"allowed":false,"line":759},"cart":{"allowed":false,"line":759},"checkout":{"allowed":false,"line":759},"account":{"allowed":false,"line":759}},"disallows":1,"lines":[758,759],"groupRanges":[[758,759]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":727},"any":{"allowed":false,"line":727},"search":{"allowed":false,"line":727},"cart":{"allowed":false,"line":727},"checkout":{"allowed":false,"line":727},"account":{"allowed":false,"line":727}},"disallows":1,"lines":[726,727],"groupRanges":[[726,727]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":62,"lines":[76,138],"groupRanges":[[76,138]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":739},"any":{"allowed":false,"line":739},"search":{"allowed":false,"line":739},"cart":{"allowed":false,"line":739},"checkout":{"allowed":false,"line":739},"account":{"allowed":false,"line":739}},"disallows":1,"lines":[738,739],"groupRanges":[[738,739]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":755},"any":{"allowed":false,"line":755},"search":{"allowed":false,"line":755},"cart":{"allowed":false,"line":755},"checkout":{"allowed":false,"line":755},"account":{"allowed":false,"line":755}},"disallows":1,"lines":[754,755],"groupRanges":[[754,755]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":65,"lines":[140,207],"groupRanges":[[140,207]]},{"token":"Bingbot","matched":"wildcard","access":"restricted","probes":{"root":{"allowed":true,"line":762},"any":{"allowed":false,"line":769},"search":{"allowed":false,"line":769},"cart":{"allowed":false,"line":769},"checkout":{"allowed":false,"line":769},"account":{"allowed":false,"line":769}},"disallows":63,"lines":[761,831],"groupRanges":[[761,831]]}],"lanes":{"training":{"level":"mixed","none":4,"full":1,"restricted":4,"n":9},"search":{"level":"mixed","none":3,"full":1,"restricted":4,"n":8},"user_directed":{"level":"mixed","none":2,"full":0,"restricted":5,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":1,"first":"2026-10-19","archive_points":0,"series":[{"m":"2023-01","s":null},{"m":"2023-02","s":null},{"m":"2023-03","s":null},{"m":"2023-04","s":null},{"m":"2023-05","s":null},{"m":"2023-06","s":null},{"m":"2023-07","s":null},{"m":"2023-08","s":null},{"m":"2023-09","s":null},{"m":"2023-10","s":null},{"m":"2023-11","s":null},{"m":"2023-12","s":null},{"m":"2024-01","s":null},{"m":"2024-02","s":null},{"m":"2024-03","s":null},{"m":"2024-04","s":null},{"m":"2024-05","s":null},{"m":"2024-06","s":null},{"m":"2024-07","s":null},{"m":"2024-08","s":null},{"m":"2024-09","s":null},{"m":"2024-10","s":null},{"m":"2024-11","s":null},{"m":"2024-12","s":null},{"m":"2025-01","s":null},{"m":"2025-02","s":null},{"m":"2025-03","s":null},{"m":"2025-04","s":null},{"m":"2025-05","s":null},{"m":"2025-06","s":null},{"m":"2025-07","s":null},{"m":"2025-08","s":null},{"m":"2025-09","s":null},{"m":"2025-10","s":null},{"m":"2025-11","s":null},{"m":"2025-12","s":null},{"m":"2026-01","s":null},{"m":"2026-02","s":null},{"m":"2026-03","s":null},{"m":"2026-04","s":null},{"m":"2026-05","s":null},{"m":"2026-06","s":null},{"m":"2026-07","s":null},{"m":"2026-08","s":null},{"m":"2026-09","s":null},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9}]},"events":[]}