The Walled Web Newsletter

All sites / Social and forums

LinkedIn

linkedin.com
Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 22 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. OAI-SearchBot and Claude-SearchBot can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Blocked

9 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

6 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Blocked

7 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Blocked
GPTBotOpenAI Blocked Named, line 4782 Says yes
ClaudeBotAnthropic Blocked Named, line 4648 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Blocked Named, line 4639 Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Not named; the catch-all rule applies Says yes
CCBotCommon Crawl Blocked Named, line 4815 Says yes
Meta-ExternalAgentMeta Blocked Named, line 4800 Suggests yes
BytespiderByteDance Blocked Named, line 4854 No statement found
MistralAI-TrainingMistral AI Blocked Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Blocked Not named; the catch-all rule applies Does not say
anthropic-aiAnthropic (legacy) · old name Blocked Named, line 4642 Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI Allowed Named, line 4291 Says yes
Claude-SearchBotAnthropic Allowed Named, line 4657 Says yes
PerplexityBotPerplexity Blocked Named, line 4788 Suggests yes
AmazonbotAmazon Blocked Not named; the catch-all rule applies Says yes
Amzn-SearchBotAmazon Blocked Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Blocked Named, line 4794 Says yes
MistralAI-IndexMistral AI Blocked Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Blocked Not named; the catch-all rule applies Suggests yes
AI assistants · Blocked
ChatGPT-UserOpenAI Blocked Named, line 4645 Says not always
Claude-UserAnthropic Blocked Named, line 4654 Says yes
Perplexity-UserPerplexity Blocked Named, line 4791 Says it mostly ignores it
Google-AgentGoogle Blocked Not named; the catch-all rule applies Says it mostly ignores it
Meta-ExternalFetcherMeta Blocked Named, line 4803 Says not always
Amzn-UserAmazon Blocked Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Blocked Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNamed
BingbotRegular search engine, for comparisonAllowedNamed

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

ApplebotHas rulesline 133Documented by its operator
Claude-WebBlockedline 4651Not in our verified list
cohere-aiBlockedline 4779Not in our verified list
Google-CloudVertexBotBlockedline 4785Documented by its operator
DiffbotBlockedline 4806Documented by its operator
omgiliBlockedline 4824Not in our verified list
omgilibotBlockedline 4827Not in our verified list
TimpibotBlockedline 4848Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 4863 lines · 397 lines about AI bots Source
# Notice: The use of robots or other automated means to access LinkedIn without# the express permission of LinkedIn is strictly prohibited.# See https://www.linkedin.com/legal/user-agreement.# LinkedIn may, in its discretion, permit certain automated access to certain LinkedIn pages,# for the limited purpose of including content in approved publicly available search engines.# If you would like to apply for permission to crawl LinkedIn, please email whitelist-crawl@linkedin.com.# Any and all permitted crawling of LinkedIn is subject to LinkedIn's Crawling Terms and Conditions.# See http://www.linkedin.com/legal/crawling-terms. User-agent: LinkedInBotAllow: / User-agent: GooglebotDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /embed/feed/update/Disallow: /endorsementsDisallow: /feed/update/Disallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /pages-extensions/FollowCompany*Disallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out*Allow: /settings/loid-email-unsubscribe-router*Allow: /settings/loid-email-unsubscribe*Allow: /help/… 131 lines not about AI bots User-agent: ApplebotDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out*Allow: /settings/loid-email-unsubscribe-router*Allow: /settings/loid-email-unsubscribe* User-agent: BingbotDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /embed/feed/update/Disallow: /endorsementsDisallow: /feed/update/Disallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /pages-extensions/FollowCompany*Disallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out*Allow: /settings/loid-email-unsubscribe-router*Allow: /settings/loid-email-unsubscribe* User-agent: msnbotDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: SlurpDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: Googlebot-ImageDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /pages-extensions/FollowCompany*Disallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: baiduspiderDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /profinder/pro/*Disallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: BaiduspiderDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /profinder/pro/*Disallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: seznambotDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: TeomaDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: YandexDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: YetiDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/loginDisallow: /uas/oauth/*Disallow: /uas/oauth2/*Disallow: /using*Disallow: /voyager/apiDisallow: /help/*/askDisallow: /help/*/ask/*Disallow: /help/*/answersDisallow: /help/*/answers/*Disallow: /help/testingDisallow: /help/testing/*Disallow: /xdoor*Disallow: /comm/Disallow: /enterprise-jobs/Allow: /business/sales/blog*Allow: /business/learning/blog*Allow: /help/Allow: /psettings/guest-controls*Allow: /psettings/guest-email-unsubscribe*Allow: /psettings/sms-unsubscribe*Allow: /psettings/guest-controls/retargeting-opt-out* User-agent: msnbot-mediaDisallow: /addContacts*Disallow: /addressBookExport*Disallow: /ambryDisallow: /analytics/Disallow: /answers*Disallow: /authwallDisallow: /badges/profile/createDisallow: /cap/Disallow: /chat/Disallow: /checkpoint/Disallow: /companyDir*Disallow: /connections*Disallow: /csp/Disallow: /e/Disallow: /edurec*Disallow: /endorsementsDisallow: /find/Disallow: /fizzy/adminDisallow: /groups/Disallow: /groupAnswers*Disallow: /groupSharingMsg*Disallow: /inviteFromProfile*Disallow: /inviteMany*Disallow: /jobs?runSearch*Disallow: /jobs/view/externalApply/Disallow: /jobs-guest/Disallow: /api/jobPostings/jobs*Disallow: /jsearch*Disallow: /job-apply/Disallow: /learning*?auth=trueDisallow: /learning*&auth=trueDisallow: /learning/articles/Disallow: /learning/certificates/Disallow: /learning/embed/Disallow: /learning/events/Disallow: /learning/instructors?Disallow: /learning/instructors$Disallow: /learning/instructors/Disallow: /learning/login*?redirect=Disallow: /learning/login*&redirect=Disallow: /learning/me?Disallow: /learning/me$Disallow: /learning/me/Disallow: /learning/memberbinding?Disallow: /learning/memberbinding$Disallow: /learning/memberbinding/Disallow: /learning/search?Disallow: /learning/search$Disallow: /learning/search/Disallow: /learning/settings?Disallow: /learning/settings$Disallow: /learning/settings/Disallow: /lite/Disallow: /li/trackDisallow: /mbox*Disallow: /me/Disallow: /memberInvite*Disallow: /messaging/Disallow: /mob/trackingDisallow: /mwlite/Disallow: /msgToConns*Disallow: /myprofile*Disallow: /networkDisallow: /newsArticle*Disallow: /news?viewArticle*Disallow: /nus-trk*Disallow: /oauth/v2/*Disallow: /oauth2/v2/*Disallow: /organization-guest/Disallow: /osview/Disallow: /otherContactsDisallow: /people/iweReconnectActionDisallow: /platform-telemetry/Disallow: /ppl/Disallow: /profile/Disallow: /profile/viewDisallow: /psettings/Disallow: /reconnect*Disallow: /redirect*Disallow: /redir*Disallow: /requestList*Disallow: /salary-explorer/apiDisallow: /search*Disallow: /secure/Disallow: /settings/Disallow: /shareArticle*Disallow: /signature*Disallow: /slink*Disallow: /start/Disallow: /svpRecommendations*Disallow: /topic/Disallow: /title/Disallow: /uas/login… 1252 lines not about AI bots… 3363 more lines; see the source file

Everything else we checked

Front doorWe did not visit: the rules ask our bot to stay out.
llms.txtDisallowed for AgentAccessIndexBot by robots.txt
ai.txtDisallowed for AgentAccessIndexBot by robots.txt
TDMRepDisallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report)
Page tagsHomepage not read.
Anti-bot softwareServed through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIHomepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of ServiceNo Terms of Service page found.
Not foundNo Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

45 versions on record since 1 Jan 2023

OverallJan 2023: Overall blockedFeb 2023: Overall blockedMar 2023: Overall blockedApr 2023: Overall blockedMay 2023: Overall blockedJun 2023: Overall blockedJul 2023: Overall blockedAug 2023: Overall blockedSep 2023: Overall blockedOct 2023: Overall blockedNov 2023: Overall blockedDec 2023: Overall blockedJan 2024: Overall blockedFeb 2024: Overall blockedMar 2024: Overall blockedApr 2024: Overall blockedMay 2024: Overall blockedJun 2024: Overall blockedJul 2024: Overall blockedAug 2024: Overall blockedSep 2024: Overall blockedOct 2024: Overall blockedNov 2024: Overall blockedDec 2024: Overall blockedJan 2025: Overall blockedFeb 2025: Overall blockedMar 2025: Overall blockedApr 2025: Overall blockedMay 2025: Overall blockedJun 2025: Overall blockedJul 2025: Overall blockedAug 2025: Overall blockedSep 2025: Overall blockedOct 2025: Overall blockedNov 2025: Overall blockedDec 2025: Overall partialJan 2026: Overall partialFeb 2026: Overall partialMar 2026: Overall partialApr 2026: Overall partialMay 2026: Overall partialJun 2026: Overall partialJul 2026: Overall partialAug 2026: Overall partialSep 2026: Overall partialOct 2026: Overall partialTrainingJan 2023: Training Some blockedFeb 2023: Training Some blockedMar 2023: Training Some blockedApr 2023: Training Some blockedMay 2023: Training Some blockedJun 2023: Training Some blockedJul 2023: Training Some blockedAug 2023: Training Some blockedSep 2023: Training Some blockedOct 2023: Training Some blockedNov 2023: Training Some blockedDec 2023: Training Some blockedJan 2024: Training Some blockedFeb 2024: Training Some blockedMar 2024: Training Some blockedApr 2024: Training Some blockedMay 2024: Training Some blockedJun 2024: Training Some blockedJul 2024: Training Some blockedAug 2024: Training Some blockedSep 2024: Training Some blockedOct 2024: Training Some blockedNov 2024: Training Some blockedDec 2024: Training Some blockedJan 2025: Training Some blockedFeb 2025: Training Some blockedMar 2025: Training Some blockedApr 2025: Training Some blockedMay 2025: Training Some blockedJun 2025: Training Some blockedJul 2025: Training Some blockedAug 2025: Training Some blockedSep 2025: Training Some blockedOct 2025: Training Some blockedNov 2025: Training Some blockedDec 2025: Training Some blockedJan 2026: Training Some blockedFeb 2026: Training Some blockedMar 2026: Training Some blockedApr 2026: Training BlockedMay 2026: Training BlockedJun 2026: Training BlockedJul 2026: Training BlockedAug 2026: Training BlockedSep 2026: Training BlockedOct 2026: Training BlockedSearchJan 2023: Search BlockedFeb 2023: Search BlockedMar 2023: Search BlockedApr 2023: Search BlockedMay 2023: Search BlockedJun 2023: Search BlockedJul 2023: Search BlockedAug 2023: Search BlockedSep 2023: Search BlockedOct 2023: Search BlockedNov 2023: Search BlockedDec 2023: Search BlockedJan 2024: Search BlockedFeb 2024: Search BlockedMar 2024: Search BlockedApr 2024: Search BlockedMay 2024: Search BlockedJun 2024: Search BlockedJul 2024: Search BlockedAug 2024: Search BlockedSep 2024: Search BlockedOct 2024: Search BlockedNov 2024: Search BlockedDec 2024: Search BlockedJan 2025: Search BlockedFeb 2025: Search BlockedMar 2025: Search BlockedApr 2025: Search BlockedMay 2025: Search BlockedJun 2025: Search BlockedJul 2025: Search BlockedAug 2025: Search BlockedSep 2025: Search BlockedOct 2025: Search BlockedNov 2025: Search BlockedDec 2025: Search Some blockedJan 2026: Search Some blockedFeb 2026: Search Some blockedMar 2026: Search Some blockedApr 2026: Search Some blockedMay 2026: Search Some blockedJun 2026: Search Some blockedJul 2026: Search Some blockedAug 2026: Search Some blockedSep 2026: Search Some blockedOct 2026: Search Some blockedAssistantsJan 2023: Assistants BlockedFeb 2023: Assistants BlockedMar 2023: Assistants BlockedApr 2023: Assistants BlockedMay 2023: Assistants BlockedJun 2023: Assistants BlockedJul 2023: Assistants BlockedAug 2023: Assistants BlockedSep 2023: Assistants BlockedOct 2023: Assistants BlockedNov 2023: Assistants BlockedDec 2023: Assistants BlockedJan 2024: Assistants BlockedFeb 2024: Assistants BlockedMar 2024: Assistants BlockedApr 2024: Assistants BlockedMay 2024: Assistants BlockedJun 2024: Assistants BlockedJul 2024: Assistants BlockedAug 2024: Assistants BlockedSep 2024: Assistants BlockedOct 2024: Assistants BlockedNov 2024: Assistants BlockedDec 2024: Assistants BlockedJan 2025: Assistants BlockedFeb 2025: Assistants BlockedMar 2025: Assistants BlockedApr 2025: Assistants BlockedMay 2025: Assistants BlockedJun 2025: Assistants BlockedJul 2025: Assistants BlockedAug 2025: Assistants BlockedSep 2025: Assistants BlockedOct 2025: Assistants BlockedNov 2025: Assistants BlockedDec 2025: Assistants BlockedJan 2026: Assistants BlockedFeb 2026: Assistants BlockedMar 2026: Assistants BlockedApr 2026: Assistants BlockedMay 2026: Assistants BlockedJun 2026: Assistants BlockedJul 2026: Assistants BlockedAug 2026: Assistants BlockedSep 2026: Assistants BlockedOct 2026: Assistants BlockedBotsblockedJan 2023: 23 of 24 AI bots blockedFeb 2023: 23 of 24 AI bots blockedMar 2023: 23 of 24 AI bots blockedApr 2023: 23 of 24 AI bots blockedMay 2023: 23 of 24 AI bots blockedJun 2023: 23 of 24 AI bots blockedJul 2023: 23 of 24 AI bots blockedAug 2023: 23 of 24 AI bots blockedSep 2023: 23 of 24 AI bots blockedOct 2023: 23 of 24 AI bots blockedNov 2023: 23 of 24 AI bots blockedDec 2023: 23 of 24 AI bots blockedJan 2024: 23 of 24 AI bots blockedFeb 2024: 23 of 24 AI bots blockedMar 2024: 23 of 24 AI bots blockedApr 2024: 23 of 24 AI bots blockedMay 2024: 23 of 24 AI bots blockedJun 2024: 23 of 24 AI bots blockedJul 2024: 23 of 24 AI bots blockedAug 2024: 23 of 24 AI bots blockedSep 2024: 23 of 24 AI bots blockedOct 2024: 23 of 24 AI bots blockedNov 2024: 23 of 24 AI bots blockedDec 2024: 23 of 24 AI bots blockedJan 2025: 23 of 24 AI bots blockedFeb 2025: 23 of 24 AI bots blockedMar 2025: 23 of 24 AI bots blockedApr 2025: 23 of 24 AI bots blockedMay 2025: 23 of 24 AI bots blockedJun 2025: 23 of 24 AI bots blockedJul 2025: 23 of 24 AI bots blockedAug 2025: 23 of 24 AI bots blockedSep 2025: 23 of 24 AI bots blockedOct 2025: 23 of 24 AI bots blockedNov 2025: 23 of 24 AI bots blockedDec 2025: 22 of 24 AI bots blockedJan 2026: 22 of 24 AI bots blockedFeb 2026: 22 of 24 AI bots blockedMar 2026: 22 of 24 AI bots blockedApr 2026: 23 of 24 AI bots blockedMay 2026: 23 of 24 AI bots blockedJun 2026: 23 of 24 AI bots blockedJul 2026: 22 of 24 AI bots blockedAug 2026: 22 of 24 AI bots blockedSep 2026: 22 of 24 AI bots blockedOct 2026: 22 of 24 AI bots blocked222023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Mar to Jul 2026
    Mixed Now blocked: Bytespider. Now allowed: Claude-SearchBot.
    Found in the Internet Archive's copies: changed between 1 Mar 2026 and 2 Jul 2026. Evidence: before · after
    Show exactly what changed
    (files too large to diff inline)
  • Dec 2025
    Less blocking Now allowed: OAI-SearchBot.
    Found in the Internet Archive's copies: changed between 1 Nov 2025 and 1 Dec 2025. Evidence: before · after
    Show exactly what changed
    (files too large to diff inline)
  • Sep 2025
    Signals Named GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent, anthropic-ai and 7 more explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Jul 2025 and 1 Sep 2025. Evidence: before · after
    Show exactly what changed
    (files too large to diff inline)

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.