The Walled Web Newsletter

All sites / Local, listings and reviews

Indeed

indeed.com
Blocks some

Its robots.txt (the public rules file for bots) mentions GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and 16 more, but still lets all 12 major AI bots into the site. But the homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site. The written rules say come in; the front door disagrees.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Allowed

0 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Allowed

0 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Allowed
GPTBotOpenAI Allowed Named, line 426 Says yes
ClaudeBotAnthropic Allowed Named, line 426 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Allowed Named, line 186 Says yes
Applebot-ExtendedApple · opt-out name, not a bot Allowed Named, line 426 Says yes
CCBotCommon Crawl Allowed Named, line 426 Says yes
Meta-ExternalAgentMeta Allowed Named, line 426 Suggests yes
BytespiderByteDance Allowed Named, line 426 No statement found
MistralAI-TrainingMistral AI Allowed Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Allowed Named, line 426 Does not say
anthropic-aiAnthropic (legacy) · old name Allowed Named, line 426 Unknown
Bots for AI search · Allowed
OAI-SearchBotOpenAI Allowed Named, line 186 Says yes
Claude-SearchBotAnthropic Allowed Named, line 186 Says yes
PerplexityBotPerplexity Allowed Named, line 186 Suggests yes
AmazonbotAmazon Allowed Named, line 426 Says yes
Amzn-SearchBotAmazon Allowed Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Allowed Named, line 186 Says yes
MistralAI-IndexMistral AI Allowed Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Allowed Named, line 186 Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI Allowed Named, line 186 Says not always
Claude-UserAnthropic Allowed Named, line 186 Says yes
Perplexity-UserPerplexity Allowed Named, line 186 Says it mostly ignores it
Google-AgentGoogle Allowed Named, line 186 Says it mostly ignores it
Meta-ExternalFetcherMeta Allowed Named, line 186 Says not always
Amzn-UserAmazon Allowed Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Allowed Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNamed
BingbotRegular search engine, for comparisonAllowedNot named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

PetalBotHas rulesline 192Not in our verified list
YouBotHas rulesline 196Not in our verified list
Gemini-Deep-ResearchHas rulesline 198Not in our verified list
Google-CloudVertexBotHas rulesline 202Documented by its operator
xAI-GrokHas rulesline 205Operator publishes no such name
Grok-DeepSearchHas rulesline 206Operator publishes no such name
GoogleOtherBlockedline 405Documented by its operator
omgiliBlockedline 406Not in our verified list
omgilibotBlockedline 407Not in our verified list
WebzioBlockedline 408Not in our verified list
Webzio-ExtendedBlockedline 409Not in our verified list
FacebookBotHas rulesline 429Not in our verified list
cohere-training-data-crawlerHas rulesline 434Not in our verified list
DeepSeekBotHas rulesline 439Not in our verified list
GrokBotHas rulesline 440Operator publishes no such name
DiffbotHas rulesline 441Documented by its operator
TimpibotHas rulesline 444Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 605 lines · 384 lines about AI bots Source
# Default rules for all unspecified bots. Note that these apply to Bingbot too.User-agent: *Allow: /Allow: /*&start=0&Allow: /*&start=10&Allow: /*&start=20&Allow: /*&start=30&Allow: /*&start=40&Allow: /*&start=50&Allow: /*&start=60&Allow: /*&start=70&Allow: /*&start=80&Allow: /*&start=90&Disallow: /*rt=ncDisallow: /*&alid=Disallow: /*&calert=Disallow: /*&iafilter=Disallow: /*&mna=Disallow: /*?rssDisallow: /*&serpstart=Disallow: /*/vaclkDisallow: /addlLoc/Disallow: /ads/Disallow: /advanced_searchDisallow: /alertDisallow: /api/fetch/mc-anonDisallow: /api/getrecjobsDisallow: /applystartDisallow: /cmp/_/Disallow: /cmp/_c/claim/*Disallow: /cmp/_rpc/Disallow: /cmp/addlinkDisallow: /cmp/addvideoDisallow: /cmp/Login*Disallow: /cmp/*/analyticsDisallow: /cmp/*/company-questionsDisallow: /cmp/*/peopleDisallow: /cmp/*/write-reviewDisallow: /community/Disallow: /company/*Disallow: /conversion/Disallow: /cookiemigrator/Disallow: /create-resume/lp/Disallow: /cdn-cgi/Disallow: /%E4%BB%95%E4%BA%8B?Disallow: /%E5%B7%A5%E4%BD%9C/Disallow: /%E6%B1%82%E4%BA%BA/JP/Disallow: /%E6%B1%82%E4%BA%BA/title/Disallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?Disallow: /%EC%B7%A8%EC%97%85/KR/Disallow: /%EC%B7%A8%EC%97%85/title/Disallow: /empleo/Disallow: /emploi/Disallow: /emplois/FR/Disallow: /emplois/title/Disallow: /emprego/Disallow: /empregos/BR/Disallow: /empregos/titleDisallow: /forum/profileDisallow: /g/Disallow: /generate-questionDisallow: /graphqlDisallow: /imgpingDisallow: /ita?Disallow: /ita/v1/publisherDisallow: /ja/clk?Disallow: /ja/imp.gifDisallow: /job/Disallow: /Job/Disallow: /jobb/Disallow: /jobrollDisallow: /jobs/AE/Disallow: /jobs/AQ/Disallow: /Jobs/AT/Disallow: /jobs/AU/Disallow: /jobs/BE/Disallow: /jobs/BH/Disallow: /jobs/CA/Disallow: /jobs/CZ/Disallow: /jobs/DE/Disallow: /Jobs/DE/Disallow: /jobs/DK/Disallow: /jobs/FI/Disallow: /jobs/GB/Disallow: /jobs/GR/Disallow: /jobs/HK/Disallow: /jobs/HU/Disallow: /jobs/ID/Disallow: /jobs/IE/Disallow: /jobs/IL/Disallow: /jobs/IN/Disallow: /jobs/KW/Disallow: /jobs/LU/Disallow: /jobs/MY/Disallow: /jobs/NO/Disallow: /jobs/NZ/Disallow: /jobs/OM/Disallow: /jobs/PE/Disallow: /jobs/PH/Disallow: /jobs/PK/Disallow: /jobs/PT/Disallow: /jobs/QA/Disallow: /jobs/RO/Disallow: /jobs/RU/Disallow: /jobs/SA/Disallow: /jobs/SG/Disallow: /jobs/titleDisallow: /Jobs/titleDisallow: /jobs/TR/Disallow: /jobs/TW/Disallow: /jobs/US/Disallow: /jobs/VE/Disallow: /jobs/ZA/Disallow: /jobtrends/trends/log/*Disallow: /jwidget.js*Disallow: /me/*/pdfDisallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?Disallow: /m/moreLocDisallow: /m/newjobsDisallow: /m/recommendedDisallow: /m/rpc/Disallow: /m/viewjobDisallow: /my/Disallow: /*oc=1Disallow: /ofertas/ES/Disallow: /ofertas/title/Disallow: /offerta-lavoroDisallow: /offerte-lavoro/IT/Disallow: /offerte-lavoro/titleDisallow: /pagead/Disallow: /poka%C5%BCprac%C4%99?Disallow: /praca/Disallow: /preferencesDisallow: /promo/Disallow: /*radius=Disallow: /rc/Disallow: /rdr/Disallow: /recommendedjobsDisallow: /record-page-tagsDisallow: /resumes/accountDisallow: /resumes/advanced?Disallow: /resumes/alertDisallow: /resumes*radius=Disallow: /resumes*rb=Disallow: /resumes*res=Disallow: /resumes/rpc/Disallow: /resumes*sort=Disallow: /resumes*start=Disallow: /rpc/Disallow: /r/*/pdfDisallow: /rssDisallow: /setprefsDisallow: /*sid=Disallow: /*sp=0Disallow: /*&start=Disallow: /stcDisallow: /Stellen/CH/Disallow: /Stellen/titleDisallow: /tellafriendDisallow: /tmn/ccs/Disallow: /tos/banner.jsDisallow: /trabajo/Disallow: /uie/clkDisallow: /urlDisallow: /v1/alertsDisallow: /v1/impressionsDisallow: /vacature/Disallow: /vacature-bekijkenDisallow: /vacatures/NL/Disallow: /vacatures/title/Disallow: /ver-empleoDisallow: /ver-empregoDisallow: /ver-ofertaDisallow: /viewjobDisallow: /visajobbDisallow: /voir-emploiDisallow: /Zeige-JobDisallow: //applystartDisallow: /hire/o/ats-partnersDisallow: /hire/ats-partnersDisallow: /offers  … 184 lines not about AI bots# Rules for traditional search engines and for retrieval/grounding bots.User-agent: GooglebotUser-agent: Google-ExtendedUser-agent: ChatGPT-UserUser-agent: SlurpUser-agent: DuckDuckBotUser-agent: YahooJPUser-agent: PetalBotUser-agent: PerplexityBotUser-agent: DuckAssistBotUser-agent: Perplexity-UserUser-agent: YouBotUser-agent: Claude-UserUser-agent: Gemini-Deep-ResearchUser-agent: Claude-SearchBotUser-agent: OAI-SearchBotUser-agent: Google-AgentUser-agent: Google-CloudVertexBotUser-agent: Meta-WebIndexerUser-agent: Meta-ExternalFetcherUser-agent: xAI-GrokUser-agent: Grok-DeepSearchAllow: /Allow: /*&start=0&Allow: /*&start=10&Allow: /*&start=20&Allow: /*&start=30&Allow: /*&start=40&Allow: /*&start=50&Allow: /*&start=60&Allow: /*&start=70&Allow: /*&start=80&Allow: /*&start=90&Disallow: /*rt=ncDisallow: /*&alid=Disallow: /*&calert=Disallow: /*&iafilter=Disallow: /*&mna=Disallow: /*?rssDisallow: /*&serpstart=Disallow: /*/vaclkDisallow: /addlLoc/Disallow: /ads/Disallow: /advanced_searchDisallow: /alertDisallow: /api/fetch/mc-anonDisallow: /api/getrecjobsDisallow: /applystartDisallow: /cmp/_/Disallow: /cmp/_c/claim/*Disallow: /cmp/_rpc/Disallow: /cmp/addlinkDisallow: /cmp/addvideoDisallow: /cmp/Login*Disallow: /cmp/*/analyticsDisallow: /cmp/*/company-questionsDisallow: /cmp/*/peopleDisallow: /cmp/*/write-reviewDisallow: /community/Disallow: /company/*Disallow: /conversion/Disallow: /cookiemigrator/Disallow: /create-resume/lp/Disallow: /cdn-cgi/Disallow: /%E4%BB%95%E4%BA%8B?Disallow: /%E5%B7%A5%E4%BD%9C/Disallow: /%E6%B1%82%E4%BA%BA/JP/Disallow: /%E6%B1%82%E4%BA%BA/title/Disallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?Disallow: /%EC%B7%A8%EC%97%85/KR/Disallow: /%EC%B7%A8%EC%97%85/title/Disallow: /empleo/Disallow: /emploi/Disallow: /emplois/FR/Disallow: /emplois/title/Disallow: /emprego/Disallow: /empregos/BR/Disallow: /empregos/titleDisallow: /forum/profileDisallow: /g/Disallow: /generate-questionDisallow: /graphqlDisallow: /imgpingDisallow: /ita?Disallow: /ita/v1/publisherDisallow: /ja/clk?Disallow: /ja/imp.gifDisallow: /job/Disallow: /Job/Disallow: /jobb/Disallow: /jobrollDisallow: /jobs/AE/Disallow: /jobs/AQ/Disallow: /Jobs/AT/Disallow: /jobs/AU/Disallow: /jobs/BE/Disallow: /jobs/BH/Disallow: /jobs/CA/Disallow: /jobs/CZ/Disallow: /jobs/DE/Disallow: /Jobs/DE/Disallow: /jobs/DK/Disallow: /jobs/FI/Disallow: /jobs/GB/Disallow: /jobs/GR/Disallow: /jobs/HK/Disallow: /jobs/HU/Disallow: /jobs/ID/Disallow: /jobs/IE/Disallow: /jobs/IL/Disallow: /jobs/IN/Disallow: /jobs/KW/Disallow: /jobs/LU/Disallow: /jobs/MY/Disallow: /jobs/NO/Disallow: /jobs/NZ/Disallow: /jobs/OM/Disallow: /jobs/PE/Disallow: /jobs/PH/Disallow: /jobs/PK/Disallow: /jobs/PT/Disallow: /jobs/QA/Disallow: /jobs/RO/Disallow: /jobs/RU/Disallow: /jobs/SA/Disallow: /jobs/SG/Disallow: /jobs/titleDisallow: /Jobs/titleDisallow: /jobs/TR/Disallow: /jobs/TW/Disallow: /jobs/US/Disallow: /jobs/VE/Disallow: /jobs/ZA/Disallow: /jobtrends/trends/log/*Disallow: /jwidget.js*Disallow: /me/*/pdfDisallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?Disallow: /m/moreLocDisallow: /m/newjobsDisallow: /m/recommendedDisallow: /m/rpc/Disallow: /my/Disallow: /*oc=1Disallow: /ofertas/ES/Disallow: /ofertas/title/Disallow: /offerte-lavoro/IT/Disallow: /offerte-lavoro/titleDisallow: /pagead/Disallow: /poka%C5%BCprac%C4%99?Disallow: /praca/Disallow: /preferencesDisallow: /promo/Disallow: /*radius=Disallow: /rc/Disallow: /rdr/Disallow: /recommendedjobsDisallow: /record-page-tagsDisallow: /resumes/accountDisallow: /resumes/advanced?Disallow: /resumes/alertDisallow: /resumes*radius=Disallow: /resumes*rb=Disallow: /resumes*res=Disallow: /resumes/rpc/Disallow: /resumes*sort=Disallow: /resumes*start=Disallow: /rpc/Disallow: /r/*/pdfDisallow: /rssDisallow: /setprefsDisallow: /*sid=Disallow: /*sp=0Disallow: /*&start=Disallow: /stcDisallow: /Stellen/CH/Disallow: /Stellen/titleDisallow: /tellafriendDisallow: /tmn/ccs/Disallow: /tos/banner.jsDisallow: /trabajo/Disallow: /uie/clkDisallow: /urlDisallow: /v1/alertsDisallow: /v1/impressionsDisallow: /vacature/Disallow: /vacatures/NL/Disallow: /vacatures/title/Disallow: //applystartDisallow: /hire/o/ats-partnersDisallow: /hire/ats-partnersDisallow: /offers  User-agent: Mediapartners-GoogleAllow: /  User-agent: AdsBot-GoogleUser-agent: AdsBot-Google-MobileAllow: /create-resume/lp/Allow: /*sid=Disallow: /rpc/  User-agent: adidxbotAllow: /*sid=  User-agent: Googlebot-ImageUser-agent: msnbot-mediaDisallow: /trendgraph/*  … 21 lines not about AI bots# Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots.User-Agent: OmniExplorer_BotUser-agent: ia_archiverUser-agent: 008User-agent: JobdiggerSpiderUser-agent: CliqzbotUser-agent: GoogleOtherUser-agent: omgiliUser-agent: omgilibotUser-agent: WebzioUser-agent: Webzio-ExtendedUser-agent: ScrapyUser-agent: MyCentralAIScraperBotDisallow: /   # Rules for Social Media / Link Preview bots.User-agent: facebookexternalhitUser-agent: WhatsAppUser-agent: snapchatUser-agent: TelegramBotUser-agent: TwitterbotUser-agent: Slackbot-LinkExpandingAllow: /  … 11 lines not about AI bots# Rules for Foundational (Training) bots.User-agent: GPTBotUser-agent: CCBotUser-agent: anthropic-aiUser-agent: FacebookBotUser-agent: AmazonBotUser-agent: Applebot-ExtendedUser-agent: BytespiderUser-agent: BaiduspiderUser-agent: cohere-training-data-crawlerUser-agent: FriendlyCrawlerUser-agent: img2datasetUser-agent: Meta-ExternalAgentUser-agent: ClaudeBotUser-agent: DeepSeekBotUser-agent: GrokBotUser-agent: DiffbotUser-agent: AI2BotUser-agent: PanguBotUser-agent: TimpibotAllow: /*&start=0&Allow: /*&start=10&Allow: /*&start=20&Allow: /*&start=30&Allow: /*&start=40&Allow: /*&start=50&Allow: /*&start=60&Allow: /*&start=70&Allow: /*&start=80&Allow: /*&start=90&Disallow: /*rt=ncDisallow: /*&alid=Disallow: /*&calert=Disallow: /*&iafilter=Disallow: /*&mna=Disallow: /*?rssDisallow: /*&serpstart=Disallow: /*/vaclkDisallow: /addlLoc/Disallow: /ads/Disallow: /advanced_searchDisallow: /alertDisallow: /api/fetch/mc-anonDisallow: /api/getrecjobsDisallow: /applystartDisallow: /community/Disallow: /company/*Disallow: /conversion/Disallow: /cookiemigrator/Disallow: /create-resume/lp/Disallow: /cdn-cgi/Disallow: /%E4%BB%95%E4%BA%8B?Disallow: /%E6%B1%82%E4%BA%BA/JP/Disallow: /%E6%B1%82%E4%BA%BA/title/Disallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?Disallow: /%EC%B7%A8%EC%97%85/KR/Disallow: /%EC%B7%A8%EC%97%85/title/Disallow: /empleo/Disallow: /emploi/Disallow: /emplois/FR/Disallow: /emplois/title/Disallow: /emprego/Disallow: /forum/profileDisallow: /g/Disallow: /generate-questionDisallow: /graphqlDisallow: /imgpingDisallow: /ita?Disallow: /ita/v1/publisherDisallow: /ja/clk?Disallow: /ja/imp.gifDisallow: /job/Disallow: /Job/Disallow: /jobrollDisallow: /jobtrends/trends/log/*Disallow: /jwidget.js*Disallow: /me/*/pdfDisallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?Disallow: /m/moreLocDisallow: /m/newjobsDisallow: /m/recommendedDisallow: /m/rpc/Disallow: /my/Disallow: /*oc=1Disallow: /pagead/Disallow: /poka%C5%BCprac%C4%99?Disallow: /preferencesDisallow: /*radius=Disallow: /rc/Disallow: /rdr/Disallow: /recommendedjobsDisallow: /record-page-tagsDisallow: /resumes/accountDisallow: /resumes/advanced?Disallow: /resumes/alertDisallow: /resumes*radius=Disallow: /resumes*rb=Disallow: /resumes*res=Disallow: /resumes/rpc/Disallow: /resumes*sort=Disallow: /resumes*start=Disallow: /rpc/Disallow: /r/*/pdfDisallow: /rssDisallow: /setprefsDisallow: /*sid=Disallow: /*sp=0Disallow: /*&start=Disallow: /stcDisallow: /Stellen/CH/Disallow: /Stellen/titleDisallow: /tellafriendDisallow: /tmn/ccs/Disallow: /tos/banner.jsDisallow: /uie/clkDisallow: /urlDisallow: /v1/alertsDisallow: /v1/impressionsDisallow: /vacature/Disallow: /vacature-bekijkenDisallow: /ver-empleoDisallow: /ver-empregoDisallow: /ver-ofertaDisallow: /visajobbDisallow: /voir-emploiDisallow: /Zeige-JobDisallow: /hire/o/ats-partnersDisallow: /hire/ats-partnersDisallow: /offersDisallow: /c/info/Disallow: /hire/how-to-hire/Disallow: /hire/interview-questions/Disallow: /recruiting/interviewfragen/Disallow: /recruiting/stellenbeschreibung/Disallow: /recrutement/description-du-poste/Disallow: /recrutement/questions-entretien-dembauche/Disallow: /personeel/functiebeschrijving/Disallow: /companies/Disallow: /cmp/Disallow: /career-advice/Disallow: /conselho-de-carreira/Disallow: /hire/job-description/Disallow: /karriere-guide/Disallow: /conseils-carrieres/Disallow: /conseils-carriere/ Disallow: /orientacion-profesional/Disallow: /porady-zawodowe/Disallow: /guida-alla-carriera/Disallow: /職涯貼士/Disallow: /carrieregids/Disallow: /karriarrad/Disallow: /orientacion-laboral/ Disallow: /career/Disallow: /*jobs.htmlDisallow: /jobsDisallow: /viewjobDisallow: /certifications/Disallow: /trabajoDisallow: /ofertasDisallow: /%D0%B2%D0%B0%D0%BA%D0%B0%D0%BD%D1%81%D0%B8%D0%B8Disallow: /emploisDisallow: /%E6%B1%82%E4%BA%BADisallow: /m/jobsDisallow: /m/viewjobDisallow: /vacaturesDisallow: /offerte-lavoroDisallow: /Vi%E1%BB%87c-l%C3%A0mDisallow: /JobsDisallow: /%D9%88%D8%B8%D8%A7%D8%A6%D9%81Disallow: /%EC%B7%A8%EC%97%85Disallow: /JobbDisallow: /empregosDisallow: /pracaDisallow: /%E5%B7%A5%E4%BD%9CDisallow: /jobbDisallow: /%E0%B8%87%E0%B8%B2%E0%B8%99Disallow: /lowongan-kerjaDisallow: /StellenDisallow: /q-Disallow: /l-

Notes about AI written in the file: "Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots." · "Rules for Foundational (Training) bots."

Everything else we checked

Front doorHuman check The homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site (HTTP 403).GET https://www.indeed.com/ as WalledWebBot, from a residential connection in the United States
llms.txtRefused (HTTP 403)https://www.indeed.com/llms.txt
ai.txtRefused (HTTP 403)
TDMRepRefused (HTTP 403)/.well-known/tdmrep.json (W3C Community Group report)
Page tagsHomepage not read.
Anti-bot softwareServed through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIHomepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of ServiceNo Terms of Service page found.
Not foundNo Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

43 versions on record since 1 Jan 2023

OverallJan 2023: Overall allowedFeb 2023: Overall allowedMar 2023: Overall partialApr 2023: Overall partialMay 2023: Overall partialJun 2023: Overall partialJul 2023: Overall allowedAug 2023: Overall allowedSep 2023: Overall allowedOct 2023: Overall allowedNov 2023: Overall allowedDec 2023: Overall allowedJan 2024: Overall allowedFeb 2024: Overall allowedMar 2024: Overall allowedApr 2024: Overall allowedMay 2024: Overall allowedJun 2024: Overall allowedJul 2024: Overall allowedAug 2024: Overall allowedSep 2024: Overall allowedOct 2024: Overall allowedNov 2024: Overall allowedDec 2024: Overall allowedJan 2025: Overall allowedFeb 2025: Overall allowedMar 2025: Overall allowedApr 2025: Overall allowedMay 2025: Overall allowedJun 2025: Overall allowedJul 2025: Overall allowedAug 2025: Overall allowedSep 2025: Overall allowedOct 2025: Overall allowedNov 2025: Overall allowedDec 2025: Overall allowedJan 2026: Overall allowedFeb 2026: Overall allowedMar 2026: Overall allowedApr 2026: Overall allowedMay 2026: Overall allowedJun 2026: Overall allowedJul 2026: Overall allowedAug 2026: Overall allowedSep 2026: Overall allowedOct 2026: Overall allowedTrainingJan 2023: Training AllowedFeb 2023: Training AllowedMar 2023: Training Some blockedApr 2023: Training Some blockedMay 2023: Training Some blockedJun 2023: Training Some blockedJul 2023: Training AllowedAug 2023: Training AllowedSep 2023: Training AllowedOct 2023: Training AllowedNov 2023: Training AllowedDec 2023: Training AllowedJan 2024: Training AllowedFeb 2024: Training AllowedMar 2024: Training AllowedApr 2024: Training AllowedMay 2024: Training AllowedJun 2024: Training AllowedJul 2024: Training AllowedAug 2024: Training AllowedSep 2024: Training AllowedOct 2024: Training AllowedNov 2024: Training AllowedDec 2024: Training AllowedJan 2025: Training AllowedFeb 2025: Training AllowedMar 2025: Training AllowedApr 2025: Training AllowedMay 2025: Training AllowedJun 2025: Training AllowedJul 2025: Training AllowedAug 2025: Training AllowedSep 2025: Training AllowedOct 2025: Training AllowedNov 2025: Training AllowedDec 2025: Training AllowedJan 2026: Training AllowedFeb 2026: Training AllowedMar 2026: Training AllowedApr 2026: Training AllowedMay 2026: Training AllowedJun 2026: Training AllowedJul 2026: Training AllowedAug 2026: Training AllowedSep 2026: Training AllowedOct 2026: Training AllowedSearchJan 2023: Search AllowedFeb 2023: Search AllowedMar 2023: Search AllowedApr 2023: Search AllowedMay 2023: Search AllowedJun 2023: Search AllowedJul 2023: Search AllowedAug 2023: Search AllowedSep 2023: Search AllowedOct 2023: Search AllowedNov 2023: Search AllowedDec 2023: Search AllowedJan 2024: Search AllowedFeb 2024: Search AllowedMar 2024: Search AllowedApr 2024: Search AllowedMay 2024: Search AllowedJun 2024: Search AllowedJul 2024: Search AllowedAug 2024: Search AllowedSep 2024: Search AllowedOct 2024: Search AllowedNov 2024: Search AllowedDec 2024: Search AllowedJan 2025: Search AllowedFeb 2025: Search AllowedMar 2025: Search AllowedApr 2025: Search AllowedMay 2025: Search AllowedJun 2025: Search AllowedJul 2025: Search AllowedAug 2025: Search AllowedSep 2025: Search AllowedOct 2025: Search AllowedNov 2025: Search AllowedDec 2025: Search AllowedJan 2026: Search AllowedFeb 2026: Search AllowedMar 2026: Search AllowedApr 2026: Search AllowedMay 2026: Search AllowedJun 2026: Search AllowedJul 2026: Search AllowedAug 2026: Search AllowedSep 2026: Search AllowedOct 2026: Search AllowedAssistantsJan 2023: Assistants AllowedFeb 2023: Assistants AllowedMar 2023: Assistants AllowedApr 2023: Assistants AllowedMay 2023: Assistants AllowedJun 2023: Assistants AllowedJul 2023: Assistants AllowedAug 2023: Assistants AllowedSep 2023: Assistants AllowedOct 2023: Assistants AllowedNov 2023: Assistants AllowedDec 2023: Assistants AllowedJan 2024: Assistants AllowedFeb 2024: Assistants AllowedMar 2024: Assistants AllowedApr 2024: Assistants AllowedMay 2024: Assistants AllowedJun 2024: Assistants AllowedJul 2024: Assistants AllowedAug 2024: Assistants AllowedSep 2024: Assistants AllowedOct 2024: Assistants AllowedNov 2024: Assistants AllowedDec 2024: Assistants AllowedJan 2025: Assistants AllowedFeb 2025: Assistants AllowedMar 2025: Assistants AllowedApr 2025: Assistants AllowedMay 2025: Assistants AllowedJun 2025: Assistants AllowedJul 2025: Assistants AllowedAug 2025: Assistants AllowedSep 2025: Assistants AllowedOct 2025: Assistants AllowedNov 2025: Assistants AllowedDec 2025: Assistants AllowedJan 2026: Assistants AllowedFeb 2026: Assistants AllowedMar 2026: Assistants AllowedApr 2026: Assistants AllowedMay 2026: Assistants AllowedJun 2026: Assistants AllowedJul 2026: Assistants AllowedAug 2026: Assistants AllowedSep 2026: Assistants AllowedOct 2026: Assistants AllowedBotsblockedMar 2023: 1 of 24 AI bots blockedApr 2023: 1 of 24 AI bots blockedMay 2023: 1 of 24 AI bots blockedJun 2023: 1 of 24 AI bots blocked02023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Apr to Sep 2026
    Signals Named ClaudeBot, Meta-ExternalAgent, AI2Bot, Claude-SearchBot, PerplexityBot, DuckAssistBot and 5 more explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 3 Apr 2026 and 10 Sep 2026. Evidence: before · after
    Show exactly what changed
    Allow: /
    - Allow: /hire/*?*isid=
    - Allow: /personeel/*?*isid=
    - Allow: /reclutamiento/*?*isid=
    - Allow: /recruiting/*?*isid=
    - Allow: /recrutement/*?*isid=
    + Allow: /*&start=0&
    + Allow: /*&start=10&
    + Allow: /*&start=20&
    + Allow: /*&start=30&
    + Allow: /*&start=40&
    + Allow: /*&start=50&
    + Allow: /*&start=60&
    + Allow: /*&start=70&
    + Allow: /*&start=80&
    + Allow: /*&start=90&
    Disallow: /*rt=nc
    …
    Disallow: /*?rss
    + Disallow: /*&serpstart=
    + Disallow: /*/vaclk
    Disallow: /addlLoc/
    …
    Disallow: /%E5%B7%A5%E4%BD%9C/
    - Disallow: /%E5%B7%A5%E4%BD%9C/CN/
    - Disallow: /%E5%B7%A5%E4%BD%9C/title/
    Disallow: /%E6%B1%82%E4%BA%BA/JP/
    …
    Disallow: /ita?
    + Disallow: /ita/v1/publisher
    Disallow: /ja/clk?
    …
    Disallow: /jobb/
    - Disallow: /jobb/SE/
    - Disallow: /jobb/title/
    Disallow: /jobroll
    …
    Disallow: /praca/
    - Disallow: /praca/PL/
    - Disallow: /praca/title/
    Disallow: /preferences
    …
    Disallow: /recommendedjobs
    + Disallow: /record-page-tags
    Disallow: /resumes/account
    …
    Disallow: /Stellen/title
    - Disallow: /survey/*
    Disallow: /tellafriend
    …
    Disallow: /trabajo/
    - Disallow: /trabajo/AR/
    - Disallow: /trabajo/CL/
    - Disallow: /trabajo/CO/
    - Disallow: /trabajo/MX/
    - Disallow: /trabajo/title/
    + Disallow: /uie/clk
    Disallow: /url
    …
    … 399 more lines
  • Nov 2024
    Signals Named Applebot-Extended, Bytespider, OAI-SearchBot, Amazonbot and ChatGPT-User explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Sep 2024 and 1 Nov 2024. Evidence: before · after
    Show exactly what changed
    Disallow: /cmp/addvideo
    + Disallow: /cmp/Login*
    Disallow: /cmp/*/analytics
    …
    Disallow: /*sp=0
    + Disallow: /*&start=
    Disallow: /stc
    …
    Disallow: /Zeige-Job?
    + Disallow: //applystart
    User-agent: Googlebot
    + User-agent: OAI-SearchBot
    + User-agent: ChatGPT-User
    Allow: /
    …
    Allow: /recrutement/*?*isid=
    + Allow: /viewjob?
    Disallow: /*rt=nc
    …
    Disallow: /*&serpstart=
    + Disallow: /*/vaclk
    Disallow: /addlLoc/
    …
    Disallow: /cmp/addvideo
    + Disallow: /cmp/Login*
    Disallow: /cmp/*/analytics
    …
    Disallow: /graphql
    + Disallow: /hire/o/ats-partners
    Disallow: /imgping
    …
    Disallow: /*sp=0
    + Disallow: /*&start=100
    Disallow: /stc
    …
    Disallow: /vacatures/title/
    + Disallow: //applystart
    …
    Disallow: /cmp/addvideo
    + Disallow: /cmp/Login*
    Disallow: /cmp/*/analytics
    …
    Disallow: /graphql
    + Disallow: /hire/o/ats-partners
    Disallow: /imgping
    …
    Disallow: /*sp=0
    + Disallow: /*&start=
    Disallow: /stc
    …
    Disallow: /Zeige-Job?
    + Disallow: //applystart
    …
    + User-agent: GoogleOther
    + Disallow: /
    +
    … 13 more lines
  • Feb 2024
    Signals Named CCBot and anthropic-ai explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Jan 2024 and 1 Feb 2024. Evidence: before · after
    Show exactly what changed
    Disallow: /api/getrecjobs
    - Disallow: /applystart/
    + Disallow: /applystart
    Disallow: /cmp/_/
    …
    Disallow: /m/jobalerts?
    + Disallow: /m/moreLoc
    Disallow: /m/newjobs
    …
    Disallow: /api/getrecjobs
    - Disallow: /applystart/
    + Disallow: /applystart
    Disallow: /cmp/_/
    …
    Disallow: /m/jobalerts?
    + Disallow: /m/moreLoc
    Disallow: /m/newjobs
    …
    Disallow: /api/getrecjobs
    - Disallow: /applystart/
    + Disallow: /applystart
    Disallow: /cmp/_/
    …
    Disallow: /m/jobalerts?
    + Disallow: /m/moreLoc
    Disallow: /m/newjobs
    …
    User-agent: GPTBot
    - Disallow: /c/info/
    - Disallow: /hire/how-to-hire/
    - Disallow: /hire/interview-questions/
    - Disallow: /recruiting/interviewfragen/
    - Disallow: /recruiting/stellenbeschreibung/
    - Disallow: /recrutement/description-du-poste/
    - Disallow: /recrutement/questions-entretien-dembauche/
    - Disallow: /personeel/functiebeschrijving/
    - Disallow: /companies/
    - Disallow: /cmp/
    - Disallow: /career-advice/
    - Disallow: /conselho-de-carreira/
    - Disallow: /hire/job-description/
    - Disallow: /karriere-guide/
    - Disallow: /conseils-carrieres/
    - Disallow: /conseils-carriere/
    - Disallow: /orientacion-profesional/
    - Disallow: /porady-zawodowe/
    - Disallow: /guida-alla-carriera/
    - Disallow: /職涯貼士/
    - Disallow: /carrieregids/
    - Disallow: /karriarrad/
    - Disallow: /orientacion-laboral/
    - Disallow: /career/
    - Disallow: /*jobs.html
    - Disallow: /jobs
    - Disallow: /viewjob
    - Disallow: /certifications/
    -
    User-agent: Google-Extended
    + User-agent: CCBot
    + User-agent: anthropic-ai
    … 1 more lines
  • Nov 2023
    Signals Named Google-Extended explicitly, without changing what it may reach.
    Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after
    Show exactly what changed
    Disallow: /api/getrecjobs
    + Disallow: /applystart/
    Disallow: /cmp/_/
    …
    Disallow: /api/getrecjobs
    + Disallow: /applystart/
    Disallow: /cmp/_/
    …
    Disallow: /api/getrecjobs
    + Disallow: /applystart/
    Disallow: /cmp/_/
    …
    Disallow: /career/
    - Disallow: jobs.html
    + Disallow: /*jobs.html
    Disallow: /jobs
    Disallow: /viewjob
    - Disallow: /certifications/
    + Disallow: /certifications/
    +
    + User-agent: Google-Extended
    + Disallow: /c/info/
    + Disallow: /hire/how-to-hire/
    + Disallow: /hire/interview-questions/
    + Disallow: /recruiting/interviewfragen/
    + Disallow: /recruiting/stellenbeschreibung/
    + Disallow: /recrutement/description-du-poste/
    + Disallow: /recrutement/questions-entretien-dembauche/
    + Disallow: /personeel/functiebeschrijving/
    + Disallow: /companies/
    + Disallow: /cmp/
    + Disallow: /career-advice/
    + Disallow: /conselho-de-carreira/
    + Disallow: /hire/job-description/
    + Disallow: /karriere-guide/
    + Disallow: /conseils-carrieres/
    + Disallow: /conseils-carriere/
    + Disallow: /orientacion-profesional/
    + Disallow: /porady-zawodowe/
    + Disallow: /guida-alla-carriera/
    + Disallow: /職涯貼士/
    + Disallow: /carrieregids/
    + Disallow: /karriarrad/
    + Disallow: /orientacion-laboral/
    + Disallow: /career/
    + Disallow: /*jobs.html
    + Disallow: /jobs
    + Disallow: /viewjob
    + Disallow: /certifications/
  • Sep 2023
    Signals Named GPTBot explicitly, without changing what it may reach.
    Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
    Show exactly what changed
    Allow: /hire/*?*isid=
    + Allow: /personeel/*?*isid=
    + Allow: /reclutamiento/*?*isid=
    + Allow: /recruiting/*?*isid=
    + Allow: /recrutement/*?*isid=
    Disallow: /*rt=nc
    …
    Disallow: /*?rss
    + Disallow: /addlLoc/
    + Disallow: /ads/
    Disallow: /advanced_search
    …
    Disallow: /cmp/*/write-review
    - Disallow: /community/*
    + Disallow: /community/
    Disallow: /company/*
    …
    Disallow: /setprefs
    + Disallow: /*sid=
    Disallow: /*sp=0
    …
    Allow: /hire/*?*isid=
    + Allow: /personeel/*?*isid=
    + Allow: /reclutamiento/*?*isid=
    + Allow: /recruiting/*?*isid=
    + Allow: /recrutement/*?*isid=
    Disallow: /*rt=nc
    …
    Disallow: /*?rss
    + Disallow: /*&serpstart=
    Disallow: /addlLoc/
    …
    Disallow: /cmp/*/write-review
    - Disallow: /community/*
    + Disallow: /community/
    Disallow: /company/*
    …
    Disallow: /setprefs
    + Disallow: /*sid=
    Disallow: /*sp=0
    …
    Allow: /hire/*?*isid=
    + Allow: /personeel/*?*isid=
    + Allow: /reclutamiento/*?*isid=
    + Allow: /recruiting/*?*isid=
    + Allow: /recrutement/*?*isid=
    Disallow: /*rt=nc
    …
    Disallow: /setprefs
    + Disallow: /*sid=
    Disallow: /*sp=0
    …
    User-agent: AdsBot-Google
    + User-agent: AdsBot-Google-Mobile
    Allow: /create-resume/lp/
    Allow: /*sid=
    + Disallow: /rpc/
    …
    Allow: /
    … 30 more lines
  • Jul 2023
    Less blocking Now allowed: CCBot.
    Found in the Internet Archive's copies: changed between 1 Jun 2023 and 1 Jul 2023. Evidence: before · after
    Show exactly what changed
    Disallow: /cmp/*/company-questions
    - Disallow: /cmp/*/jobs/
    Disallow: /cmp/*/people
    …
    Disallow: /Stellen/title
    + Disallow: /survey/*
    Disallow: /tellafriend
    + Disallow: /tmn/ccs/
    Disallow: /tos/banner.js
    …
    Disallow: /cmp/*/company-questions
    - Disallow: /cmp/*/jobs/
    Disallow: /cmp/*/people
    …
    Disallow: /ofertas/title/
    - Disallow: /offerta-lavoro
    Disallow: /offerte-lavoro/IT/
    …
    Disallow: /pagead/
    - Disallow: /poka%C5%BCprac%C4%99?
    Disallow: /praca/
    …
    Disallow: /Stellen/title
    + Disallow: /survey/*
    Disallow: /tellafriend
    + Disallow: /tmn/ccs/
    Disallow: /tos/banner.js
    …
    Disallow: /vacature/
    - Disallow: /vacature-bekijken?
    Disallow: /vacatures/NL/
    Disallow: /vacatures/title/
    - Disallow: /ver-empleo?
    - Disallow: /ver-emprego?
    - Disallow: /ver-oferta?
    - Disallow: /visajobb?
    - Disallow: /voir-emploi?
    - Disallow: /Zeige-Job?
    …
    Disallow: /cmp/*/company-questions
    - Disallow: /cmp/*/jobs/
    Disallow: /cmp/*/people
    …
    Disallow: /Stellen/title
    + Disallow: /survey/*
    Disallow: /tellafriend
    + Disallow: /tmn/ccs/
    Disallow: /tos/banner.js
    …
    - User-agent: CCBot
    - Disallow: /
    + User-agent: facebookexternalhit
    + User-agent: WhatsApp
    + User-agent: snapchat
    + User-agent: TelegramBot
    + User-agent: Twitterbot
    + Allow: /
  • Mar 2023
    More blocking Now blocked: CCBot.
    Found in the Internet Archive's copies: changed between 1 Feb 2023 and 1 Mar 2023. Evidence: before · after
    Show exactly what changed
    Disallow: /alert
    + Disallow: /api/fetch/mc-anon
    Disallow: /api/getrecjobs
    …
    Disallow: /create-resume/lp/
    + Disallow: /cdn-cgi/
    Disallow: /%E4%BB%95%E4%BA%8B?
    …
    Disallow: /g/
    + Disallow: /graphql
    Disallow: /imgping
    …
    Disallow: /me/*/pdf
    + Disallow: /m/basecamp/analytics.js
    Disallow: /m/jobalerts?
    …
    Disallow: /alert
    + Disallow: /api/fetch/mc-anon
    Disallow: /api/getrecjobs
    …
    Disallow: /g/
    + Disallow: /graphql
    Disallow: /imgping
    …
    Disallow: /me/*/pdf
    + Disallow: /m/basecamp/analytics.js
    Disallow: /m/jobalerts?
    …
    Disallow: /alert
    + Disallow: /api/fetch/mc-anon
    Disallow: /api/getrecjobs
    …
    Disallow: /g/
    + Disallow: /graphql
    Disallow: /imgping
    …
    Disallow: /me/*/pdf
    + Disallow: /m/basecamp/analytics.js
    Disallow: /m/jobalerts?
    …
    + User-agent: CCBot
    + Disallow: /

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.