{"domain":"indeed.com","name":"Indeed","category":"local","subcategory":"jobs","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) mentions GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and 16 more, but still lets all 12 major AI bots into the site. But the homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site. The written rules say come in; the front door disagrees.","computed_at":"2026-10-20T03:43:15.071Z","last_checked":"2026-10-19T13:33:24.994Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.indeed.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.indeed.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.indeed.com/robots.txt","hops":[{"from":"https://indeed.com/robots.txt","to":"https://www.indeed.com/robots.txt","status":301}],"blob":"1189922fa6724d518dfa4f636276c1f8cd5397f2fbd7ab997183c4a0fd6391a4","text":"# Default rules for all unspecified bots. Note that these apply to Bingbot too.\nUser-agent: *\nAllow: /\nAllow: /*&start=0&\nAllow: /*&start=10&\nAllow: /*&start=20&\nAllow: /*&start=30&\nAllow: /*&start=40&\nAllow: /*&start=50&\nAllow: /*&start=60&\nAllow: /*&start=70&\nAllow: /*&start=80&\nAllow: /*&start=90&\nDisallow: /*rt=nc\nDisallow: /*&alid=\nDisallow: /*&calert=\nDisallow: /*&iafilter=\nDisallow: /*&mna=\nDisallow: /*?rss\nDisallow: /*&serpstart=\nDisallow: /*/vaclk\nDisallow: /addlLoc/\nDisallow: /ads/\nDisallow: /advanced_search\nDisallow: /alert\nDisallow: /api/fetch/mc-anon\nDisallow: /api/getrecjobs\nDisallow: /applystart\nDisallow: /cmp/_/\nDisallow: /cmp/_c/claim/*\nDisallow: /cmp/_rpc/\nDisallow: /cmp/addlink\nDisallow: /cmp/addvideo\nDisallow: /cmp/Login*\nDisallow: /cmp/*/analytics\nDisallow: /cmp/*/company-questions\nDisallow: /cmp/*/people\nDisallow: /cmp/*/write-review\nDisallow: /community/\nDisallow: /company/*\nDisallow: /conversion/\nDisallow: /cookiemigrator/\nDisallow: /create-resume/lp/\nDisallow: /cdn-cgi/\nDisallow: /%E4%BB%95%E4%BA%8B?\nDisallow: /%E5%B7%A5%E4%BD%9C/\nDisallow: /%E6%B1%82%E4%BA%BA/JP/\nDisallow: /%E6%B1%82%E4%BA%BA/title/\nDisallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?\nDisallow: /%EC%B7%A8%EC%97%85/KR/\nDisallow: /%EC%B7%A8%EC%97%85/title/\nDisallow: /empleo/\nDisallow: /emploi/\nDisallow: /emplois/FR/\nDisallow: /emplois/title/\nDisallow: /emprego/\nDisallow: /empregos/BR/\nDisallow: /empregos/title\nDisallow: /forum/profile\nDisallow: /g/\nDisallow: /generate-question\nDisallow: /graphql\nDisallow: /imgping\nDisallow: /ita?\nDisallow: /ita/v1/publisher\nDisallow: /ja/clk?\nDisallow: /ja/imp.gif\nDisallow: /job/\nDisallow: /Job/\nDisallow: /jobb/\nDisallow: /jobroll\nDisallow: /jobs/AE/\nDisallow: /jobs/AQ/\nDisallow: /Jobs/AT/\nDisallow: /jobs/AU/\nDisallow: /jobs/BE/\nDisallow: /jobs/BH/\nDisallow: /jobs/CA/\nDisallow: /jobs/CZ/\nDisallow: /jobs/DE/\nDisallow: /Jobs/DE/\nDisallow: /jobs/DK/\nDisallow: /jobs/FI/\nDisallow: /jobs/GB/\nDisallow: /jobs/GR/\nDisallow: /jobs/HK/\nDisallow: /jobs/HU/\nDisallow: /jobs/ID/\nDisallow: /jobs/IE/\nDisallow: /jobs/IL/\nDisallow: /jobs/IN/\nDisallow: /jobs/KW/\nDisallow: /jobs/LU/\nDisallow: /jobs/MY/\nDisallow: /jobs/NO/\nDisallow: /jobs/NZ/\nDisallow: /jobs/OM/\nDisallow: /jobs/PE/\nDisallow: /jobs/PH/\nDisallow: /jobs/PK/\nDisallow: /jobs/PT/\nDisallow: /jobs/QA/\nDisallow: /jobs/RO/\nDisallow: /jobs/RU/\nDisallow: /jobs/SA/\nDisallow: /jobs/SG/\nDisallow: /jobs/title\nDisallow: /Jobs/title\nDisallow: /jobs/TR/\nDisallow: /jobs/TW/\nDisallow: /jobs/US/\nDisallow: /jobs/VE/\nDisallow: /jobs/ZA/\nDisallow: /jobtrends/trends/log/*\nDisallow: /jwidget.js*\nDisallow: /me/*/pdf\nDisallow: /m/basecamp/analytics.js\nDisallow: /m/jobalerts?\nDisallow: /m/moreLoc\nDisallow: /m/newjobs\nDisallow: /m/recommended\nDisallow: /m/rpc/\nDisallow: /m/viewjob\nDisallow: /my/\nDisallow: /*oc=1\nDisallow: /ofertas/ES/\nDisallow: /ofertas/title/\nDisallow: /offerta-lavoro\nDisallow: /offerte-lavoro/IT/\nDisallow: /offerte-lavoro/title\nDisallow: /pagead/\nDisallow: /poka%C5%BCprac%C4%99?\nDisallow: /praca/\nDisallow: /preferences\nDisallow: /promo/\nDisallow: /*radius=\nDisallow: /rc/\nDisallow: /rdr/\nDisallow: /recommendedjobs\nDisallow: /record-page-tags\nDisallow: /resumes/account\nDisallow: /resumes/advanced?\nDisallow: /resumes/alert\nDisallow: /resumes*radius=\nDisallow: /resumes*rb=\nDisallow: /resumes*res=\nDisallow: /resumes/rpc/\nDisallow: /resumes*sort=\nDisallow: /resumes*start=\nDisallow: /rpc/\nDisallow: /r/*/pdf\nDisallow: /rss\nDisallow: /setprefs\nDisallow: /*sid=\nDisallow: /*sp=0\nDisallow: /*&start=\nDisallow: /stc\nDisallow: /Stellen/CH/\nDisallow: /Stellen/title\nDisallow: /tellafriend\nDisallow: /tmn/ccs/\nDisallow: /tos/banner.js\nDisallow: /trabajo/\nDisallow: /uie/clk\nDisallow: /url\nDisallow: /v1/alerts\nDisallow: /v1/impressions\nDisallow: /vacature/\nDisallow: /vacature-bekijken\nDisallow: /vacatures/NL/\nDisallow: /vacatures/title/\nDisallow: /ver-empleo\nDisallow: /ver-emprego\nDisallow: /ver-oferta\nDisallow: /viewjob\nDisallow: /visajobb\nDisallow: /voir-emploi\nDisallow: /Zeige-Job\nDisallow: //applystart\nDisallow: /hire/o/ats-partners\nDisallow: /hire/ats-partners\nDisallow: /offers\n\n\n# Rules for traditional search engines and for retrieval/grounding bots.\nUser-agent: Googlebot\nUser-agent: Google-Extended\nUser-agent: ChatGPT-User\nUser-agent: Slurp\nUser-agent: DuckDuckBot\nUser-agent: YahooJP\nUser-agent: PetalBot\nUser-agent: PerplexityBot\nUser-agent: DuckAssistBot\nUser-agent: Perplexity-User\nUser-agent: YouBot\nUser-agent: Claude-User\nUser-agent: Gemini-Deep-Research\nUser-agent: Claude-SearchBot\nUser-agent: OAI-SearchBot\nUser-agent: Google-Agent\nUser-agent: Google-CloudVertexBot\nUser-agent: Meta-WebIndexer\nUser-agent: Meta-ExternalFetcher\nUser-agent: xAI-Grok\nUser-agent: Grok-DeepSearch\nAllow: /\nAllow: /*&start=0&\nAllow: /*&start=10&\nAllow: /*&start=20&\nAllow: /*&start=30&\nAllow: /*&start=40&\nAllow: /*&start=50&\nAllow: /*&start=60&\nAllow: /*&start=70&\nAllow: /*&start=80&\nAllow: /*&start=90&\nDisallow: /*rt=nc\nDisallow: /*&alid=\nDisallow: /*&calert=\nDisallow: /*&iafilter=\nDisallow: /*&mna=\nDisallow: /*?rss\nDisallow: /*&serpstart=\nDisallow: /*/vaclk\nDisallow: /addlLoc/\nDisallow: /ads/\nDisallow: /advanced_search\nDisallow: /alert\nDisallow: /api/fetch/mc-anon\nDisallow: /api/getrecjobs\nDisallow: /applystart\nDisallow: /cmp/_/\nDisallow: /cmp/_c/claim/*\nDisallow: /cmp/_rpc/\nDisallow: /cmp/addlink\nDisallow: /cmp/addvideo\nDisallow: /cmp/Login*\nDisallow: /cmp/*/analytics\nDisallow: /cmp/*/company-questions\nDisallow: /cmp/*/people\nDisallow: /cmp/*/write-review\nDisallow: /community/\nDisallow: /company/*\nDisallow: /conversion/\nDisallow: /cookiemigrator/\nDisallow: /create-resume/lp/\nDisallow: /cdn-cgi/\nDisallow: /%E4%BB%95%E4%BA%8B?\nDisallow: /%E5%B7%A5%E4%BD%9C/\nDisallow: /%E6%B1%82%E4%BA%BA/JP/\nDisallow: /%E6%B1%82%E4%BA%BA/title/\nDisallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?\nDisallow: /%EC%B7%A8%EC%97%85/KR/\nDisallow: /%EC%B7%A8%EC%97%85/title/\nDisallow: /empleo/\nDisallow: /emploi/\nDisallow: /emplois/FR/\nDisallow: /emplois/title/\nDisallow: /emprego/\nDisallow: /empregos/BR/\nDisallow: /empregos/title\nDisallow: /forum/profile\nDisallow: /g/\nDisallow: /generate-question\nDisallow: /graphql\nDisallow: /imgping\nDisallow: /ita?\nDisallow: /ita/v1/publisher\nDisallow: /ja/clk?\nDisallow: /ja/imp.gif\nDisallow: /job/\nDisallow: /Job/\nDisallow: /jobb/\nDisallow: /jobroll\nDisallow: /jobs/AE/\nDisallow: /jobs/AQ/\nDisallow: /Jobs/AT/\nDisallow: /jobs/AU/\nDisallow: /jobs/BE/\nDisallow: /jobs/BH/\nDisallow: /jobs/CA/\nDisallow: /jobs/CZ/\nDisallow: /jobs/DE/\nDisallow: /Jobs/DE/\nDisallow: /jobs/DK/\nDisallow: /jobs/FI/\nDisallow: /jobs/GB/\nDisallow: /jobs/GR/\nDisallow: /jobs/HK/\nDisallow: /jobs/HU/\nDisallow: /jobs/ID/\nDisallow: /jobs/IE/\nDisallow: /jobs/IL/\nDisallow: /jobs/IN/\nDisallow: /jobs/KW/\nDisallow: /jobs/LU/\nDisallow: /jobs/MY/\nDisallow: /jobs/NO/\nDisallow: /jobs/NZ/\nDisallow: /jobs/OM/\nDisallow: /jobs/PE/\nDisallow: /jobs/PH/\nDisallow: /jobs/PK/\nDisallow: /jobs/PT/\nDisallow: /jobs/QA/\nDisallow: /jobs/RO/\nDisallow: /jobs/RU/\nDisallow: /jobs/SA/\nDisallow: /jobs/SG/\nDisallow: /jobs/title\nDisallow: /Jobs/title\nDisallow: /jobs/TR/\nDisallow: /jobs/TW/\nDisallow: /jobs/US/\nDisallow: /jobs/VE/\nDisallow: /jobs/ZA/\nDisallow: /jobtrends/trends/log/*\nDisallow: /jwidget.js*\nDisallow: /me/*/pdf\nDisallow: /m/basecamp/analytics.js\nDisallow: /m/jobalerts?\nDisallow: /m/moreLoc\nDisallow: /m/newjobs\nDisallow: /m/recommended\nDisallow: /m/rpc/\nDisallow: /my/\nDisallow: /*oc=1\nDisallow: /ofertas/ES/\nDisallow: /ofertas/title/\nDisallow: /offerte-lavoro/IT/\nDisallow: /offerte-lavoro/title\nDisallow: /pagead/\nDisallow: /poka%C5%BCprac%C4%99?\nDisallow: /praca/\nDisallow: /preferences\nDisallow: /promo/\nDisallow: /*radius=\nDisallow: /rc/\nDisallow: /rdr/\nDisallow: /recommendedjobs\nDisallow: /record-page-tags\nDisallow: /resumes/account\nDisallow: /resumes/advanced?\nDisallow: /resumes/alert\nDisallow: /resumes*radius=\nDisallow: /resumes*rb=\nDisallow: /resumes*res=\nDisallow: /resumes/rpc/\nDisallow: /resumes*sort=\nDisallow: /resumes*start=\nDisallow: /rpc/\nDisallow: /r/*/pdf\nDisallow: /rss\nDisallow: /setprefs\nDisallow: /*sid=\nDisallow: /*sp=0\nDisallow: /*&start=\nDisallow: /stc\nDisallow: /Stellen/CH/\nDisallow: /Stellen/title\nDisallow: /tellafriend\nDisallow: /tmn/ccs/\nDisallow: /tos/banner.js\nDisallow: /trabajo/\nDisallow: /uie/clk\nDisallow: /url\nDisallow: /v1/alerts\nDisallow: /v1/impressions\nDisallow: /vacature/\nDisallow: /vacatures/NL/\nDisallow: /vacatures/title/\nDisallow: //applystart\nDisallow: /hire/o/ats-partners\nDisallow: /hire/ats-partners\nDisallow: /offers\n\n\nUser-agent: Mediapartners-Google\nAllow: /\n\n\nUser-agent: AdsBot-Google\nUser-agent: AdsBot-Google-Mobile\nAllow: /create-resume/lp/\nAllow: /*sid=\nDisallow: /rpc/\n\n\nUser-agent: adidxbot\nAllow: /*sid=\n\n\nUser-agent: Googlebot-Image\nUser-agent: msnbot-media\nDisallow: /trendgraph/*\n\n\n# Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots.\nUser-Agent: OmniExplorer_Bot\nUser-agent: ia_archiver\nUser-agent: 008\nUser-agent: JobdiggerSpider\nUser-agent: Cliqzbot\nUser-agent: GoogleOther\nUser-agent: omgili\nUser-agent: omgilibot\nUser-agent: Webzio\nUser-agent: Webzio-Extended\nUser-agent: Scrapy\nUser-agent: MyCentralAIScraperBot\nDisallow: / \n\n\n# Rules for Social Media / Link Preview bots.\nUser-agent: facebookexternalhit\nUser-agent: WhatsApp\nUser-agent: snapchat\nUser-agent: TelegramBot\nUser-agent: Twitterbot\nUser-agent: Slackbot-LinkExpanding\nAllow: /\n\n\n# Rules for Foundational (Training) bots.\nUser-agent: GPTBot\nUser-agent: CCBot\nUser-agent: anthropic-ai\nUser-agent: FacebookBot\nUser-agent: AmazonBot\nUser-agent: Applebot-Extended\nUser-agent: Bytespider\nUser-agent: Baiduspider\nUser-agent: cohere-training-data-crawler\nUser-agent: FriendlyCrawler\nUser-agent: img2dataset\nUser-agent: Meta-ExternalAgent\nUser-agent: ClaudeBot\nUser-agent: DeepSeekBot\nUser-agent: GrokBot\nUser-agent: Diffbot\nUser-agent: AI2Bot\nUser-agent: PanguBot\nUser-agent: Timpibot\nAllow: /*&start=0&\nAllow: /*&start=10&\nAllow: /*&start=20&\nAllow: /*&start=30&\nAllow: /*&start=40&\nAllow: /*&start=50&\nAllow: /*&start=60&\nAllow: /*&start=70&\nAllow: /*&start=80&\nAllow: /*&start=90&\nDisallow: /*rt=nc\nDisallow: /*&alid=\nDisallow: /*&calert=\nDisallow: /*&iafilter=\nDisallow: /*&mna=\nDisallow: /*?rss\nDisallow: /*&serpstart=\nDisallow: /*/vaclk\nDisallow: /addlLoc/\nDisallow: /ads/\nDisallow: /advanced_search\nDisallow: /alert\nDisallow: /api/fetch/mc-anon\nDisallow: /api/getrecjobs\nDisallow: /applystart\nDisallow: /community/\nDisallow: /company/*\nDisallow: /conversion/\nDisallow: /cookiemigrator/\nDisallow: /create-resume/lp/\nDisallow: /cdn-cgi/\nDisallow: /%E4%BB%95%E4%BA%8B?\nDisallow: /%E6%B1%82%E4%BA%BA/JP/\nDisallow: /%E6%B1%82%E4%BA%BA/title/\nDisallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?\nDisallow: /%EC%B7%A8%EC%97%85/KR/\nDisallow: /%EC%B7%A8%EC%97%85/title/\nDisallow: /empleo/\nDisallow: /emploi/\nDisallow: /emplois/FR/\nDisallow: /emplois/title/\nDisallow: /emprego/\nDisallow: /forum/profile\nDisallow: /g/\nDisallow: /generate-question\nDisallow: /graphql\nDisallow: /imgping\nDisallow: /ita?\nDisallow: /ita/v1/publisher\nDisallow: /ja/clk?\nDisallow: /ja/imp.gif\nDisallow: /job/\nDisallow: /Job/\nDisallow: /jobroll\nDisallow: /jobtrends/trends/log/*\nDisallow: /jwidget.js*\nDisallow: /me/*/pdf\nDisallow: /m/basecamp/analytics.js\nDisallow: /m/jobalerts?\nDisallow: /m/moreLoc\nDisallow: /m/newjobs\nDisallow: /m/recommended\nDisallow: /m/rpc/\nDisallow: /my/\nDisallow: /*oc=1\nDisallow: /pagead/\nDisallow: /poka%C5%BCprac%C4%99?\nDisallow: /preferences\nDisallow: /*radius=\nDisallow: /rc/\nDisallow: /rdr/\nDisallow: /recommendedjobs\nDisallow: /record-page-tags\nDisallow: /resumes/account\nDisallow: /resumes/advanced?\nDisallow: /resumes/alert\nDisallow: /resumes*radius=\nDisallow: /resumes*rb=\nDisallow: /resumes*res=\nDisallow: /resumes/rpc/\nDisallow: /resumes*sort=\nDisallow: /resumes*start=\nDisallow: /rpc/\nDisallow: /r/*/pdf\nDisallow: /rss\nDisallow: /setprefs\nDisallow: /*sid=\nDisallow: /*sp=0\nDisallow: /*&start=\nDisallow: /stc\nDisallow: /Stellen/CH/\nDisallow: /Stellen/title\nDisallow: /tellafriend\nDisallow: /tmn/ccs/\nDisallow: /tos/banner.js\nDisallow: /uie/clk\nDisallow: /url\nDisallow: /v1/alerts\nDisallow: /v1/impressions\nDisallow: /vacature/\nDisallow: /vacature-bekijken\nDisallow: /ver-empleo\nDisallow: /ver-emprego\nDisallow: /ver-oferta\nDisallow: /visajobb\nDisallow: /voir-emploi\nDisallow: /Zeige-Job\nDisallow: /hire/o/ats-partners\nDisallow: /hire/ats-partners\nDisallow: /offers\nDisallow: /c/info/\nDisallow: /hire/how-to-hire/\nDisallow: /hire/interview-questions/\nDisallow: /recruiting/interviewfragen/\nDisallow: /recruiting/stellenbeschreibung/\nDisallow: /recrutement/description-du-poste/\nDisallow: /recrutement/questions-entretien-dembauche/\nDisallow: /personeel/functiebeschrijving/\nDisallow: /companies/\nDisallow: /cmp/\nDisallow: /career-advice/\nDisallow: /conselho-de-carreira/\nDisallow: /hire/job-description/\nDisallow: /karriere-guide/\nDisallow: /conseils-carrieres/\nDisallow: /conseils-carriere/ \nDisallow: /orientacion-profesional/\nDisallow: /porady-zawodowe/\nDisallow: /guida-alla-carriera/\nDisallow: /職涯貼士/\nDisallow: /carrieregids/\nDisallow: /karriarrad/\nDisallow: /orientacion-laboral/ \nDisallow: /career/\nDisallow: /*jobs.html\nDisallow: /jobs\nDisallow: /viewjob\nDisallow: /certifications/\nDisallow: /trabajo\nDisallow: /ofertas\nDisallow: /%D0%B2%D0%B0%D0%BA%D0%B0%D0%BD%D1%81%D0%B8%D0%B8\nDisallow: /emplois\nDisallow: /%E6%B1%82%E4%BA%BA\nDisallow: /m/jobs\nDisallow: /m/viewjob\nDisallow: /vacatures\nDisallow: /offerte-lavoro\nDisallow: /Vi%E1%BB%87c-l%C3%A0m\nDisallow: /Jobs\nDisallow: /%D9%88%D8%B8%D8%A7%D8%A6%D9%81\nDisallow: /%EC%B7%A8%EC%97%85\nDisallow: /Jobb\nDisallow: /empregos\nDisallow: /praca\nDisallow: /%E5%B7%A5%E4%BD%9C\nDisallow: /jobb\nDisallow: /%E0%B8%87%E0%B8%B2%E0%B8%99\nDisallow: /lowongan-kerja\nDisallow: /Stellen\nDisallow: /q-\nDisallow: /l-","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":399,"text":"Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots."},{"line":425,"text":"Rules for Foundational (Training) bots."}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","Googlebot","Google-Extended","ChatGPT-User","Slurp","DuckDuckBot","YahooJP","PetalBot","PerplexityBot","DuckAssistBot","Perplexity-User","YouBot","Claude-User","Gemini-Deep-Research","Claude-SearchBot","OAI-SearchBot","Google-Agent","Google-CloudVertexBot","Meta-WebIndexer","Meta-ExternalFetcher","xAI-Grok","Grok-DeepSearch","Mediapartners-Google","AdsBot-Google","AdsBot-Google-Mobile","adidxbot","Googlebot-Image","msnbot-media","OmniExplorer_Bot","ia_archiver","008","JobdiggerSpider","Cliqzbot","GoogleOther","omgili","omgilibot","Webzio","Webzio-Extended","Scrapy","MyCentralAIScraperBot","facebookexternalhit","WhatsApp","snapchat","TelegramBot","Twitterbot","Slackbot-LinkExpanding","GPTBot","CCBot","anthropic-ai","FacebookBot","AmazonBot","Applebot-Extended","Bytespider","Baiduspider","cohere-training-data-crawler","FriendlyCrawler","img2dataset","Meta-ExternalAgent","ClaudeBot","DeepSeekBot","GrokBot","Diffbot","AI2Bot","PanguBot","Timpibot"],"lines":605},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":151,"lines":[426,605],"groupRanges":[[426,605]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Google-Agent","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":207},"any":{"allowed":true,"line":207},"search":{"allowed":true,"line":207},"cart":{"allowed":true,"line":207},"checkout":{"allowed":true,"line":207},"account":{"allowed":true,"line":207}},"disallows":159,"lines":[186,376],"groupRanges":[[186,376]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":3},"any":{"allowed":true,"line":3},"search":{"allowed":true,"line":3},"cart":{"allowed":true,"line":3},"checkout":{"allowed":true,"line":3},"account":{"allowed":true,"line":3}},"disallows":169,"lines":[2,182],"groupRanges":[[2,182]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"refused","http":403,"url":"https://www.indeed.com/llms.txt"},"ai_txt":{"state":"refused","http":403,"url":"https://www.indeed.com/ai.txt"},"tdmrep":{"state":"refused","http":403,"url":"https://www.indeed.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":43,"first":"2023-01-01","archive_points":42,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[{"domain":"indeed.com","name":"Indeed","category":"local","date":"2026-09","precision":"month","source":"archive","provisional":false,"between":["2026-04-03","2026-09-10"],"summary":"Named ClaudeBot, Meta-ExternalAgent, AI2Bot, Claude-SearchBot, PerplexityBot, DuckAssistBot and 5 more explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20260403031633/https://www.indeed.com/robots.txt","https://web.archive.org/web/20260910002742/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Allow: /"},{"t":"del","s":"Allow: /hire/*?*isid="},{"t":"del","s":"Allow: /personeel/*?*isid="},{"t":"del","s":"Allow: /reclutamiento/*?*isid="},{"t":"del","s":"Allow: /recruiting/*?*isid="},{"t":"del","s":"Allow: /recrutement/*?*isid="},{"t":"add","s":"Allow: /*&start=0&"},{"t":"add","s":"Allow: /*&start=10&"},{"t":"add","s":"Allow: /*&start=20&"},{"t":"add","s":"Allow: /*&start=30&"},{"t":"add","s":"Allow: /*&start=40&"},{"t":"add","s":"Allow: /*&start=50&"},{"t":"add","s":"Allow: /*&start=60&"},{"t":"add","s":"Allow: /*&start=70&"},{"t":"add","s":"Allow: /*&start=80&"},{"t":"add","s":"Allow: /*&start=90&"},{"t":"ctx","s":"Disallow: /*rt=nc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*?rss"},{"t":"add","s":"Disallow: /*&serpstart="},{"t":"add","s":"Disallow: /*/vaclk"},{"t":"ctx","s":"Disallow: /addlLoc/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /%E5%B7%A5%E4%BD%9C/"},{"t":"del","s":"Disallow: /%E5%B7%A5%E4%BD%9C/CN/"},{"t":"del","s":"Disallow: /%E5%B7%A5%E4%BD%9C/title/"},{"t":"ctx","s":"Disallow: /%E6%B1%82%E4%BA%BA/JP/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /ita?"},{"t":"add","s":"Disallow: /ita/v1/publisher"},{"t":"ctx","s":"Disallow: /ja/clk?"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /jobb/"},{"t":"del","s":"Disallow: /jobb/SE/"},{"t":"del","s":"Disallow: /jobb/title/"},{"t":"ctx","s":"Disallow: /jobroll"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /praca/"},{"t":"del","s":"Disallow: /praca/PL/"},{"t":"del","s":"Disallow: /praca/title/"},{"t":"ctx","s":"Disallow: /preferences"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /recommendedjobs"},{"t":"add","s":"Disallow: /record-page-tags"},{"t":"ctx","s":"Disallow: /resumes/account"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Stellen/title"},{"t":"del","s":"Disallow: /survey/*"},{"t":"ctx","s":"Disallow: /tellafriend"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /trabajo/"},{"t":"del","s":"Disallow: /trabajo/AR/"},{"t":"del","s":"Disallow: /trabajo/CL/"},{"t":"del","s":"Disallow: /trabajo/CO/"},{"t":"del","s":"Disallow: /trabajo/MX/"},{"t":"del","s":"Disallow: /trabajo/title/"},{"t":"add","s":"Disallow: /uie/clk"},{"t":"ctx","s":"Disallow: /url"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"gap","s":"… 399 more lines"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2024-11","precision":"month","source":"archive","provisional":false,"between":["2024-09-01","2024-11-01"],"summary":"Named Applebot-Extended, Bytespider, OAI-SearchBot, Amazonbot and ChatGPT-User explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20240901061716/https://www.indeed.com/robots.txt","https://web.archive.org/web/20241101024141/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /cmp/addvideo"},{"t":"add","s":"Disallow: /cmp/Login*"},{"t":"ctx","s":"Disallow: /cmp/*/analytics"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"add","s":"Disallow: /*&start="},{"t":"ctx","s":"Disallow: /stc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Zeige-Job?"},{"t":"add","s":"Disallow: //applystart"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: Googlebot"},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"ctx","s":"Allow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /recrutement/*?*isid="},{"t":"add","s":"Allow: /viewjob?"},{"t":"ctx","s":"Disallow: /*rt=nc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*&serpstart="},{"t":"add","s":"Disallow: /*/vaclk"},{"t":"ctx","s":"Disallow: /addlLoc/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/addvideo"},{"t":"add","s":"Disallow: /cmp/Login*"},{"t":"ctx","s":"Disallow: /cmp/*/analytics"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /graphql"},{"t":"add","s":"Disallow: /hire/o/ats-partners"},{"t":"ctx","s":"Disallow: /imgping"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"add","s":"Disallow: /*&start=100"},{"t":"ctx","s":"Disallow: /stc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /vacatures/title/"},{"t":"add","s":"Disallow: //applystart"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/addvideo"},{"t":"add","s":"Disallow: /cmp/Login*"},{"t":"ctx","s":"Disallow: /cmp/*/analytics"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /graphql"},{"t":"add","s":"Disallow: /hire/o/ats-partners"},{"t":"ctx","s":"Disallow: /imgping"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"add","s":"Disallow: /*&start="},{"t":"ctx","s":"Disallow: /stc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Zeige-Job?"},{"t":"add","s":"Disallow: //applystart"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: GoogleOther"},{"t":"add","s":"Disallow: / "},{"t":"add","s":""},{"t":"gap","s":"… 13 more lines"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2024-02","precision":"month","source":"archive","provisional":false,"between":["2024-01-01","2024-02-01"],"summary":"Named CCBot and anthropic-ai explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20240101000523/https://www.indeed.com/robots.txt","https://web.archive.org/web/20240201054137/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"del","s":"Disallow: /applystart/"},{"t":"add","s":"Disallow: /applystart"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"add","s":"Disallow: /m/moreLoc"},{"t":"ctx","s":"Disallow: /m/newjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"del","s":"Disallow: /applystart/"},{"t":"add","s":"Disallow: /applystart"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"add","s":"Disallow: /m/moreLoc"},{"t":"ctx","s":"Disallow: /m/newjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"del","s":"Disallow: /applystart/"},{"t":"add","s":"Disallow: /applystart"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"add","s":"Disallow: /m/moreLoc"},{"t":"ctx","s":"Disallow: /m/newjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /c/info/"},{"t":"del","s":"Disallow: /hire/how-to-hire/"},{"t":"del","s":"Disallow: /hire/interview-questions/"},{"t":"del","s":"Disallow: /recruiting/interviewfragen/"},{"t":"del","s":"Disallow: /recruiting/stellenbeschreibung/"},{"t":"del","s":"Disallow: /recrutement/description-du-poste/"},{"t":"del","s":"Disallow: /recrutement/questions-entretien-dembauche/"},{"t":"del","s":"Disallow: /personeel/functiebeschrijving/"},{"t":"del","s":"Disallow: /companies/"},{"t":"del","s":"Disallow: /cmp/"},{"t":"del","s":"Disallow: /career-advice/"},{"t":"del","s":"Disallow: /conselho-de-carreira/"},{"t":"del","s":"Disallow: /hire/job-description/"},{"t":"del","s":"Disallow: /karriere-guide/"},{"t":"del","s":"Disallow: /conseils-carrieres/"},{"t":"del","s":"Disallow: /conseils-carriere/ "},{"t":"del","s":"Disallow: /orientacion-profesional/"},{"t":"del","s":"Disallow: /porady-zawodowe/"},{"t":"del","s":"Disallow: /guida-alla-carriera/"},{"t":"del","s":"Disallow: /職涯貼士/"},{"t":"del","s":"Disallow: /carrieregids/"},{"t":"del","s":"Disallow: /karriarrad/"},{"t":"del","s":"Disallow: /orientacion-laboral/ "},{"t":"del","s":"Disallow: /career/"},{"t":"del","s":"Disallow: /*jobs.html"},{"t":"del","s":"Disallow: /jobs"},{"t":"del","s":"Disallow: /viewjob"},{"t":"del","s":"Disallow: /certifications/"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Google-Extended"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"gap","s":"… 1 more lines"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Named Google-Extended explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20231001061852/https://www.indeed.com/robots.txt","https://web.archive.org/web/20231101002911/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"add","s":"Disallow: /applystart/"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"add","s":"Disallow: /applystart/"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"add","s":"Disallow: /applystart/"},{"t":"ctx","s":"Disallow: /cmp/_/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /career/"},{"t":"del","s":"Disallow: jobs.html"},{"t":"add","s":"Disallow: /*jobs.html"},{"t":"ctx","s":"Disallow: /jobs"},{"t":"ctx","s":"Disallow: /viewjob"},{"t":"del","s":"Disallow: /certifications/ "},{"t":"add","s":"Disallow: /certifications/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /c/info/"},{"t":"add","s":"Disallow: /hire/how-to-hire/"},{"t":"add","s":"Disallow: /hire/interview-questions/"},{"t":"add","s":"Disallow: /recruiting/interviewfragen/"},{"t":"add","s":"Disallow: /recruiting/stellenbeschreibung/"},{"t":"add","s":"Disallow: /recrutement/description-du-poste/"},{"t":"add","s":"Disallow: /recrutement/questions-entretien-dembauche/"},{"t":"add","s":"Disallow: /personeel/functiebeschrijving/"},{"t":"add","s":"Disallow: /companies/"},{"t":"add","s":"Disallow: /cmp/"},{"t":"add","s":"Disallow: /career-advice/"},{"t":"add","s":"Disallow: /conselho-de-carreira/"},{"t":"add","s":"Disallow: /hire/job-description/"},{"t":"add","s":"Disallow: /karriere-guide/"},{"t":"add","s":"Disallow: /conseils-carrieres/"},{"t":"add","s":"Disallow: /conseils-carriere/ "},{"t":"add","s":"Disallow: /orientacion-profesional/"},{"t":"add","s":"Disallow: /porady-zawodowe/"},{"t":"add","s":"Disallow: /guida-alla-carriera/"},{"t":"add","s":"Disallow: /職涯貼士/"},{"t":"add","s":"Disallow: /carrieregids/"},{"t":"add","s":"Disallow: /karriarrad/"},{"t":"add","s":"Disallow: /orientacion-laboral/ "},{"t":"add","s":"Disallow: /career/"},{"t":"add","s":"Disallow: /*jobs.html"},{"t":"add","s":"Disallow: /jobs"},{"t":"add","s":"Disallow: /viewjob"},{"t":"add","s":"Disallow: /certifications/"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Named GPTBot explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20230801015342/https://www.indeed.com/robots.txt","https://web.archive.org/web/20230901000836/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Allow: /hire/*?*isid="},{"t":"add","s":"Allow: /personeel/*?*isid="},{"t":"add","s":"Allow: /reclutamiento/*?*isid="},{"t":"add","s":"Allow: /recruiting/*?*isid="},{"t":"add","s":"Allow: /recrutement/*?*isid="},{"t":"ctx","s":"Disallow: /*rt=nc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*?rss"},{"t":"add","s":"Disallow: /addlLoc/"},{"t":"add","s":"Disallow: /ads/"},{"t":"ctx","s":"Disallow: /advanced_search"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/*/write-review"},{"t":"del","s":"Disallow: /community/*"},{"t":"add","s":"Disallow: /community/"},{"t":"ctx","s":"Disallow: /company/*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /setprefs"},{"t":"add","s":"Disallow: /*sid="},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /hire/*?*isid="},{"t":"add","s":"Allow: /personeel/*?*isid="},{"t":"add","s":"Allow: /reclutamiento/*?*isid="},{"t":"add","s":"Allow: /recruiting/*?*isid="},{"t":"add","s":"Allow: /recrutement/*?*isid="},{"t":"ctx","s":"Disallow: /*rt=nc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*?rss"},{"t":"add","s":"Disallow: /*&serpstart="},{"t":"ctx","s":"Disallow: /addlLoc/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/*/write-review"},{"t":"del","s":"Disallow: /community/*"},{"t":"add","s":"Disallow: /community/"},{"t":"ctx","s":"Disallow: /company/*"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /setprefs"},{"t":"add","s":"Disallow: /*sid="},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /hire/*?*isid="},{"t":"add","s":"Allow: /personeel/*?*isid="},{"t":"add","s":"Allow: /reclutamiento/*?*isid="},{"t":"add","s":"Allow: /recruiting/*?*isid="},{"t":"add","s":"Allow: /recrutement/*?*isid="},{"t":"ctx","s":"Disallow: /*rt=nc"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /setprefs"},{"t":"add","s":"Disallow: /*sid="},{"t":"ctx","s":"Disallow: /*sp=0"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: AdsBot-Google"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile"},{"t":"ctx","s":"Allow: /create-resume/lp/"},{"t":"ctx","s":"Allow: /*sid="},{"t":"add","s":"Disallow: /rpc/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /"},{"t":"gap","s":"… 30 more lines"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2023-07","precision":"month","source":"archive","provisional":false,"between":["2023-06-01","2023-07-01"],"summary":"Now allowed: CCBot.","dir":"down","up":[],"down":["CCBot"],"major":true,"signals":false,"status_from":"partial","status_to":"allowed","evidence":["https://web.archive.org/web/20230601000225/https://www.indeed.com/robots.txt","https://web.archive.org/web/20230701040238/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /cmp/*/company-questions"},{"t":"del","s":"Disallow: /cmp/*/jobs/"},{"t":"ctx","s":"Disallow: /cmp/*/people"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Stellen/title"},{"t":"add","s":"Disallow: /survey/*"},{"t":"ctx","s":"Disallow: /tellafriend"},{"t":"add","s":"Disallow: /tmn/ccs/"},{"t":"ctx","s":"Disallow: /tos/banner.js"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/*/company-questions"},{"t":"del","s":"Disallow: /cmp/*/jobs/"},{"t":"ctx","s":"Disallow: /cmp/*/people"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /ofertas/title/"},{"t":"del","s":"Disallow: /offerta-lavoro"},{"t":"ctx","s":"Disallow: /offerte-lavoro/IT/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /pagead/"},{"t":"del","s":"Disallow: /poka%C5%BCprac%C4%99?"},{"t":"ctx","s":"Disallow: /praca/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Stellen/title"},{"t":"add","s":"Disallow: /survey/*"},{"t":"ctx","s":"Disallow: /tellafriend"},{"t":"add","s":"Disallow: /tmn/ccs/"},{"t":"ctx","s":"Disallow: /tos/banner.js"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /vacature/"},{"t":"del","s":"Disallow: /vacature-bekijken?"},{"t":"ctx","s":"Disallow: /vacatures/NL/"},{"t":"ctx","s":"Disallow: /vacatures/title/"},{"t":"del","s":"Disallow: /ver-empleo?"},{"t":"del","s":"Disallow: /ver-emprego?"},{"t":"del","s":"Disallow: /ver-oferta?"},{"t":"del","s":"Disallow: /visajobb?"},{"t":"del","s":"Disallow: /voir-emploi?"},{"t":"del","s":"Disallow: /Zeige-Job?"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /cmp/*/company-questions"},{"t":"del","s":"Disallow: /cmp/*/jobs/"},{"t":"ctx","s":"Disallow: /cmp/*/people"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Stellen/title"},{"t":"add","s":"Disallow: /survey/*"},{"t":"ctx","s":"Disallow: /tellafriend"},{"t":"add","s":"Disallow: /tmn/ccs/"},{"t":"ctx","s":"Disallow: /tos/banner.js"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: CCBot"},{"t":"del","s":"Disallow: /"},{"t":"add","s":"User-agent: facebookexternalhit"},{"t":"add","s":"User-agent: WhatsApp"},{"t":"add","s":"User-agent: snapchat"},{"t":"add","s":"User-agent: TelegramBot"},{"t":"add","s":"User-agent: Twitterbot"},{"t":"add","s":"Allow: /"}]},{"domain":"indeed.com","name":"Indeed","category":"local","date":"2023-03","precision":"month","source":"archive","provisional":false,"between":["2023-02-01","2023-03-01"],"summary":"Now blocked: CCBot.","dir":"up","up":["CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230201100836/https://www.indeed.com/robots.txt","https://web.archive.org/web/20230301002753/https://www.indeed.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /alert"},{"t":"add","s":"Disallow: /api/fetch/mc-anon"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /create-resume/lp/"},{"t":"add","s":"Disallow: /cdn-cgi/"},{"t":"ctx","s":"Disallow: /%E4%BB%95%E4%BA%8B?"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /g/"},{"t":"add","s":"Disallow: /graphql"},{"t":"ctx","s":"Disallow: /imgping"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /me/*/pdf"},{"t":"add","s":"Disallow: /m/basecamp/analytics.js"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /alert"},{"t":"add","s":"Disallow: /api/fetch/mc-anon"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /g/"},{"t":"add","s":"Disallow: /graphql"},{"t":"ctx","s":"Disallow: /imgping"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /me/*/pdf"},{"t":"add","s":"Disallow: /m/basecamp/analytics.js"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /alert"},{"t":"add","s":"Disallow: /api/fetch/mc-anon"},{"t":"ctx","s":"Disallow: /api/getrecjobs"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /g/"},{"t":"add","s":"Disallow: /graphql"},{"t":"ctx","s":"Disallow: /imgping"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /me/*/pdf"},{"t":"add","s":"Disallow: /m/basecamp/analytics.js"},{"t":"ctx","s":"Disallow: /m/jobalerts?"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"}]}]}