All sites / Local, listings and reviews
Glassdoor
glassdoor.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 7 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 3 more can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
6 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
1 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 214 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 214 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 214 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Named, line 214 | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 236 | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 244 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 214 | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Blocked | Named, line 214 | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| GoogleOther | Blocked | line 218 | Documented by its operator |
| Claude-Web | Blocked | line 219 | Not in our verified list |
| Perplexity | Blocked | line 221 | Not in our verified list |
| Cohere | Blocked | line 222 | Not in our verified list |
| cohere-ai | Blocked | line 223 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 225 | Documented by its operator |
| Omgilibot | Blocked | line 238 | Not in our verified list |
| Omgili | Blocked | line 240 | Not in our verified list |
| FacebookBot | Blocked | line 242 | Not in our verified list |
| Diffbot | Blocked | line 246 | Documented by its operator |
| Youbot | Blocked | line 248 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# Greetings, human beings!,## If you're sniffing around this file, and you're not a robot, we're looking to meet curious folks such as yourself.## Think you have what it takes to join the best white-hat SEO growth hackers on the planet, and help improve the way people everywhere find jobs?## Run - don't crawl - to apply to join Glassdoor's SEO team here http://jobs.glassdoor.com###### Rules for ANY User-AgentUser-agent: *Disallow: /*?*hostSite=*Disallow: /1347171559/Disallow: /about/board/Disallow: /about/contact/Disallow: /about/faq/Disallow: /about/forCareerCenters/Disallow: /about/forLibraries/Disallow: /about/forStudents/Disallow: /about/guidelines/Disallow: /about/index/Disallow: /about/jobs/Disallow: /about/learn/Disallow: /about/overview/Disallow: /about/privacy/Disallow: /about/syndicationCenter/Disallow: /about/team/Disallow: /about/terms/Disallow: /about/widgetTerms/Disallow: /ajax/Disallow: /abtestDisallow: /browse/Disallow: /chatDisallow: /Compare/chooseDisallow: /employerinfo/Disallow: /employerInfo/Disallow: /Explore/browse-companiesDisallow: /home/Disallow: /integrations/facebook/glassdoor/eepDisallow: /jobview/Disallow: /legal/Disallow: /lists/Disallow: /more/Disallow: /partner/Disallow: /partner-center/Disallow: /partners/company/Disallow: /partners/insights/Disallow: /partners/jobs/Disallow: /partners/reports/Disallow: /partners/resumeViewDisallow: /partners/settings/Disallow: /partsDisallow: /PollsDisallow: /profile/Allow: /profile/login_input.htmAllow: /profile/joinNow_input.htmAllow: /member/profile/loginAllow: /member/profile/joinNowDisallow: /Resume/user-profile/Disallow: /rss/*Disallow: /search/Disallow: /Search/Disallow: /survey/Disallow: /surveysDisallow: /surveys/Disallow: /util/Disallow: /getAdSlotContentsAjax.htmDisallow: /developer/index.htmDisallow: /developer/widget/builder/Disallow: /event-ingestion Disallow: /employers/ecDisallow: /employers/enhanced/billing_*Disallow: /slink.htmDisallow: /*encryptedUserIdDisallow: /*followIdDisallow: /*userValidationKeyDisallow: */trackClickAsync.htm Disallow: /mz-survey/Disallow: /user-activation/Disallow: /member/Disallow: /resume/build/Disallow: /userprofile/Disallow: /sourcing$Disallow: /knowyourworth/Disallow: /Reviews/index.htm?Disallow: *filter.searchKeyword= #logging relatedDisallow: */lib$Disallow: */lib/Disallow: */globalize/Disallow: */globalize$Disallow: */ASCIISumThreshold$Disallow: */LogClient$Disallow: */MsgBuilder$Disallow: */UserAgent$Disallow: */Constants$Disallow: */init/Disallow: */init$Disallow: */LogServer$Disallow: */GDLogger$Disallow: */gd-perf$Disallow: */gd-site-hdr-dropdown$Disallow: */bundles$Disallow: */wait$Disallow: */extend$Disallow: */strings$Disallow: */strings/Disallow: */document$Disallow: */*Ajax.htmDisallow: */json$Disallow: */json/ # Blocking track urls (ACQ-2468)Disallow: /track #Blocking non standard job view and job search URLs, and paginated job SERP URLs (TRFC-2831) Disallow: /job-listing/*_IE*.htmDisallow: /job-listing/JV.htm?*Disallow: /Job/*_IP* Disallow: /Job/bwl.htm # TRFC-3125 Block 'sex jobs' jobs infosite pages from being indexedDisallow: /Jobs/*-sex-*Jobs-EI* # TRFC-4037 Block page from being indexedDisallow: /Reviews/Barbizon-scam-* # Block Pages with misaligned search intent, impacting site links SERPDisallow: /Jobs/Glassdoor-Jobs-E100431.htmDisallow: /Reviews/Glassdoor-Reviews-E100431.htmDisallow: /Salary/Glassdoor-Salaries-E100431.htmDisallow: /Reviews/EMPLOYERS-Reviews-E25702.htmDisallow: /Reviews/Anonymous-Reviews-E5182641.htmDisallow: /Salaries/search-salary-SRCH_KO0,6.htm #Blocking low-traffic employer profile pagesDisallow: /Photos/Disallow: /Location/All-*-Office-Locations* # Block pages with low SEO potential to reduce crawlingDisallow: /Compare/-vs-Disallow: /Salary/_DEPT*Disallow: /Interview/_IL # Block specific query filter parametersDisallow: *filter.searchCategoryDisallow: *filter.overallRatingDisallow: *filter.employmentStatusDisallow: *sortDisallow: *experienceLevelDisallow: *locationIdDisallow: *filter.experienceDisallow: *filter.jobFunctionDisallow: *filter.location # Blocking pagination on employer infosite TR-12Disallow: /Jobs/*_P*.htm*Disallow: /Jobs/*_IP*.htm*Disallow: /Reviews/*_P*.htm*Disallow: /Reviews/*_IP*.htm*Allow: /Reviews/*-reviews-SRCH_*_IP2.htm*Disallow: /Interview/*_P*.htm*Disallow: /Interview/*_IP*.htm*Disallow: /Benefits/*_IP*.htm*Disallow: /Salaries/*_IP*.htm*Allow: /Salaries/*_IP2.htm*Allow: /Salaries/*_IP3.htm*Allow: /Salaries/*_IP4.htm*Allow: /Salaries/*_IP5.htm* # Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)Disallow: /1060761/* # Indeed callback onlyDisallow: /job-listing/indeedApplyCallback.htm # API EndpointsDisallow: /graphDisallow: /api-web/Disallow: /api/Disallow: /employers/engagement/api/ #AnalyticsDisallow: /gebDisallow: /brand-views #TypeaheadsDisallow: /autocompleteDisallow: /searchsuggestDisallow: /findCityAjax.htm #### END Rules for ANY User-Agent User-agent: ia_archiverDisallow: /Allow: */index.htm User-Agent: OmniExplorer_BotDisallow: / User-agent: Mediapartners-GoogleAllow: / User-agent: BaiduspiderDisallow: /Allow: */index.htm … 212 lines not about AI bots####Partially Block High-Quality AI/LLM BotsUser-agent: GPTBotUser-agent: Google-ExtendedUser-agent: AmazonbotUser-agent: anthropic-aiUser-agent: GoogleOtherUser-agent: Claude-WebUser-agent: ClaudeBotUser-agent: PerplexityUser-agent: CohereUser-agent: cohere-aiUser-agent: Applebot-ExtendedUser-agent: Google-CloudVertexBotDisallow: /Allow: /blog/Allow: /Award/Allow: /About/Allow: /employers/Disallow: /employers/ecDisallow: /employers/engagement/api/Disallow: /employers/enhanced/billing_* #### Block Low-Quality AI/LLM BotsUser-agent: CCBotDisallow: /User-agent: OmgilibotDisallow: /User-agent: OmgiliDisallow: /User-agent: FacebookBotDisallow: /User-agent: BytespiderDisallow: /User-agent: DiffbotDisallow: /User-agent: YoubotDisallow: /User-agent: FriendlyCrawlerDisallow: /User-agent: img2datasetDisallow: / #### End AI/LLM Bot Blocks … 6 lines not about AI bots
Notes about AI written in the file: "### Rules for ANY User-Agent" · "### END Rules for ANY User-Agent" · "###Partially Block High-Quality AI/LLM Bots" · "### Block Low-Quality AI/LLM Bots"
Everything else we checked
| Front door | Human check The homepage showed our bot, which says who it is, a Cloudflare challenge page instead of the site (HTTP 403).GET https://www.glassdoor.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| llms.txt | Refused (HTTP 403)https://www.glassdoor.com/llms.txt |
| ai.txt | Refused (HTTP 403) |
| TDMRep | Refused (HTTP 403)/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
42 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- May to Oct 2026More blocking Now blocked: ClaudeBot.Changed between the Internet Archive's copy from 9 May 2026 and our own check on 19 Oct 2026. Not confirmed yet: seen once so far. Evidence: before · after
Show exactly what changed
- # Block Glassdoor jobs. Intent is to remove misleading site links SERP. Details at TRFC-3197+ # Block Pages with misaligned search intent, impacting site links SERPDisallow: /Jobs/Glassdoor-Jobs-E100431.htm+ Disallow: /Reviews/Glassdoor-Reviews-E100431.htm+ Disallow: /Salary/Glassdoor-Salaries-E100431.htm+ Disallow: /Reviews/EMPLOYERS-Reviews-E25702.htm+ Disallow: /Reviews/Anonymous-Reviews-E5182641.htm+ Disallow: /Salaries/search-salary-SRCH_KO0,6.htm+ #Blocking low-traffic employer profile pages+ Disallow: /Photos/+ Disallow: /Location/All-*-Office-Locations*++ # Block pages with low SEO potential to reduce crawling+ Disallow: /Compare/-vs-+ Disallow: /Salary/_DEPT*+ Disallow: /Interview/_IL++ # Block specific query filter parameters+ Disallow: *filter.searchCategory+ Disallow: *filter.overallRating+ Disallow: *filter.employmentStatus+ Disallow: *sort+ Disallow: *experienceLevel+ Disallow: *locationId+ Disallow: *filter.experience+ Disallow: *filter.jobFunction+ Disallow: *filter.location+# Blocking pagination on employer infosite TR-12…User-agent: Claude-Web+ User-agent: ClaudeBotUser-agent: Perplexity…Allow: /About/+ Allow: /employers/+ Disallow: /employers/ec+ Disallow: /employers/engagement/api/+ Disallow: /employers/enhanced/billing_* - Aug 2025Less blocking Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.Found in the Internet Archive's copies: changed between 5 Jun 2025 and 1 Aug 2025. Evidence: before · after
Show exactly what changed
#-+ #### Rules for ANY User-AgentUser-agent: *…Disallow: /about/privacy/- Disallow: /about/privacy/Disallow: /about/syndicationCenter/…Disallow: /browse/+ Disallow: /chatDisallow: /Compare/choose…Allow: /profile/joinNow_input.htm+ Allow: /member/profile/login+ Allow: /member/profile/joinNowDisallow: /Resume/user-profile/Disallow: /rss/*- Disallow: /Rungs/Disallow: /search/…Disallow: /surveys+ Disallow: /surveys/Disallow: /util/…Disallow: /developer/widget/builder/- Disallow: /hammer/- Disallow: /brand-viewsDisallow: /event-ingestion…Disallow: /sourcing$- Disallow: /searchsuggest$Disallow: /knowyourworth/- Disallow: /Scoop/*- Allow: /Scoop- Allow: /Scoop/index.htmDisallow: /Reviews/index.htm?…- Disallow: /geb/*+ # Indeed callback only+ Disallow: /job-listing/indeedApplyCallback.htm++ # API Endpoints+ Disallow: /graph+ Disallow: /api-web/+ Disallow: /api/+ Disallow: /employers/engagement/api/++ #Analytics+ Disallow: /gebDisallow: /brand-views+ #Typeaheads+ Disallow: /autocomplete+ Disallow: /searchsuggest+ Disallow: /findCityAjax.htm++ #### END Rules for ANY User-Agent+… 77 more lines - Aug 2024More blocking Now blocked: ClaudeBot, Applebot-Extended and PerplexityBot.Found in the Internet Archive's copies: changed between 1 Jul 2024 and 1 Aug 2024. Evidence: before · after
Show exactly what changed
- # Block ChatGPT Bot+ #### Block AI/LLM BotsUser-agent: GPTBot…+ User-agent: Claude-Web+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Youbot+ Disallow: /++ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: img2dataset+ Disallow: /++ #### End AI/LLM Bot Blocks+ - Mar 2024More blocking Now blocked: CCBot, Bytespider, anthropic-ai, Amazonbot and ChatGPT-User.Found in the Internet Archive's copies: changed between 1 Jan 2024 and 1 Mar 2024. Evidence: before · after
Show exactly what changed
- # USA (en_US)# Greetings, human beings!,…- # Block ChatGPT Bot+ # Block ChatGPT BotUser-agent: GPTBot…+ User-agent: CCBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /++ User-agent: Omgili+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: GoogleOther+ Disallow: /++ User-agent: Amazonbot+ Disallow: /+ - Nov 2023
- Sep 2023More blocking Now blocked: GPTBot.Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
Show exactly what changed
- # USA+ # USA (en_US)# Greetings, human beings!,…Disallow: /getAdSlotContentsAjax.htm+ Disallow: /developer/index.htmDisallow: /developer/widget/builder/…Disallow: /Reviews/index.htm?+ Disallow: *filter.searchKeyword=…-# Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)…+ Disallow: /geb/*+ Disallow: /brand-views+User-agent: ia_archiver…- #- # Note that this file has the extension '.text' rather than the more-standard '.txt'- # to keep it from being pre-compiled as a servlet. (*.txt files are precompiled, and- # there doesn't seem to be a way to turn this off.)- #-+ # Block ChatGPT Bot+ User-agent: GPTBot+ Disallow: /
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.