{"domain":"glassdoor.com","name":"Glassdoor","category":"local","subcategory":"jobs","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 7 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Meta-ExternalAgent, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 3 more can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.089Z","last_checked":"2026-10-19T13:33:29.985Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.glassdoor.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.glassdoor.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.glassdoor.com/robots.txt","hops":[{"from":"https://glassdoor.com/robots.txt","to":"https://www.glassdoor.com/robots.txt","status":301}],"blob":"965b3496a766a4c5a44566188304e0f43fe39d033088c07ee36d10f7dec5d5cf","text":"# Greetings, human beings!,\n#\n# If you're sniffing around this file, and you're not a robot, we're looking to meet curious folks such as yourself.\n#\n# Think you have what it takes to join the best white-hat SEO growth hackers on the planet, and help improve the way people everywhere find jobs?\n#\n# Run - don't crawl - to apply to join Glassdoor's SEO team here http://jobs.glassdoor.com\n#\n#\n#### Rules for ANY User-Agent\nUser-agent: *\nDisallow: /*?*hostSite=*\nDisallow: /1347171559/\nDisallow: /about/board/\nDisallow: /about/contact/\nDisallow: /about/faq/\nDisallow: /about/forCareerCenters/\nDisallow: /about/forLibraries/\nDisallow: /about/forStudents/\nDisallow: /about/guidelines/\nDisallow: /about/index/\nDisallow: /about/jobs/\nDisallow: /about/learn/\nDisallow: /about/overview/\nDisallow: /about/privacy/\nDisallow: /about/syndicationCenter/\nDisallow: /about/team/\nDisallow: /about/terms/\nDisallow: /about/widgetTerms/\nDisallow: /ajax/\nDisallow: /abtest\nDisallow: /browse/\nDisallow: /chat\nDisallow: /Compare/choose\nDisallow: /employerinfo/\nDisallow: /employerInfo/\nDisallow: /Explore/browse-companies\nDisallow: /home/\nDisallow: /integrations/facebook/glassdoor/eep\nDisallow: /jobview/\nDisallow: /legal/\nDisallow: /lists/\nDisallow: /more/\nDisallow: /partner/\nDisallow: /partner-center/\nDisallow: /partners/company/\nDisallow: /partners/insights/\nDisallow: /partners/jobs/\nDisallow: /partners/reports/\nDisallow: /partners/resumeView\nDisallow: /partners/settings/\nDisallow: /parts\nDisallow: /Polls\nDisallow: /profile/\nAllow:    /profile/login_input.htm\nAllow:    /profile/joinNow_input.htm\nAllow:    /member/profile/login\nAllow:    /member/profile/joinNow\nDisallow: /Resume/user-profile/\nDisallow: /rss/*\nDisallow: /search/\nDisallow: /Search/\nDisallow: /survey/\nDisallow: /surveys\nDisallow: /surveys/\nDisallow: /util/\nDisallow: /getAdSlotContentsAjax.htm\nDisallow: /developer/index.htm\nDisallow: /developer/widget/builder/\nDisallow: /event-ingestion\n\nDisallow: /employers/ec\nDisallow: /employers/enhanced/billing_*\nDisallow: /slink.htm\nDisallow: /*encryptedUserId\nDisallow: /*followId\nDisallow: /*userValidationKey\nDisallow: */trackClickAsync.htm\n\nDisallow: /mz-survey/\nDisallow: /user-activation/\nDisallow: /member/\nDisallow: /resume/build/\nDisallow: /userprofile/\nDisallow: /sourcing$\nDisallow: /knowyourworth/\nDisallow: /Reviews/index.htm?\nDisallow: *filter.searchKeyword=\n\n#logging related\nDisallow: */lib$\nDisallow: */lib/\nDisallow: */globalize/\nDisallow: */globalize$\nDisallow: */ASCIISumThreshold$\nDisallow: */LogClient$\nDisallow: */MsgBuilder$\nDisallow: */UserAgent$\nDisallow: */Constants$\nDisallow: */init/\nDisallow: */init$\nDisallow: */LogServer$\nDisallow: */GDLogger$\nDisallow: */gd-perf$\nDisallow: */gd-site-hdr-dropdown$\nDisallow: */bundles$\nDisallow: */wait$\nDisallow: */extend$\nDisallow: */strings$\nDisallow: */strings/\nDisallow: */document$\nDisallow: */*Ajax.htm\nDisallow: */json$\nDisallow: */json/\n\n# Blocking track urls (ACQ-2468)\nDisallow: /track\n\n#Blocking non standard job view and job search URLs, and paginated job SERP URLs (TRFC-2831)\n\nDisallow: /job-listing/*_IE*.htm\nDisallow: /job-listing/JV.htm?*\nDisallow: /Job/*_IP*\n\nDisallow: /Job/bwl.htm\n\n# TRFC-3125 Block 'sex jobs' jobs infosite pages from being indexed\nDisallow: /Jobs/*-sex-*Jobs-EI*\n\n# TRFC-4037 Block page from being indexed\nDisallow: /Reviews/Barbizon-scam-*\n\n# Block Pages with misaligned search intent, impacting site links SERP\nDisallow: /Jobs/Glassdoor-Jobs-E100431.htm\nDisallow: /Reviews/Glassdoor-Reviews-E100431.htm\nDisallow: /Salary/Glassdoor-Salaries-E100431.htm\nDisallow: /Reviews/EMPLOYERS-Reviews-E25702.htm\nDisallow: /Reviews/Anonymous-Reviews-E5182641.htm\nDisallow: /Salaries/search-salary-SRCH_KO0,6.htm\n\n#Blocking low-traffic employer profile pages\nDisallow: /Photos/\nDisallow: /Location/All-*-Office-Locations*\n\n# Block pages with low SEO potential to reduce crawling\nDisallow: /Compare/-vs-\nDisallow: /Salary/_DEPT*\nDisallow: /Interview/_IL\n\n# Block specific query filter parameters\nDisallow: *filter.searchCategory\nDisallow: *filter.overallRating\nDisallow: *filter.employmentStatus\nDisallow: *sort\nDisallow: *experienceLevel\nDisallow: *locationId\nDisallow: *filter.experience\nDisallow: *filter.jobFunction\nDisallow: *filter.location\n\n# Blocking pagination on employer infosite TR-12\nDisallow: /Jobs/*_P*.htm*\nDisallow: /Jobs/*_IP*.htm*\nDisallow: /Reviews/*_P*.htm*\nDisallow: /Reviews/*_IP*.htm*\nAllow: /Reviews/*-reviews-SRCH_*_IP2.htm*\nDisallow: /Interview/*_P*.htm*\nDisallow: /Interview/*_IP*.htm*\nDisallow: /Benefits/*_IP*.htm*\nDisallow: /Salaries/*_IP*.htm*\nAllow: /Salaries/*_IP2.htm*\nAllow: /Salaries/*_IP3.htm*\nAllow: /Salaries/*_IP4.htm*\nAllow: /Salaries/*_IP5.htm*\n\n# Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)\nDisallow: /1060761/*\n\n# Indeed callback only\nDisallow: /job-listing/indeedApplyCallback.htm\n\n# API Endpoints\nDisallow: /graph\nDisallow: /api-web/\nDisallow: /api/\nDisallow: /employers/engagement/api/\n\n#Analytics\nDisallow: /geb\nDisallow: /brand-views\n\n#Typeaheads\nDisallow: /autocomplete\nDisallow: /searchsuggest\nDisallow: /findCityAjax.htm\n\n#### END Rules for ANY User-Agent\n\nUser-agent: ia_archiver\nDisallow: /\nAllow: */index.htm\n\nUser-Agent: OmniExplorer_Bot\nDisallow: /\n\nUser-agent: Mediapartners-Google\nAllow: /\n\nUser-agent: Baiduspider\nDisallow: /\nAllow: */index.htm\n\n####Partially Block High-Quality AI/LLM Bots\nUser-agent: GPTBot\nUser-agent: Google-Extended\nUser-agent: Amazonbot\nUser-agent: anthropic-ai\nUser-agent: GoogleOther\nUser-agent: Claude-Web\nUser-agent: ClaudeBot\nUser-agent: Perplexity\nUser-agent: Cohere\nUser-agent: cohere-ai\nUser-agent: Applebot-Extended\nUser-agent: Google-CloudVertexBot\nDisallow: /\nAllow: /blog/\nAllow: /Award/\nAllow: /About/\nAllow: /employers/\nDisallow: /employers/ec\nDisallow: /employers/engagement/api/\nDisallow: /employers/enhanced/billing_*\n\n#### Block Low-Quality AI/LLM Bots\nUser-agent: CCBot\nDisallow: /\nUser-agent: Omgilibot\nDisallow: /\nUser-agent: Omgili\nDisallow: /\nUser-agent: FacebookBot\nDisallow: /\nUser-agent: Bytespider\nDisallow: /\nUser-agent: Diffbot\nDisallow: /\nUser-agent: Youbot\nDisallow: /\nUser-agent: FriendlyCrawler\nDisallow: /\nUser-agent: img2dataset\nDisallow: /\n\n#### End AI/LLM Bot Blocks\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":10,"text":"### Rules for ANY User-Agent"},{"line":197,"text":"### END Rules for ANY User-Agent"},{"line":213,"text":"###Partially Block High-Quality AI/LLM Bots"},{"line":235,"text":"### Block Low-Quality AI/LLM Bots"},{"line":255,"text":"### End AI/LLM Bot Blocks"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","ia_archiver","OmniExplorer_Bot","Mediapartners-Google","Baiduspider","GPTBot","Google-Extended","Amazonbot","anthropic-ai","GoogleOther","Claude-Web","ClaudeBot","Perplexity","Cohere","cohere-ai","Applebot-Extended","Google-CloudVertexBot","CCBot","Omgilibot","Omgili","FacebookBot","Bytespider","Diffbot","Youbot","FriendlyCrawler","img2dataset"],"lines":256},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":237},"any":{"allowed":false,"line":237},"search":{"allowed":false,"line":237},"cart":{"allowed":false,"line":237},"checkout":{"allowed":false,"line":237},"account":{"allowed":false,"line":237}},"disallows":1,"lines":[236,237],"groupRanges":[[236,237]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":245},"any":{"allowed":false,"line":245},"search":{"allowed":false,"line":245},"cart":{"allowed":false,"line":245},"checkout":{"allowed":false,"line":245},"account":{"allowed":false,"line":245}},"disallows":1,"lines":[244,245],"groupRanges":[[244,245]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":226},"any":{"allowed":false,"line":226},"search":{"allowed":false,"line":226},"cart":{"allowed":false,"line":226},"checkout":{"allowed":false,"line":226},"account":{"allowed":false,"line":226}},"disallows":4,"lines":[214,233],"groupRanges":[[214,233]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":141,"lines":[11,195],"groupRanges":[[11,195]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":1,"full":7,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"refused","http":403,"url":"https://www.glassdoor.com/llms.txt"},"ai_txt":{"state":"refused","http":403,"url":"https://www.glassdoor.com/ai.txt"},"tdmrep":{"state":"refused","http":403,"url":"https://www.glassdoor.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":42,"first":"2023-01-01","archive_points":41,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"mixed"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":6},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":12},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7}]},"events":[{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2026-10","precision":"month","source":"archive+checker","provisional":true,"between":["2026-05-09","2026-10-19"],"summary":"Now blocked: ClaudeBot.","dir":"up","up":["ClaudeBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260509081437/https://www.glassdoor.com/robots.txt","https://www.glassdoor.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Block Glassdoor jobs. Intent is to remove misleading site links SERP. Details at TRFC-3197"},{"t":"add","s":"# Block Pages with misaligned search intent, impacting site links SERP"},{"t":"ctx","s":"Disallow: /Jobs/Glassdoor-Jobs-E100431.htm"},{"t":"add","s":"Disallow: /Reviews/Glassdoor-Reviews-E100431.htm"},{"t":"add","s":"Disallow: /Salary/Glassdoor-Salaries-E100431.htm"},{"t":"add","s":"Disallow: /Reviews/EMPLOYERS-Reviews-E25702.htm"},{"t":"add","s":"Disallow: /Reviews/Anonymous-Reviews-E5182641.htm"},{"t":"add","s":"Disallow: /Salaries/search-salary-SRCH_KO0,6.htm"},{"t":"ctx","s":""},{"t":"add","s":"#Blocking low-traffic employer profile pages"},{"t":"add","s":"Disallow: /Photos/"},{"t":"add","s":"Disallow: /Location/All-*-Office-Locations*"},{"t":"add","s":""},{"t":"add","s":"# Block pages with low SEO potential to reduce crawling"},{"t":"add","s":"Disallow: /Compare/-vs-"},{"t":"add","s":"Disallow: /Salary/_DEPT*"},{"t":"add","s":"Disallow: /Interview/_IL"},{"t":"add","s":""},{"t":"add","s":"# Block specific query filter parameters"},{"t":"add","s":"Disallow: *filter.searchCategory"},{"t":"add","s":"Disallow: *filter.overallRating"},{"t":"add","s":"Disallow: *filter.employmentStatus"},{"t":"add","s":"Disallow: *sort"},{"t":"add","s":"Disallow: *experienceLevel"},{"t":"add","s":"Disallow: *locationId"},{"t":"add","s":"Disallow: *filter.experience"},{"t":"add","s":"Disallow: *filter.jobFunction"},{"t":"add","s":"Disallow: *filter.location"},{"t":"add","s":""},{"t":"ctx","s":"# Blocking pagination on employer infosite TR-12"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"ctx","s":"User-agent: Perplexity"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /About/"},{"t":"add","s":"Allow: /employers/"},{"t":"add","s":"Disallow: /employers/ec"},{"t":"add","s":"Disallow: /employers/engagement/api/"},{"t":"add","s":"Disallow: /employers/enhanced/billing_*"},{"t":"ctx","s":""}]},{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2025-08","precision":"month","source":"archive","provisional":false,"between":["2025-06-05","2025-08-01"],"summary":"Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.","dir":"down","up":[],"down":["ClaudeBot","PerplexityBot","ChatGPT-User"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250605080148/https://www.glassdoor.com/robots.txt","https://web.archive.org/web/20250801123523/https://www.glassdoor.com/robots.txt"],"diff":[{"t":"ctx","s":"#"},{"t":"del","s":""},{"t":"add","s":"#### Rules for ANY User-Agent"},{"t":"ctx","s":"User-agent: *"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /about/privacy/"},{"t":"del","s":"Disallow: /about/privacy/"},{"t":"ctx","s":"Disallow: /about/syndicationCenter/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /browse/"},{"t":"add","s":"Disallow: /chat"},{"t":"ctx","s":"Disallow: /Compare/choose"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow:    /profile/joinNow_input.htm"},{"t":"add","s":"Allow:    /member/profile/login"},{"t":"add","s":"Allow:    /member/profile/joinNow"},{"t":"ctx","s":"Disallow: /Resume/user-profile/"},{"t":"ctx","s":"Disallow: /rss/*"},{"t":"del","s":"Disallow: /Rungs/"},{"t":"ctx","s":"Disallow: /search/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /surveys"},{"t":"add","s":"Disallow: /surveys/"},{"t":"ctx","s":"Disallow: /util/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /developer/widget/builder/"},{"t":"del","s":"Disallow: /hammer/"},{"t":"del","s":"Disallow: /brand-views"},{"t":"ctx","s":"Disallow: /event-ingestion"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /sourcing$"},{"t":"del","s":"Disallow: /searchsuggest$"},{"t":"ctx","s":"Disallow: /knowyourworth/"},{"t":"del","s":"Disallow: /Scoop/*"},{"t":"del","s":"Allow: /Scoop"},{"t":"del","s":"Allow: /Scoop/index.htm"},{"t":"ctx","s":"Disallow: /Reviews/index.htm?"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"Disallow: /geb/*"},{"t":"add","s":"# Indeed callback only"},{"t":"add","s":"Disallow: /job-listing/indeedApplyCallback.htm"},{"t":"add","s":""},{"t":"add","s":"# API Endpoints"},{"t":"add","s":"Disallow: /graph"},{"t":"add","s":"Disallow: /api-web/"},{"t":"add","s":"Disallow: /api/"},{"t":"add","s":"Disallow: /employers/engagement/api/"},{"t":"add","s":""},{"t":"add","s":"#Analytics"},{"t":"add","s":"Disallow: /geb"},{"t":"ctx","s":"Disallow: /brand-views"},{"t":"ctx","s":""},{"t":"add","s":"#Typeaheads"},{"t":"add","s":"Disallow: /autocomplete"},{"t":"add","s":"Disallow: /searchsuggest"},{"t":"add","s":"Disallow: /findCityAjax.htm"},{"t":"add","s":""},{"t":"add","s":"#### END Rules for ANY User-Agent"},{"t":"add","s":""},{"t":"gap","s":"… 77 more lines"}]},{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-07-01","2024-08-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended and PerplexityBot.","dir":"up","up":["ClaudeBot","Applebot-Extended","PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240701014547/https://www.glassdoor.com/robots.txt","https://web.archive.org/web/20240801035134/https://www.glassdoor.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Block ChatGPT Bot"},{"t":"add","s":"#### Block AI/LLM Bots"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Youbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FriendlyCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: img2dataset"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"#### End AI/LLM Bot Blocks"},{"t":"add","s":""}]},{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-01-01","2024-03-01"],"summary":"Now blocked: CCBot, Bytespider, anthropic-ai, Amazonbot and ChatGPT-User.","dir":"up","up":["CCBot","Bytespider","anthropic-ai","Amazonbot","ChatGPT-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240101052543/https://www.glassdoor.com/robots.txt","https://web.archive.org/web/20240301070717/https://www.glassdoor.com/robots.txt"],"diff":[{"t":"del","s":"# USA (en_US)"},{"t":"ctx","s":"# Greetings, human beings!,"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"# Block ChatGPT Bot "},{"t":"add","s":"# Block ChatGPT Bot"},{"t":"ctx","s":"User-agent: GPTBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GoogleOther"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-10-01","2023-11-01"],"summary":"Now blocked: Google-Extended.","dir":"up","up":["Google-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20231001014824/https://www.glassdoor.com/robots.txt","https://web.archive.org/web/20231101014034/https://www.glassdoor.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# Block Google Extended bot"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"glassdoor.com","name":"Glassdoor","category":"local","date":"2023-09","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-09-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801000037/https://www.glassdoor.com/robots.txt","https://web.archive.org/web/20230901003306/https://www.glassdoor.com/robots.txt"],"diff":[{"t":"del","s":"# USA"},{"t":"add","s":"# USA (en_US)"},{"t":"ctx","s":"# Greetings, human beings!,"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /getAdSlotContentsAjax.htm"},{"t":"add","s":"Disallow: /developer/index.htm"},{"t":"ctx","s":"Disallow: /developer/widget/builder/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /Reviews/index.htm?"},{"t":"add","s":"Disallow: *filter.searchKeyword="},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":""},{"t":"ctx","s":"# Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"Disallow: /geb/*"},{"t":"add","s":"Disallow: /brand-views"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: ia_archiver"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"#"},{"t":"del","s":"# Note that this file has the extension '.text' rather than the more-standard '.txt'"},{"t":"del","s":"# to keep it from being pre-compiled as a servlet.  (*.txt files are precompiled, and"},{"t":"del","s":"# there doesn't seem to be a way to turn this off.)"},{"t":"del","s":"#"},{"t":"del","s":""},{"t":"add","s":"# Block ChatGPT Bot "},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""}]}]}