All sites / Local, listings and reviews
Yelp
yelp.comCharges AI
Yelp says AI agents may use the site only through its paid data licensing program. Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent and 6 more directly. A catch-all rule (User-agent: *) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Blocked
9 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Blocked
8 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Blocked
7 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Blocked | |||
| GPTBotOpenAI | Blocked | Named, line 352 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 352 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 352 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Blocked | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 352 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 352 | Suggests yes |
| BytespiderByteDance | Blocked | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Blocked | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Blocked | Named, line 352 | Unknown |
| Bots for AI search · Blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 336 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 342 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 342 | Suggests yes |
| AmazonbotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Blocked | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Blocked | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 342 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 342 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 342 | Says it mostly ignores it |
| Google-AgentGoogle | Blocked | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 352 | Says not always |
| Amzn-UserAmazon | Blocked | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Blocked | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Named | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| PerplexityUser | Blocked | line 344 | Not in our verified list |
| Claude-Web | Blocked | line 357 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# By accessing Yelp's website (© 2026) you agree to Yelp's Terms of Service, available at# https://www.yelp.com/static?country=US&p=tos## If you would like to inquire about crawling Yelp, please contact us at# https://www.yelp.com/contact## As always, Asimov's Three Laws are in effect:# 1. A robot may not injure a human being or, through inaction, allow a human# being to come to harm.# 2. A robot must obey orders given it by human beings except where such# orders would conflict with the First Law.# 3. A robot must protect its own existence as long as such protection does# not conflict with the First or Second Law.## AI / LLM Crawlers and Agents:# Use of any robot, spider, service search/retrieval application, or other automated device, process or means to access, retrieve,# copy, scrape, or index any portion of the service or any content is prohibited, except as expressly permitted by Yelp.# If you would like to request permission to crawl or use Yelp content, contact us here: https://www.yelp.com/contact User-Agent: Google-InspectionToolUser-Agent: GooglebotUser-Agent: Googlebot-ImageUser-Agent: Googlebot-MobileUser-Agent: Googlebot-VideoAllow: /biz/taste-of-lurgan-lost-springsAllow: /biz_photos/*-albuquerque*Allow: /biz_photos/*-anaheim*Allow: /biz_photos/*-atlanta*Allow: /biz_photos/*-austin*Allow: /biz_photos/*-baltimore*Allow: /biz_photos/*-boston*Allow: /biz_photos/*-brooklyn*Allow: /biz_photos/*-charlotte*Allow: /biz_photos/*-chicago*Allow: /biz_photos/*-cincinnati*Allow: /biz_photos/*-columbus*Allow: /biz_photos/*-dallas*Allow: /biz_photos/*-denver*Allow: /biz_photos/*-fort-worth*Allow: /biz_photos/*-fresno*Allow: /biz_photos/*-honolulu*Allow: /biz_photos/*-houston*Allow: /biz_photos/*-indianapolis*Allow: /biz_photos/*-irvine*Allow: /biz_photos/*-jacksonville*Allow: /biz_photos/*-las-vegas*Allow: /biz_photos/*-long-beach*Allow: /biz_photos/*-los-angeles*Allow: /biz_photos/*-louisville*Allow: /biz_photos/*-manhattan*Allow: /biz_photos/*-miami*Allow: /biz_photos/*-minneapolis*Allow: /biz_photos/*-nashville*Allow: /biz_photos/*-new-orleans*Allow: /biz_photos/*-new-york*Allow: /biz_photos/*-oakland*Allow: /biz_photos/*-oklahoma-city*Allow: /biz_photos/*-orlando*Allow: /biz_photos/*-philadelphia*Allow: /biz_photos/*-phoenix*Allow: /biz_photos/*-pittsburgh*Allow: /biz_photos/*-portland*Allow: /biz_photos/*-raleigh*Allow: /biz_photos/*-richmond*Allow: /biz_photos/*-sacramento*Allow: /biz_photos/*-san-antonio*Allow: /biz_photos/*-san-diego*Allow: /biz_photos/*-san-francisco*Allow: /biz_photos/*-san-jose*Allow: /biz_photos/*-scottsdale*Allow: /biz_photos/*-seattle*Allow: /biz_photos/*-tampa*Allow: /biz_photos/*-tampa-bay*Allow: /biz_photos/*-tucson*Allow: /biz_photos/*-washington*Allow: /biz_photos/*?select_video=Allow: /biz_photos/video_url User-Agent: LinkedInBotUser-Agent: TwitterbotUser-Agent: facebookexternalhitAllow: /article/ User-Agent: AdsBot-GoogleUser-Agent: BingPreviewUser-Agent: Google-InspectionToolUser-Agent: GooglebotUser-Agent: Googlebot-ImageUser-Agent: Googlebot-MobileUser-Agent: Googlebot-VideoUser-Agent: LinkedInBotUser-Agent: Mediapartners-GoogleUser-Agent: STC-botUser-Agent: TwitterbotUser-Agent: Yahoo! SlurpUser-Agent: bingbotUser-Agent: facebookexternalhitDisallow: *US_CENSUS_NAME*Disallow: *US_FEMALE_NAME*Disallow: *US_MALE_NAME*Disallow: /1014943Disallow: /3584794Disallow: /5787254Disallow: /ad_acknowledgmentDisallow: /ad_spiceDisallow: /ad_syndication_user_trackingDisallow: /ad_visibilityDisallow: /adredir?Disallow: /adtrackDisallow: /advertise?Disallow: /answers/*/deleteDisallow: /answers/*/editDisallow: /answers/*/voteDisallow: /biz/*?*&translate=1*Disallow: /biz/*?*&translate=True*Disallow: /biz/*?translate=1*Disallow: /biz/*?translate=True*Disallow: /biz/*destination=*Disallow: /biz/*entry_point=*Disallow: /biz/impression/*Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-0Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-1Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-2Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-3Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-4Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-5Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-6Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-7Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-8Disallow: /biz/outlook-autumn-market-fundamental-catwalk-flimsy-roost-legibility-individualism-grocer-predestination-9Disallow: /biz/taste-of-lurgan-lost-springsDisallow: /biz_attributeDisallow: /biz_linkDisallow: /biz_photosDisallow: /biz_photos/*/logDisallow: /biz_photos/*/log_viewsDisallow: /biz_photos/feedback/Disallow: /biz_redirDisallow: /biz_shareDisallow: /biz_updateDisallow: /bn/Disallow: /bookmark?Disallow: /browse/reviews/recentDisallow: /client_errorsDisallow: /collections/followingDisallow: /collections/userDisallow: /cookie_syncDisallow: /csp_blockDisallow: /csp_report_onlyDisallow: /elite?Disallow: /email_actionsDisallow: /events/*/browseDisallow: /events/*/feedDisallow: /events/addphoto/Disallow: /events/cancel/Disallow: /events/cancel_rsvp/Disallow: /events/change_guest/Disallow: /events/createDisallow: /events/create/Disallow: /events/deletephoto/Disallow: /events/details/Disallow: /events/elite/Disallow: /events/email_notifications/Disallow: /events/event_talk_subscribeDisallow: /events/event_talk_unsubscribeDisallow: /events/export/Disallow: /events/get_media_slice/Disallow: /events/interested/Disallow: /events/load_more_messages/Disallow: /events/photos/Disallow: /events/record_rsvp/Disallow: /events/share/Disallow: /events/similar_snippetDisallow: /events/subscriber_snippetDisallow: /events/talk/Disallow: /events/uncancel/Disallow: /events/updateDisallow: /events/update/Disallow: /events/update_primary_photoDisallow: /events/update_subscriptionDisallow: /events/updatecaptionDisallow: /facebook_connectDisallow: /flag_content/submitDisallow: /flag_content?Disallow: /gamtargetDisallow: /location_suggestDisallow: /mail?Disallow: /menu/*/feedback_formDisallow: /message_the_businessDisallow: /mss/review_feedback/Disallow: /mtb_composerDisallow: /not_recommended_reviewsDisallow: /opportunityDisallow: /photo/Disallow: /possible_biz_ownerDisallow: /proredir?Disallow: /px.gifDisallow: /questions/*/create_answer/Disallow: /questions/*/deleteDisallow: /questions/*/editDisallow: /questions/*/subscribeDisallow: /redir?Disallow: /reservations/*/notifymeDisallow: /review_feed_auto_fetchDisallow: /search_suggestDisallow: /send_to_friendDisallow: /sit_repDisallow: /spiceDisallow: /start_orderDisallow: /syndicateDisallow: /syndication_cookie_syncDisallow: /talk/new_topicDisallow: /thanx?Disallow: /transaction_platform/start_orderDisallow: /user_detailsDisallow: /user_details_answers_givenDisallow: /user_details_bookmarksDisallow: /user_details_friendsDisallow: /user_details_reviews_selfDisallow: /user_details_thanxDisallow: /user_photosDisallow: /weekly/signupDisallow: /writeareview/Disallow: /yuv User-Agent: Google-InspectionToolUser-Agent: GooglebotUser-Agent: Googlebot-ImageUser-Agent: Googlebot-MobileUser-Agent: Googlebot-VideoDisallow: /biz_photos/*-albuquerque*/logDisallow: /biz_photos/*-albuquerque*/log_viewsDisallow: /biz_photos/*-anaheim*/logDisallow: /biz_photos/*-anaheim*/log_viewsDisallow: /biz_photos/*-atlanta*/logDisallow: /biz_photos/*-atlanta*/log_viewsDisallow: /biz_photos/*-austin*/logDisallow: /biz_photos/*-austin*/log_viewsDisallow: /biz_photos/*-baltimore*/logDisallow: /biz_photos/*-baltimore*/log_viewsDisallow: /biz_photos/*-boston*/logDisallow: /biz_photos/*-boston*/log_viewsDisallow: /biz_photos/*-brooklyn*/logDisallow: /biz_photos/*-brooklyn*/log_viewsDisallow: /biz_photos/*-charlotte*/logDisallow: /biz_photos/*-charlotte*/log_viewsDisallow: /biz_photos/*-chicago*/logDisallow: /biz_photos/*-chicago*/log_viewsDisallow: /biz_photos/*-cincinnati*/logDisallow: /biz_photos/*-cincinnati*/log_viewsDisallow: /biz_photos/*-columbus*/logDisallow: /biz_photos/*-columbus*/log_viewsDisallow: /biz_photos/*-dallas*/logDisallow: /biz_photos/*-dallas*/log_viewsDisallow: /biz_photos/*-denver*/logDisallow: /biz_photos/*-denver*/log_viewsDisallow: /biz_photos/*-fort-worth*/logDisallow: /biz_photos/*-fort-worth*/log_viewsDisallow: /biz_photos/*-fresno*/logDisallow: /biz_photos/*-fresno*/log_viewsDisallow: /biz_photos/*-honolulu*/logDisallow: /biz_photos/*-honolulu*/log_viewsDisallow: /biz_photos/*-houston*/logDisallow: /biz_photos/*-houston*/log_viewsDisallow: /biz_photos/*-indianapolis*/logDisallow: /biz_photos/*-indianapolis*/log_viewsDisallow: /biz_photos/*-irvine*/logDisallow: /biz_photos/*-irvine*/log_viewsDisallow: /biz_photos/*-jacksonville*/logDisallow: /biz_photos/*-jacksonville*/log_viewsDisallow: /biz_photos/*-las-vegas*/logDisallow: /biz_photos/*-las-vegas*/log_viewsDisallow: /biz_photos/*-long-beach*/logDisallow: /biz_photos/*-long-beach*/log_viewsDisallow: /biz_photos/*-los-angeles*/logDisallow: /biz_photos/*-los-angeles*/log_viewsDisallow: /biz_photos/*-louisville*/logDisallow: /biz_photos/*-louisville*/log_viewsDisallow: /biz_photos/*-manhattan*/logDisallow: /biz_photos/*-manhattan*/log_viewsDisallow: /biz_photos/*-miami*/logDisallow: /biz_photos/*-miami*/log_viewsDisallow: /biz_photos/*-minneapolis*/logDisallow: /biz_photos/*-minneapolis*/log_viewsDisallow: /biz_photos/*-nashville*/logDisallow: /biz_photos/*-nashville*/log_viewsDisallow: /biz_photos/*-new-orleans*/logDisallow: /biz_photos/*-new-orleans*/log_viewsDisallow: /biz_photos/*-new-york*/logDisallow: /biz_photos/*-new-york*/log_viewsDisallow: /biz_photos/*-oakland*/logDisallow: /biz_photos/*-oakland*/log_viewsDisallow: /biz_photos/*-oklahoma-city*/logDisallow: /biz_photos/*-oklahoma-city*/log_viewsDisallow: /biz_photos/*-orlando*/logDisallow: /biz_photos/*-orlando*/log_viewsDisallow: /biz_photos/*-philadelphia*/logDisallow: /biz_photos/*-philadelphia*/log_viewsDisallow: /biz_photos/*-phoenix*/logDisallow: /biz_photos/*-phoenix*/log_viewsDisallow: /biz_photos/*-pittsburgh*/logDisallow: /biz_photos/*-pittsburgh*/log_viewsDisallow: /biz_photos/*-portland*/logDisallow: /biz_photos/*-portland*/log_viewsDisallow: /biz_photos/*-raleigh*/logDisallow: /biz_photos/*-raleigh*/log_viewsDisallow: /biz_photos/*-richmond*/logDisallow: /biz_photos/*-richmond*/log_viewsDisallow: /biz_photos/*-sacramento*/logDisallow: /biz_photos/*-sacramento*/log_viewsDisallow: /biz_photos/*-san-antonio*/logDisallow: /biz_photos/*-san-antonio*/log_viewsDisallow: /biz_photos/*-san-diego*/logDisallow: /biz_photos/*-san-diego*/log_viewsDisallow: /biz_photos/*-san-francisco*/logDisallow: /biz_photos/*-san-francisco*/log_viewsDisallow: /biz_photos/*-san-jose*/logDisallow: /biz_photos/*-san-jose*/log_viewsDisallow: /biz_photos/*-scottsdale*/logDisallow: /biz_photos/*-scottsdale*/log_viewsDisallow: /biz_photos/*-seattle*/logDisallow: /biz_photos/*-seattle*/log_viewsDisallow: /biz_photos/*-tampa*/logDisallow: /biz_photos/*-tampa*/log_viewsDisallow: /biz_photos/*-tampa-bay*/logDisallow: /biz_photos/*-tampa-bay*/log_viewsDisallow: /biz_photos/*-tucson*/logDisallow: /biz_photos/*-tucson*/log_viewsDisallow: /biz_photos/*-washington*/logDisallow: /biz_photos/*-washington*/log_views User-Agent: Yahoo! SlurpUser-Agent: bingbotDisallow: /search*start=… 334 lines not about AI bots User-Agent: OAI-SearchBotAllow: /spk/Allow: /article$Allow: /article/Disallow: / User-Agent: ChatGPT-UserUser-Agent: Claude-UserUser-Agent: PerplexityUserUser-Agent: Perplexity-UserUser-Agent: Claude-SearchBotUser-Agent: PerplexityBotAllow: /article$Allow: /article/Disallow: / User-Agent: GPTBotUser-Agent: ClaudeBotUser-Agent: CCBotUser-Agent: Google-ExtendedUser-Agent: anthropic-aiUser-Agent: Claude-WebUser-Agent: Meta-ExternalAgentUser-Agent: Meta-ExternalFetcherUser-Agent: RuxitSyntheticDisallow: / User-Agent: *Disallow: / … 3 lines not about AI bots
Notes about AI written in the file: "AI / LLM Crawlers and Agents:" · "copy, scrape, or index any portion of the service or any content is prohibited, except as expressly permitted by Yelp."
Everything else we checked
| Front door | We did not visit: the rules ask our bot to stay out. |
|---|---|
| llms.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| ai.txt | Disallowed for AgentAccessIndexBot by robots.txt |
| TDMRep | Disallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report) |
| Licensing | Yelp says AI agents may use the site only through its paid data licensing program. TechCrunch, 6 Oct 2026 |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Fastly.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No Content-Signal or Content-Usage lines. |
History
History starts with our first check
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
One version on record so far (19 Oct 2026).
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.