All sites / Local, listings and reviews
Angi
angi.comBlocks some
Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (User-agent: *), which lets them into the site. But the homepage showed our bot, which says who it is, a Generic bot challenge page instead of the site. The written rules say come in; the front door disagrees.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Allowed
0 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Allowed | |||
| GPTBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Allowed | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Allowed | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# -+++++++- .:. # =+++++++= +++++. # .+++++++++. +++++. # :++++.++++: .-=-. # =++++ ++++- # .++++- =+++=. +++=: .=+++++: .=++++=. -+++= +++++ # .++++. -++++: ++++++++++++++- -++++++++++++++ +++++. # -++++ ++++- ++++++-:-+++++= .++++++::=++++++ +++++. # +++++ ++++= +++++: -++++= :+++++ =+++++ +++++. # .:++++= =++++. +++++: -++++= :+++++ -+++++ +++++. # :=+++++++++++==++++: +++++: -++++= :+++++ -+++++ +++++. # =++++++++++++++++++++= +++++: -++++= :+++++ -++++= +++++. # :+++++++=*###*=+++++++++ +++++: -++++= :+++++ -++++= +++++. # :+++++- :**##= =++++++. +++++: -++++= :+++++ -++++= +++++. # +++++. +++*#: :+++++- +++++: -++++= :+++++ =++++= +++++. # .++++= ++++*. -+++++ +++++: -++++= .++++++::=+++++= +++++. # .++++= :++++= +++++. +++++: -++++= -++++++++=++++= +++++. # +++++=:-+++++. =++++: +++++: -++++= .=++++-.:++++= +++++. # .=++++++++++- ..... ..... ..... :++++= ..... # :+++++++=. ::::: -++++- # ... .+++++=..-+++++ # :++++++++++++: # :=++++++=: ## Hello, Home Improvement Hero. Stumbled into the attic instead of the front door? # Here's a thought: Instead of snooping through our crawl space, why not build up your career with us?# Swing by https://www.angi.com/careers - it's the blueprint for your next big project.# Hammer time's over for now, human. # Onward, bots: # Door is Locked for These Bots:User-agent: AhrefsBotDisallow: / User-agent: Screaming Frog SEO SpiderDisallow: / User-agent: SemrushBotDisallow: / # YandexUser-agent: YandexDisallow: / # Everyone Else: User-agent: *Allow: /Disallow: *#! # ParametersDisallow: /*?navigation_impression_id=Disallow: /*?navigation_source_impression_id=Disallow: /*?postalCode=Disallow: /*?redirect=Disallow: /*?showcaseId=Disallow: /*?sku_impression_id=Disallow: /articles/*.htm?page= # DirectoriesDisallow: /companylist/t/Disallow: /companylist/us/xx/Disallow: /write-review/Disallow: /al-web-badge/ # AuthenticationDisallow: /oauth2/ # Sitemaps:Sitemap: https://www.angi.com/sitemap/article-sitemap.xmlSitemap: https://www.angi.com/sitemap/geo-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/nearme-sitemap.xmlSitemap: https://www.angi.com/sitemap/statecat-sitemap.xmlSitemap: https://www.angi.com/sitemap/topic-sitemap.xml
Everything else we checked
| Front door | Human check The homepage showed our bot, which says who it is, a Generic bot challenge page instead of the site (HTTP 403).GET https://www.angi.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Page tags | Homepage not read. |
| Anti-bot software | Served through Cloudflare. The homepage set a Cloudflare Bot Management cookie (__cf_bm), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
41 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Aug 2024Less blocking Now allowed: CCBot.Found in the Internet Archive's copies: changed between 1 Jul 2024 and 1 Aug 2024. Evidence: before · after
Show exactly what changed
- User-agent: CCBot- Disallow: /-User-agent: Screaming Frog SEO Spider…- User-agent: dotbot- Disallow: /-# Everyone Else:…Disallow: /companylist/us/xx/- Disallow: /sitemap-indexDisallow: /write-review/+ # Yandex+ User-agent: Yandex+ Disallow: /+# Sitemaps: - May 2024Less blocking Now allowed: GPTBot.Found in the Internet Archive's copies: changed between 1 Apr 2024 and 1 May 2024. Evidence: before · after
Show exactly what changed
- # robots.txt- User-agent: GPTBot++ # -+++++++- .:.+ # =+++++++= +++++.+ # .+++++++++. +++++.+ # :++++.++++: .-=-.+ # =++++ ++++-+ # .++++- =+++=. +++=: .=+++++: .=++++=. -+++= ++++++ # .++++. -++++: ++++++++++++++- -++++++++++++++ +++++.+ # -++++ ++++- ++++++-:-+++++= .++++++::=++++++ +++++.+ # +++++ ++++= +++++: -++++= :+++++ =+++++ +++++.+ # .:++++= =++++. +++++: -++++= :+++++ -+++++ +++++.+ # :=+++++++++++==++++: +++++: -++++= :+++++ -+++++ +++++.+ # =++++++++++++++++++++= +++++: -++++= :+++++ -++++= +++++.+ # :+++++++=*###*=+++++++++ +++++: -++++= :+++++ -++++= +++++.+ # :+++++- :**##= =++++++. +++++: -++++= :+++++ -++++= +++++.+ # +++++. +++*#: :+++++- +++++: -++++= :+++++ =++++= +++++.+ # .++++= ++++*. -+++++ +++++: -++++= .++++++::=+++++= +++++.+ # .++++= :++++= +++++. +++++: -++++= -++++++++=++++= +++++.+ # +++++=:-+++++. =++++: +++++: -++++= .=++++-.:++++= +++++.+ # .=++++++++++- ..... ..... ..... :++++= .....+ # :+++++++=. ::::: -++++-+ # ... .+++++=..-++++++ # :++++++++++++:+ # :=++++++=:+ #+ # Hello, Home Improvement Hero! Stumbled into the attic instead of the front door?+ # Here's a thought: Instead of snooping through our crawl space, why not build up your career with us?+ # Swing by https://www.angi.com/careers - it's the blueprint for your next big project.+ # Hammer time's over for now, human.+ # Onward, bots:++ # Door is Locked for These Bots:+ User-agent: AhrefsBotDisallow: /…- User-agent: AhrefsBot+ User-agent: SemrushBotDisallow: /…- User-agent: SemrushBot- Disallow: /+ # Everyone Else:User-agent: *- Disallow:+ Allow: /- Sitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xml- Sitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml+ # Parameters+ Disallow: /*?navigation_impression_id=+ Disallow: /*?navigation_source_impression_id=+ Disallow: /*?postalCode=+ Disallow: /*?redirect=+ Disallow: /*?showcaseId=+ Disallow: /*?sku_impression_id=… 24 more lines - Oct 2023More blocking Now blocked: GPTBot and CCBot.Found in the Internet Archive's copies: changed between 1 Sep 2023 and 1 Oct 2023. Evidence: before · after
Show exactly what changed
- ## robots.txt- #- # This file is to prevent the crawling and indexing of certain parts- # of your site by web crawlers and spiders run by sites like Yahoo!- # and Google. By telling these "robots" where not to go on your site,- # you save bandwidth and server resources.- #- # This file will be ignored unless it is at the root of your host:- # Used: http://example.com/robots.txt- # Ignored: http://example.com/site/robots.txt- #- # For more information about the robots.txt standard, see:- # http://www.robotstxt.org/robotstxt.html+ User-agent: GPTBot+ Disallow: /+ User-agent: CCBot+ Disallow: /++ User-agent: Screaming Frog SEO Spider+ Disallow: /++ User-agent: AhrefsBot+ Disallow: /++ User-agent: dotbot+ Disallow: /++ User-agent: SemrushBot+ Disallow: /+User-agent: *+ Disallow:+Sitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml- Sitemap: https://www.angi.com/sitemap/task-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/articles-sitemap-index.xml- Sitemap: https://www.angi.com/sitemap/pages-sitemap.xmlSitemap: https://www.angi.com/sitemap/category-sitemap.xml…Sitemap: https://www.angi.com/sitemap/statecat-sitemap.xml- Sitemap: https://www.angi.com/sitemap/tasknogeo-sitemap.xml- # CSS, JS, Images- Allow: /misc/*.css$- Allow: /misc/*.css?- Allow: /misc/*.js$- Allow: /misc/*.js?- Allow: /misc/*.gif- Allow: /misc/*.jpg- Allow: /misc/*.jpeg- Allow: /misc/*.png- Allow: /modules/*.css$- Allow: /modules/*.css?- Allow: /modules/*.js$- Allow: /modules/*.js?- Allow: /modules/*.gif- Allow: /modules/*.jpg… 73 more lines
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.