{"domain":"angi.com","name":"Angi","category":"local","subcategory":"services","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (<code>User-agent: *</code>), which lets them into the site. But the homepage showed our bot, which says who it is, a Generic bot challenge page instead of the site. The written rules say come in; the front door disagrees.","computed_at":"2026-10-20T03:43:14.897Z","last_checked":"2026-10-19T13:32:19.306Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.angi.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.angi.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.angi.com/robots.txt","hops":[{"from":"https://angi.com/robots.txt","to":"https://www.angi.com/robots.txt","status":301}],"blob":"ef531d2ec723865b3375c5dea384c2f4d0d26276f669c689ef9ce3ad34f41b75","text":"#                  -+++++++-                                            .:.                \n#                  =+++++++=                                           +++++.              \n#                 .+++++++++.                                          +++++.              \n#                 :++++.++++:                                          .-=-.               \n#                 =++++ ++++-                                                              \n#                .++++- =+++=.     +++=: .=+++++:     .=++++=. -+++=   +++++               \n#                .++++. -++++:     ++++++++++++++-   -++++++++++++++   +++++.              \n#                -++++   ++++-     ++++++-:-+++++=  .++++++::=++++++   +++++.              \n#                +++++   ++++=     +++++:   -++++=  :+++++    =+++++   +++++.              \n#              .:++++=   =++++.    +++++:   -++++=  :+++++    -+++++   +++++.              \n#          :=+++++++++++==++++:    +++++:   -++++=  :+++++    -+++++   +++++.              \n#        =++++++++++++++++++++=    +++++:   -++++=  :+++++    -++++=   +++++.              \n#      :+++++++=*###*=+++++++++    +++++:   -++++=  :+++++    -++++=   +++++.              \n#     :+++++-  :**##=   =++++++.   +++++:   -++++=  :+++++    -++++=   +++++.              \n#     +++++.   +++*#:    :+++++-   +++++:   -++++=  :+++++    =++++=   +++++.              \n#    .++++=    ++++*.     -+++++   +++++:   -++++=  .++++++::=+++++=   +++++.              \n#    .++++=   :++++=       +++++.  +++++:   -++++=   -++++++++=++++=   +++++.              \n#     +++++=:-+++++.       =++++:  +++++:   -++++=    .=++++-.:++++=   +++++.              \n#     .=++++++++++-         .....  .....     .....            :++++=   .....               \n#       :+++++++=.                                   :::::    -++++-                       \n#          ...                                      .+++++=..-+++++                        \n#                                                    :++++++++++++:                        \n#                                                      :=++++++=:     \n#\n# Hello, Home Improvement Hero. Stumbled into the attic instead of the front door? \n# Here's a thought: Instead of snooping through our crawl space, why not build up your career with us?\n# Swing by https://www.angi.com/careers - it's the blueprint for your next big project.\n# Hammer time's over for now, human. \n# Onward, bots:\n\n# Door is Locked for These Bots:\nUser-agent: AhrefsBot\nDisallow: /\n\nUser-agent: Screaming Frog SEO Spider\nDisallow: /\n\nUser-agent: SemrushBot\nDisallow: /\n\n# Yandex\nUser-agent: Yandex\nDisallow: /\n\n# Everyone Else:\n\nUser-agent: *\nAllow: /\nDisallow: *#!\n\n# Parameters\nDisallow: /*?navigation_impression_id=\nDisallow: /*?navigation_source_impression_id=\nDisallow: /*?postalCode=\nDisallow: /*?redirect=\nDisallow: /*?showcaseId=\nDisallow: /*?sku_impression_id=\nDisallow: /articles/*.htm?page=\n\n# Directories\nDisallow: /companylist/t/\nDisallow: /companylist/us/xx/\nDisallow: /write-review/\nDisallow: /al-web-badge/\n\n# Authentication\nDisallow: /oauth2/\n\n# Sitemaps:\nSitemap: https://www.angi.com/sitemap/article-sitemap.xml\nSitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml\nSitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xml\nSitemap: https://www.angi.com/sitemap/nearme-sitemap.xml\nSitemap: https://www.angi.com/sitemap/statecat-sitemap.xml\nSitemap: https://www.angi.com/sitemap/topic-sitemap.xml","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["AhrefsBot","Screaming Frog SEO Spider","SemrushBot","Yandex","*"],"lines":75},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":48},"any":{"allowed":true,"line":48},"search":{"allowed":true,"line":48},"cart":{"allowed":true,"line":48},"checkout":{"allowed":true,"line":48},"account":{"allowed":true,"line":48}},"disallows":13,"lines":[47,67],"groupRanges":[[47,67]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Generic bot challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"absent","http":404,"url":"https://www.angi.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.angi.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.angi.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":41,"first":"2023-01-01","archive_points":40,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[{"domain":"angi.com","name":"Angi","category":"local","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-07-01","2024-08-01"],"summary":"Now allowed: CCBot.","dir":"down","up":[],"down":["CCBot"],"major":true,"signals":false,"status_from":"partial","status_to":"allowed","evidence":["https://web.archive.org/web/20240701002634/https://www.angi.com/robots.txt","https://web.archive.org/web/20240801144834/https://www.angi.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"User-agent: CCBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Screaming Frog SEO Spider"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: dotbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"# Everyone Else:"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /companylist/us/xx/"},{"t":"del","s":"Disallow: /sitemap-index"},{"t":"ctx","s":"Disallow: /write-review/"},{"t":"ctx","s":""},{"t":"add","s":"# Yandex"},{"t":"add","s":"User-agent: Yandex"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"# Sitemaps:"}]},{"domain":"angi.com","name":"Angi","category":"local","date":"2024-05","precision":"month","source":"archive","provisional":false,"between":["2024-04-01","2024-05-01"],"summary":"Now allowed: GPTBot.","dir":"down","up":[],"down":["GPTBot"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240401065053/https://www.angi.com/robots.txt","https://web.archive.org/web/20240501015144/https://www.angi.com/robots.txt"],"diff":[{"t":"del","s":"# robots.txt"},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":""},{"t":"add","s":"#                  -+++++++-                                            .:.                "},{"t":"add","s":"#                  =+++++++=                                           +++++.              "},{"t":"add","s":"#                 .+++++++++.                                          +++++.              "},{"t":"add","s":"#                 :++++.++++:                                          .-=-.               "},{"t":"add","s":"#                 =++++ ++++-                                                              "},{"t":"add","s":"#                .++++- =+++=.     +++=: .=+++++:     .=++++=. -+++=   +++++               "},{"t":"add","s":"#                .++++. -++++:     ++++++++++++++-   -++++++++++++++   +++++.              "},{"t":"add","s":"#                -++++   ++++-     ++++++-:-+++++=  .++++++::=++++++   +++++.              "},{"t":"add","s":"#                +++++   ++++=     +++++:   -++++=  :+++++    =+++++   +++++.              "},{"t":"add","s":"#              .:++++=   =++++.    +++++:   -++++=  :+++++    -+++++   +++++.              "},{"t":"add","s":"#          :=+++++++++++==++++:    +++++:   -++++=  :+++++    -+++++   +++++.              "},{"t":"add","s":"#        =++++++++++++++++++++=    +++++:   -++++=  :+++++    -++++=   +++++.              "},{"t":"add","s":"#      :+++++++=*###*=+++++++++    +++++:   -++++=  :+++++    -++++=   +++++.              "},{"t":"add","s":"#     :+++++-  :**##=   =++++++.   +++++:   -++++=  :+++++    -++++=   +++++.              "},{"t":"add","s":"#     +++++.   +++*#:    :+++++-   +++++:   -++++=  :+++++    =++++=   +++++.              "},{"t":"add","s":"#    .++++=    ++++*.     -+++++   +++++:   -++++=  .++++++::=+++++=   +++++.              "},{"t":"add","s":"#    .++++=   :++++=       +++++.  +++++:   -++++=   -++++++++=++++=   +++++.              "},{"t":"add","s":"#     +++++=:-+++++.       =++++:  +++++:   -++++=    .=++++-.:++++=   +++++.              "},{"t":"add","s":"#     .=++++++++++-         .....  .....     .....            :++++=   .....               "},{"t":"add","s":"#       :+++++++=.                                   :::::    -++++-                       "},{"t":"add","s":"#          ...                                      .+++++=..-+++++                        "},{"t":"add","s":"#                                                    :++++++++++++:                        "},{"t":"add","s":"#                                                      :=++++++=:     "},{"t":"add","s":"#"},{"t":"add","s":"# Hello, Home Improvement Hero! Stumbled into the attic instead of the front door? "},{"t":"add","s":"# Here's a thought: Instead of snooping through our crawl space, why not build up your career with us?"},{"t":"add","s":"# Swing by https://www.angi.com/careers - it's the blueprint for your next big project."},{"t":"add","s":"# Hammer time's over for now, human. "},{"t":"add","s":"# Onward, bots:"},{"t":"add","s":""},{"t":"add","s":"# Door is Locked for These Bots:"},{"t":"add","s":"User-agent: AhrefsBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: AhrefsBot"},{"t":"add","s":"User-agent: SemrushBot"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: SemrushBot"},{"t":"del","s":"Disallow: /"},{"t":"add","s":"# Everyone Else:"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"del","s":"Disallow:"},{"t":"add","s":"Allow: /"},{"t":"ctx","s":""},{"t":"del","s":"Sitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xml"},{"t":"del","s":"Sitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml"},{"t":"add","s":"# Parameters"},{"t":"add","s":"Disallow: /*?navigation_impression_id="},{"t":"add","s":"Disallow: /*?navigation_source_impression_id="},{"t":"add","s":"Disallow: /*?postalCode="},{"t":"add","s":"Disallow: /*?redirect="},{"t":"add","s":"Disallow: /*?showcaseId="},{"t":"add","s":"Disallow: /*?sku_impression_id="},{"t":"gap","s":"… 24 more lines"}]},{"domain":"angi.com","name":"Angi","category":"local","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-09-01","2023-10-01"],"summary":"Now blocked: GPTBot and CCBot.","dir":"up","up":["GPTBot","CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230901210745/https://www.angi.com/robots.txt","https://web.archive.org/web/20231001001722/https://www.angi.com/robots.txt"],"diff":[{"t":"del","s":"#"},{"t":"ctx","s":"# robots.txt"},{"t":"del","s":"#"},{"t":"del","s":"# This file is to prevent the crawling and indexing of certain parts"},{"t":"del","s":"# of your site by web crawlers and spiders run by sites like Yahoo!"},{"t":"del","s":"# and Google. By telling these \"robots\" where not to go on your site,"},{"t":"del","s":"# you save bandwidth and server resources."},{"t":"del","s":"#"},{"t":"del","s":"# This file will be ignored unless it is at the root of your host:"},{"t":"del","s":"# Used:    http://example.com/robots.txt"},{"t":"del","s":"# Ignored: http://example.com/site/robots.txt"},{"t":"del","s":"#"},{"t":"del","s":"# For more information about the robots.txt standard, see:"},{"t":"del","s":"# http://www.robotstxt.org/robotstxt.html"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Screaming Frog SEO Spider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AhrefsBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: dotbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: SemrushBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: *"},{"t":"add","s":"Disallow:"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xml"},{"t":"ctx","s":"Sitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml"},{"t":"del","s":"Sitemap: https://www.angi.com/sitemap/task-sitemap-index.xml"},{"t":"ctx","s":"Sitemap: https://www.angi.com/sitemap/articles-sitemap-index.xml"},{"t":"del","s":"Sitemap: https://www.angi.com/sitemap/pages-sitemap.xml"},{"t":"ctx","s":"Sitemap: https://www.angi.com/sitemap/category-sitemap.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":"Sitemap: https://www.angi.com/sitemap/statecat-sitemap.xml"},{"t":"del","s":"Sitemap: https://www.angi.com/sitemap/tasknogeo-sitemap.xml"},{"t":"ctx","s":""},{"t":"del","s":"# CSS, JS, Images"},{"t":"del","s":"Allow: /misc/*.css$"},{"t":"del","s":"Allow: /misc/*.css?"},{"t":"del","s":"Allow: /misc/*.js$"},{"t":"del","s":"Allow: /misc/*.js?"},{"t":"del","s":"Allow: /misc/*.gif"},{"t":"del","s":"Allow: /misc/*.jpg"},{"t":"del","s":"Allow: /misc/*.jpeg"},{"t":"del","s":"Allow: /misc/*.png"},{"t":"del","s":"Allow: /modules/*.css$"},{"t":"del","s":"Allow: /modules/*.css?"},{"t":"del","s":"Allow: /modules/*.js$"},{"t":"del","s":"Allow: /modules/*.js?"},{"t":"del","s":"Allow: /modules/*.gif"},{"t":"del","s":"Allow: /modules/*.jpg"},{"t":"gap","s":"… 73 more lines"}]}]}