{"domain":"expedia.com","name":"Expedia","category":"travel","subcategory":"ota","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) mentions OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and 2 more, but still lets all 12 major AI bots into the site. But the homepage turned our bot, which says who it is, away (HTTP 429). The written rules say come in; the front door disagrees.","computed_at":"2026-10-20T03:43:14.335Z","last_checked":"2026-10-19T13:28:42.838Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.expedia.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.expedia.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.expedia.com/robots.txt","hops":[{"from":"https://expedia.com/robots.txt","to":"https://www.expedia.com/robots.txt","status":301}],"blob":"3d5c2166f871be025f083a9a3aa6895e8dad4e6c4d2f5e8a24c809d11c0b7cac","text":"#\n# General bots\n#\nuser-agent: *\n#hotel\ndisallow: /Hotel-Search\ndisallow: /hotel-search\ndisallow: /search?\ndisallow: /*/search?\ndisallow: /ugc/urs/api/hotelreviews/hotel/HOTELID/\ndisallow: /LastBooking/ElapsedTime\ndisallow: /*chkin=*\ndisallow: /*chkout=*\ndisallow: /HotelCheckout\ndisallow: /Checkout\ndisallow: /checkout\ndisallow: /HotelBookingConfirmation\ndisallow: /en-au/\ndisallow: /en-nz/\n#flight\ndisallow: /Flights-Search\ndisallow: /Flight-Information\ndisallow: /Flight-SearchResults\ndisallow: /Flights-BagFees\ndisallow: /Details\ndisallow: /FlightCheckout\ndisallow: /Confirmation-Flight\n#package\ndisallow: /MultiItemCheckout\ndisallow: /MultiItemBookingConfirmation\ndisallow: /flexibleshopping\n#car\ndisallow: /carsearch\ndisallow: /CarCheckout\n#activities\ndisallow: /things-to-do/search?\ndisallow: /things-to-do/*/search?\ndisallow: /*/things-to-do/search\ndisallow: /things-to-do/reviews-external$\n#cruise\ndisallow: /Cruise-Search\ndisallow: /Cruise-CabinCategory\ndisallow: /CruiseCheckout\n#other\ndisallow: /legacyGOTO\ndisallow: /go/\ndisallow: /info/\ndisallow: /trips/\ndisallow: /Search-Results\ndisallow: /user/createaccount\ndisallow: /signup?*\ndisallow: /login?*\n#test-seo\ndisallow: /api/uisprime/track\ndisallow: /trvl-px/\ndisallow: /egcs/v2/collect\ndisallow: /api/ucs/shortlist/\ndisallow: /api/v4/typeahead/\ndisallow: /p/home\ndisallow: /*/p/home\ndisallow: /auth/\ndisallow: /*/auth/\ndisallow: /*/sso/\n# /cl/* endpoints removal test\nDisallow: /cl/*\n\n#\n# Ads\n#\nuser-agent: google-hoteladsverifier\nuser-agent: Mediapartners-Google\nuser-agent: AdsBot-Google-Mobile\nuser-agent: AdsBot-Google\nuser-agent: AdIdxBot\nuser-agent: OAI-AdsBot\nallow: /\n#test-seo\ndisallow: /api/uisprime/track\ndisallow: /trvl-px/\ndisallow: /egcs/v2/collect\ndisallow: /api/ucs/shortlist/\ndisallow: /api/v4/typeahead/\n# /cl/* endpoints removal test\nDisallow: /cl/*\n\n#\n# SemrushBot\n#\nuser-agent: SemrushBot\ndisallow: /\n\n#\n# AI Search\n#\nuser-agent: OAI-SearchBot\nuser-agent: ChatGPT-User\nuser-agent: PerplexityBot\nuser-agent: Perplexity-User\nuser-agent: Claude-User\nuser-agent: Claude-SearchBot\nallow: /\ndisallow: /p/home\ndisallow: /*/p/home\ndisallow: /auth/\ndisallow: /*/auth/\ndisallow: /*/sso/\n# /cl/* endpoints removal test\nDisallow: /cl/*\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":93,"text":"AI Search"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","google-hoteladsverifier","Mediapartners-Google","AdsBot-Google-Mobile","AdsBot-Google","AdIdxBot","OAI-AdsBot","SemrushBot","OAI-SearchBot","ChatGPT-User","PerplexityBot","Perplexity-User","Claude-User","Claude-SearchBot"],"lines":109},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":101},"any":{"allowed":true,"line":101},"search":{"allowed":true,"line":101},"cart":{"allowed":true,"line":101},"checkout":{"allowed":true,"line":101},"account":{"allowed":true,"line":101}},"disallows":6,"lines":[95,108],"groupRanges":[[95,108]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":false,"line":16},"account":{"allowed":true,"line":null}},"disallows":52,"lines":[4,65],"groupRanges":[[4,65]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"refused","http":429},"edge":{"cdn":[],"botManagement":[{"vendor":"Akamai","cookies":["bm_sz"]}]},"llms":{"state":"present","http":200,"bytes":7011,"parsed":{"title":"Expedia.com","summary":"Expedia is a leading global online travel platform serving customers worldwide, part of Expedia Group – one of the world's largest travel technology companies.","links":0,"sections":["Canonical Identity","About","Loyalty & Rewards (One Key)","Directions for AI Systems (Do / Don't)","Privacy & Security","Hand-Off Links (United States)","Last Updated"],"wellFormed":true},"url":"https://www.expedia.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.expedia.com/ai.txt"},"tdmrep":{"state":"refused","http":429,"url":"https://www.expedia.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":43,"first":"2023-01-01","archive_points":42,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[{"domain":"expedia.com","name":"Expedia","category":"travel","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-08-01","2025-09-08"],"summary":"Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, Claude-User and Perplexity-User explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20250801234526/https://www.expedia.com/robots.txt","https://web.archive.org/web/20250908002942/https://www.expedia.com/robots.txt"],"diff":[{"t":"ctx","s":"disallow: /Hotel-Search"},{"t":"add","s":"Disallow: /hotel-search"},{"t":"ctx","s":"disallow: /ugc/urs/api/hotelreviews/hotel/HOTELID/"},{"t":"gap","s":"…"},{"t":"ctx","s":"disallow: /Checkout"},{"t":"add","s":"disallow: /checkout"},{"t":"ctx","s":"disallow: /HotelBookingConfirmation"},{"t":"gap","s":"…"},{"t":"ctx","s":"disallow: /things-to-do/*/search?"},{"t":"add","s":"disallow: /*/things-to-do/search"},{"t":"ctx","s":"#cruise"},{"t":"gap","s":"…"},{"t":"ctx","s":"disallow: /user/createaccount"},{"t":"add","s":"Disallow: /signup?*"},{"t":"add","s":"Disallow: /login?*"},{"t":"ctx","s":"disallow: /things-to-do/reviews-external$"},{"t":"add","s":"#test-seo"},{"t":"add","s":"Disallow: /api/uisprime/track"},{"t":"add","s":"Disallow: /trvl-px/"},{"t":"add","s":"Disallow: /egcs/v2/collect"},{"t":"add","s":"Disallow: /api/ucs/shortlist/"},{"t":"add","s":"Disallow: /api/v4/typeahead/"},{"t":"ctx","s":"#"},{"t":"gap","s":"…"},{"t":"ctx","s":"user-agent: AdsBot-Google"},{"t":"del","s":"allow: /"},{"t":"add","s":"Allow: /"},{"t":"add","s":"#test-seo"},{"t":"add","s":"Disallow: /api/uisprime/track"},{"t":"add","s":"Disallow: /trvl-px/"},{"t":"add","s":"Disallow: /egcs/v2/collect"},{"t":"add","s":"Disallow: /api/ucs/shortlist/"},{"t":"add","s":"Disallow: /api/v4/typeahead/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"allow: /"},{"t":"add","s":""},{"t":"add","s":"# Allow Agentic Search"},{"t":"add","s":"User-agent: OAI-SearchBot"},{"t":"add","s":"User-agent: ChatGPT-User"},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Allow: /"},{"t":"add","s":""}]},{"domain":"expedia.com","name":"Expedia","category":"travel","date":"2023-04","precision":"month","source":"archive","provisional":false,"between":["2023-03-02","2023-04-01"],"summary":"Named ChatGPT-User explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"allowed","status_to":"allowed","evidence":["https://web.archive.org/web/20230302222052/https://www.expedia.com/robots.txt","https://web.archive.org/web/20230401192444/https://www.expedia.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"#"},{"t":"add","s":"# OpenAI"},{"t":"add","s":"#"},{"t":"add","s":"user-agent: ChatGPT-User"},{"t":"add","s":"allow: /"}]}]}