{"domain":"united.com","name":"United Airlines","category":"travel","subcategory":"airline","country":"US","tags":[],"notes":"","status":"allowed","status_label":"Allows AI","rationale":"Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (<code>User-agent: *</code>), which lets them into the site. (We used the Internet Archive's copy from 2 Oct 2026, because the site would not give our bot its robots.txt: no answer within 15 seconds.)","computed_at":"2026-10-20T03:43:14.252Z","last_checked":"2026-10-19T13:27:30.313Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://united.com"},"robots":{"via":{"type":"archive","date":"2026-10-02","url":"https://web.archive.org/web/20261002032712/https://www.united.com/robots.txt","blob":"23eb228ec691d3c96a6a9dc24d2901ecef97b51eae4429420e773d76cb0ddc02"},"kind":"unreachable","http":null,"error":"TimeoutError","url":"https://www.united.com/robots.txt","hops":[{"from":"https://united.com/robots.txt","to":"https://www.united.com/robots.txt","status":302}],"blob":"23eb228ec691d3c96a6a9dc24d2901ecef97b51eae4429420e773d76cb0ddc02","text":"User-agent: *\r\nDisallow: /CMS/ContinentalDocuments/pdfs/agency/\r\nDisallow: /cms/ContinentalDocuments/pdfs/agency/\r\nDisallow: /CMS/UnitedDocuments/pdfs/agency/\r\nDisallow: /cms/UnitedDocuments/pdfs/agency/\r\nDisallow: /web/en-US/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/en-us/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/en-US/apps/vendors/emailForm.aspx\r\nDisallow: /web/en-us/apps/vendors/emailform.aspx\r\nDisallow: /web/es/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/es/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/es/apps/vendors/emailForm.aspx\r\nDisallow: /web/es/apps/vendors/emailform.aspx\r\nDisallow: /web/ja-JP/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/ja-jp/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/ja-JP/apps/vendors/emailForm.aspx\r\nDisallow: /web/ja-jp/apps/vendors/emailform.aspx\r\nDisallow: /web/de-DE/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/de-de/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/de-DE/apps/vendors/emailForm.aspx\r\nDisallow: /web/de-de/apps/vendors/emailform.aspx\r\nDisallow: /web/zh-CN/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/zh-cn/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/zh-CN/apps/vendors/emailForm.aspx\r\nDisallow: /web/zh-cn/apps/vendors/emailform.aspx\r\nDisallow: /web/fr/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/fr/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/fr/apps/vendors/emailForm.aspx\r\nDisallow: /web/fr/apps/vendors/emailform.aspx\r\nDisallow: /web/pt/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/pt/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/pt/apps/vendors/emailForm.aspx\r\nDisallow: /web/pt/apps/vendors/emailform.aspx\r\nDisallow: /web/ko/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/ko/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/ko/apps/vendors/emailForm.aspx\r\nDisallow: /web/ko/apps/vendors/emailform.aspx\r\nDisallow: /web/zh-HK/apps/booking/flight/fareCalendar.aspx\r\nDisallow: /web/zh-hk/apps/booking/flight/farecalendar.aspx\r\nDisallow: /web/zh-HK/apps/vendors/emailForm.aspx\r\nDisallow: /web/zh-hk/apps/vendors/emailform.aspx\r\nDisallow: /web/en-US/apps/reservation/main.aspx\r\nDisallow: /web/en-US/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/en-us/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/es/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/ja-JP/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/ja-jp/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/de-DE/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/de-de/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/zh-cn/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/zh-CN/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/fr/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/pt/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/ko/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/zh-HK/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/zh-hk/apps/reservation/flight/view/default.aspx\r\nDisallow: /web/en-US/apps/reservation/flight/cancelFlightConfirm.aspx\r\nDisallow: /web/es/apps/reservation/flight/cancelFlightConfirm.aspx\r\nDisallow: /web/fr/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/ja-JP/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/ja-jp/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/de-DE/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/de-de/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/zh-CN/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/zh-cn/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/ko/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/pt/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/zh-HK/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /web/zh-hk/apps/reservation/flight/cancelflightconfirm.aspx\r\nDisallow: /api/*\r\n\r\nUser-agent: HuaweiSymantecSpider\r\nDisallow: /web/ScriptResource.axd\r\nDisallow: /web/en-US/apps/booking/AutoComplete.asmx\r\nDisallow: /web/WebResource.axd\r\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","HuaweiSymantecSpider"],"lines":76},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":69,"lines":[1,70],"groupRanges":[[1,70]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":42,"first":"2023-01-01","archive_points":42,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[]}