{"domain":"britishairways.com","name":"British Airways","category":"travel","subcategory":"airline","country":"GB","tags":[],"notes":"","status":"allowed","status_label":"Allows AI","rationale":"Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (<code>User-agent: *</code>), which lets them into the site. (We used the Internet Archive's copy from 13 Oct 2026, because the site would not give our bot its robots.txt: no answer within 15 seconds.)","computed_at":"2026-10-20T03:43:14.277Z","last_checked":"2026-10-19T13:46:05.933Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://www.britishairways.com"},"robots":{"via":{"type":"archive","date":"2026-10-13","url":"https://web.archive.org/web/20261013054506/https://www.britishairways.com/robots.txt","blob":"ff31d7e0463c64b0b2073b58dc664061c1d4d70e404a922a4a872d609c34ca46"},"kind":"unreachable","http":null,"error":"TimeoutError","url":"https://www.britishairways.com/robots.txt","hops":[],"blob":"ff31d7e0463c64b0b2073b58dc664061c1d4d70e404a922a4a872d609c34ca46","text":"User-agent: gsa-crawler\nAllow: /*/traveltrade/\nAllow: /*/travel-partner-connect/\n\nUser-agent: *GraphConnectors*\nAllow: /*/traveltrade/\nAllow: /*/travel-partner-connect/\n \n# -----------------------------\n# Disallow rules (general)\n# -----------------------------\nUser-agent: *\nDisallow: /travel-partner-connect/\nDisallow: /travel/pkomatch/\nDisallow: /*reevoo_display\nDisallow: /*?eId=120001&wfpId=custrelreplies&case=\nDisallow: /*.page\nDisallow: /*/bamediacentre/\nDisallow: /*/assets/pdfs\nDisallow: /*/mobiledataupdate\nDisallow: /*/trade/\nDisallow: /*/traveltrade\n\n# -----------------------------\n# Disallow rules (general - added after tech audit 2025)\n# -----------------------------\nUser-agent: *\nDisallow: /content/britishairways-com-content/\nDisallow: /beta/geo.jsp\nDisallow: /travel/ba_vsg18.jsp/cookies-vsg/public/\nDisallow: /travel/ba_vsg18.jsp/gg-m-cont/public/\nDisallow: /travel/ba_vsg18.jsp/prvdet-vsg/public/\nDisallow: /travel/ba_vsg18.jsp/termcn-vsg/public/\nDisallow: /travel/ba18.jsp/2025-q1-bonus-avios/public/en_ie\nDisallow: /travel/cookietoast/public/\nDisallow: /*/test1$\nDisallow: /*/test-pages/\nDisallow: /travel/recentflightsearches/\n\n# -----------------------------\n# Disallow rules (due to insights from log file analysis)\n# -----------------------------\nUser-agent: *\nDisallow: /travel/home/public/*/404.html\n\n# -----------------------------\n# Disallow rules (language subpaths)\n# -----------------------------\nUser-agent: *\nDisallow: /*zh/\nDisallow: /*pt/\nDisallow: /*pl/\nDisallow: /*ja/\nDisallow: /*it/\nDisallow: /*fr/\nDisallow: /*es/\nDisallow: /*en/\nDisallow: /*de/\nDisallow: /*/zh-\nDisallow: /*/ru-\nDisallow: /*/pt-\nDisallow: /*/pl-\nDisallow: /*/ko-\nDisallow: /*/ja-\nDisallow: /*/it-\nDisallow: /*/fr-\nDisallow: /*/es-\nDisallow: /*/en-\nDisallow: /*/de-\n\n# -----------------------------\n# Disallow rules (avoid server overload due to excessive requests for these URLs - instead of blocking bots via Akamai)\n# -----------------------------\nUser-agent: *\nDisallow: /badp/common/baEntryPoint\n\n# -----------------------------\n# Disallow rules (travel subdirectories)\n# -----------------------------\nUser-agent: *\nDisallow: /*/travel/low-price-finder/public/\nDisallow: /*/travel/home/public/en_gb/user\nDisallow: /*/travel/holiday-finder/public/\nDisallow: /*/travel/fx/\nDisallow: /travel/holiday-finder/\nDisallow: /travel/low-price-finder/\nDisallow: /travel/hoteldetailsproxy/public/\n\n# -----------------------------\n# Disallow rules (unsupported hotel-information locales)\n# -----------------------------\nUser-agent: *\nDisallow: /en-kn/destinations/hotel-information/\nDisallow: /en-hr/destinations/hotel-information/\nDisallow: /en-be/destinations/hotel-information/\nDisallow: /es-mx/destinations/hotel-information/\nDisallow: /en-ie/destinations/hotel-information/\nDisallow: /en-ae/destinations/hotel-information/\nDisallow: /en-mx/destinations/hotel-information/\nDisallow: /en-ag/destinations/hotel-information/\nDisallow: /de-gb/destinations/hotel-information/\nDisallow: /en-se/destinations/hotel-information/\nDisallow: /en-il/destinations/hotel-information/\nDisallow: /en-cz/destinations/hotel-information/\nDisallow: /en-hu/destinations/hotel-information/\nDisallow: /en-fi/destinations/hotel-information/\nDisallow: /de-at/destinations/hotel-information/\nDisallow: /en-sg/destinations/hotel-information/\nDisallow: /en-cl/destinations/hotel-information/\nDisallow: /it-gb/destinations/hotel-information/\nDisallow: /zh-gb/destinations/hotel-information/\nDisallow: /fr-gb/destinations/hotel-information/\nDisallow: /en-dk/destinations/hotel-information/\nDisallow: /en-za/destinations/hotel-information/\nDisallow: /en-gr/destinations/hotel-information/\nDisallow: /en-tt/destinations/hotel-information/\nDisallow: /en-no/destinations/hotel-information/\nDisallow: /en-eg/destinations/hotel-information/\nDisallow: /en-bb/destinations/hotel-information/\nDisallow: /es-gb/destinations/hotel-information/\nDisallow: /en-cy/destinations/hotel-information/\nDisallow: /pt-gb/destinations/hotel-information/\nDisallow: /en-ro/destinations/hotel-information/\nDisallow: /it-ch/destinations/hotel-information/\nDisallow: /es-au/destinations/hotel-information/\nDisallow: /en-hk/destinations/hotel-information/\nDisallow: /en-tc/destinations/hotel-information/\nDisallow: /pt-at/destinations/hotel-information/\nDisallow: /en-bm/destinations/hotel-information/\nDisallow: /en-at/destinations/hotel-information/\nDisallow: /en-gd/destinations/hotel-information/\nDisallow: /ja-gb/destinations/hotel-information/\nDisallow: /en-au/destinations/hotel-information/\nDisallow: /en-ch/destinations/hotel-information/\nDisallow: /fr-at/destinations/hotel-information/\nDisallow: /en-tr/destinations/hotel-information/\nDisallow: /fr-ch/destinations/hotel-information/\nDisallow: /en-nl/destinations/hotel-information/\nDisallow: /en-bg/destinations/hotel-information/\nDisallow: /en-nz/destinations/hotel-information/\nDisallow: /en-jm/destinations/hotel-information/\nDisallow: /en-ky/destinations/hotel-information/\nDisallow: /en-bs/destinations/hotel-information/\nDisallow: /zh-cn/destinations/hotel-information/\nDisallow: /fr-au/destinations/hotel-information/\nDisallow: /en-ng/destinations/hotel-information/\nDisallow: /de-ch/destinations/hotel-information/\nDisallow: /es-at/destinations/hotel-information/\nDisallow: /es-cl/destinations/hotel-information/\nDisallow: /en-lc/destinations/hotel-information/\nDisallow: /en-qa/destinations/hotel-information/\nDisallow: /de-au/destinations/hotel-information/\nDisallow: /en-bh/destinations/hotel-information/\nDisallow: /en-cn/destinations/hotel-information/\nDisallow: /en-sa/destinations/hotel-information/\nDisallow: /en-jp/destinations/hotel-information/\nDisallow: /en-gh/destinations/hotel-information/\nDisallow: /en-br/destinations/hotel-information/\nDisallow: /en-kw/destinations/hotel-information/\n\n# -----------------------------\n# Allow rules (travel subdirectories)\n# -----------------------------\nUser-agent: *\nAllow: /travel/holiday-finder/*/en_gb\nAllow: /travel/low-price-finder/*/en_gb\n\n# -----------------------------\n# Allow rules (specific root/folder exceptions)\n# -----------------------------\nUser-agent: *\nAllow: /travel-partner-connect/$\nAllow: /*_resources-*.json\n \n# -----------------------------\n# Allow rules (language subpaths)\n# -----------------------------\nUser-agent: *\nAllow: /*it/it/\nAllow: /*fr/fr/\nAllow: /*es/es/\nAllow: /*en/us/\nAllow: /*en/it/\nAllow: /*en/in/\nAllow: /*en/fr/\nAllow: /*en/es/\nAllow: /*en/de/\nAllow: /*en/ca/\nAllow: /*de/de/\nAllow: /*/zh_\nAllow: /*/ru_\nAllow: /*/pt_\nAllow: /*/pl_\nAllow: /*/ko_\nAllow: /*/ja_\nAllow: /*/it_\nAllow: /*/fr_\nAllow: /*/es_\nAllow: /*/en_\nAllow: /*/de_\n \n# -----------------------------\n# Allow rules (content sections)\n# -----------------------------\nUser-agent: *\nAllow: /*/travel-guide/\nAllow: /*/travel-guide?\nAllow: /*/offers/\nAllow: /*/offers?\nAllow: /*/holidays/\nAllow: /*/holidays?\nAllow: /*/flights/\nAllow: /*/flights?\nAllow: /*/business-travel/\nAllow: /*/business-travel?\nAllow: /*/information/\nAllow: /*/information?\nAllow: /*/executive-club/\nAllow: /*/executive-club$\nAllow: /*/the-british-airways-club?\n \n# -----------------------------\n# Allow rules (hotel information & executive club)\n# -----------------------------\nUser-agent: *\nAllow: /it-it/destinations/hotel-information/\nAllow: /*fr-fr/destinations/hotel-information/\nAllow: /*es-es/destinations/hotel-information/\nAllow: /*en-us/destinations/hotel-information/\nAllow: /*en-it/destinations/hotel-information/\nAllow: /*en-in/destinations/hotel-information/\nAllow: /*en-gb/destinations/hotel-information/\nAllow: /*en-fr/destinations/hotel-information/\nAllow: /*en-es/destinations/hotel-information/\nAllow: /*en-de/destinations/hotel-information/\nAllow: /*en-ca/destinations/hotel-information/\nAllow: /*de-de/destinations/hotel-information/\n\n# -----------------------------\n# Allow rules (new login pages)\n# -----------------------------\nUser-agent: *\nAllow: /nx/b/account/*/account/pre-login/\nAllow: /nx/b/search/*/*/pnr-search/\n\n# ------------------------------\n# Allow rules (applications)\n# ------------------------------\nUser-agent: *\nAllow: /cms/global/ensighten/\nAllow: /etc.clientlibs/\n\n# -----------------------------\n# Allow rules (Nexus JS)\n# -----------------------------\nUser-agent: *\nAllow: /nx*/_next/static/*.js\n\n# -----------------------------\n# Allow rules (Test 050126)\n# -----------------------------\nUser-agent: *\nAllow: /*/destinations/new-york/holidays-in-new-york$\nAllow: /*/destinations/dubai/dubai-holidays$\nAllow: /*/destinations/maldives/holidays-in-maldives$\nAllow: /*/destinations/barbados/holidays-in-barbados$\nAllow: /*/destinations/las-vegas/holidays-in-las-vegas$\nAllow: /*/destinations/mauritius/holidays-in-mauritius$\nAllow: /*/destinations/florida/florida-holidays$\nAllow: /*/destinations/orlando/orlando-holidays$\nAllow: /*/destinations/miami/miami-holidays$\nAllow: /*/destinations/phuket/holidays-in-phuket$\nAllow: /*/destinations/cape-town/cape-town-holidays$\nAllow: /*/destinations/rome/holidays-in-rome$\nAllow: /*/destinations/barcelona/holidays-in-barcelona$\nAllow: /*/destinations/malta/malta-holidays$\nAllow: /*/destinations/tenerife/tenerife-holidays$\nAllow: /*/destinations/malaga/malaga-holidays$\nAllow: /*/destinations/holiday-types/all-inclusive-holidays$\nAllow: /*/destinations/holiday-types/city-breaks$\nAllow: /*/flights-and-holidays/holidays$\nAllow: /*/destinations/malaga/flights-to-malaga$\nAllow: /*/destinations/benidorm/benidorm-holidays$\nAllow: /*/destinations/nice/flights-to-nice$\nAllow: /*/destinations/gran-canaria/flights-to-gran-canaria$\n\nSitemap: https://www.britishairways.com/sitemapindex.xml\nSitemap: https://www.britishairways.com/homepage.xml\nSitemap: https://www.britishairways.com/seo-hotels-sitemap.xml","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["gsa-crawler","*GraphConnectors*","*"],"lines":289},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":117,"lines":[12,22,27,38,43,44,49,69,74,75,80,87,92,159,164,166,171,173,178,200,205,220,225,237,242,244,249,251,256,257,262,285],"groupRanges":[[12,22],[27,38],[43,44],[49,69],[74,75],[80,87],[92,159],[164,166],[171,173],[178,200],[205,220],[225,237],[242,244],[249,251],[256,257],[262,285]]}],"lanes":{"training":{"level":"full","none":0,"full":9,"restricted":0,"n":9},"search":{"level":"full","none":0,"full":8,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"robots.txt unreachable; RFC 9309 says assume everything is disallowed"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"skipped","reason":"robots.txt unreachable; RFC 9309 says assume everything is disallowed"},"ai_txt":{"state":"skipped","reason":"robots.txt unreachable; RFC 9309 says assume everything is disallowed"},"tdmrep":{"state":"skipped","reason":"robots.txt unreachable; RFC 9309 says assume everything is disallowed"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":27,"first":"2023-01-01","archive_points":27,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2026-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0}]},"events":[]}