The Walled Web Newsletter

All sites / Travel

Airbnb

airbnb.com
Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 5 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot. Google-Extended, CCBot, Meta-ExternalAgent, OAI-SearchBot and 5 more can reach most of it.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

5 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Allowed

0 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI Blocked Named, line 690 Says yes
ClaudeBotAnthropic Blocked Named, line 694 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Allowed Not named; the catch-all rule applies Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Named, line 698 Says yes
CCBotCommon Crawl Allowed Not named; the catch-all rule applies Says yes
Meta-ExternalAgentMeta Allowed Named, line 415 Suggests yes
BytespiderByteDance Allowed Not named; the catch-all rule applies No statement found
MistralAI-TrainingMistral AI Blocked Named, line 710 Says yes
AI2BotAllen Institute for AI Blocked Named, line 706 Does not say
anthropic-aiAnthropic (legacy) · old name Allowed Named, line 547 Unknown
Bots for AI search · Allowed
OAI-SearchBotOpenAI Allowed Named, line 285 Says yes
Claude-SearchBotAnthropic Allowed Not named; the catch-all rule applies Says yes
PerplexityBotPerplexity Allowed Named, line 480 Suggests yes
AmazonbotAmazon Allowed Not named; the catch-all rule applies Says yes
Amzn-SearchBotAmazon Allowed Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Allowed Not named; the catch-all rule applies Says yes
MistralAI-IndexMistral AI Allowed Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Allowed Not named; the catch-all rule applies Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI Allowed Named, line 350 Says not always
Claude-UserAnthropic Allowed Not named; the catch-all rule applies Says yes
Perplexity-UserPerplexity Allowed Not named; the catch-all rule applies Says it mostly ignores it
Google-AgentGoogle Allowed Not named; the catch-all rule applies Says it mostly ignores it
Meta-ExternalFetcherMeta Allowed Not named; the catch-all rule applies Says not always
Amzn-UserAmazon Allowed Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Allowed Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNamed
BingbotRegular search engine, for comparisonAllowedNamed

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

cohere-aiHas rulesline 614Not in our verified list
ImagesiftBotBlockedline 684Not in our verified list
Webzio-ExtendedBlockedline 702Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 780 lines · 409 lines about AI bots Source
##              ///////#             //     //#            //       //#           //         //                           ///             ///                      ///#          //           //                                          ///                      ///#         //     ///     //               //// ///  ///  /// ////   /// ////     /// ////    /// ////#        //   ///   ///   //            //////////  ///  ////////// ///////////  //////////  ///////////#       //   //       //   //          ///     ///  ///  ///        ///      /// ///     /// ///      ///#      //    //       //    //        ///      ///  ///  ///        ///      /// ///     /// ///      ///#     //      //     //      //        ///     ///  ///  ///        ///     ///  ///     /// ///     ///#    //        //   //        //        //////////  ///  ///        //////////   ///     /// //////////#    //         /////         //#    //         /////         //#     //      ///   ///      //#       //////         //////###    Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.#    If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://careers.airbnb.com/positions/?_departments=engineering#    If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us.#    Either way, thanks for stopping by! There's no need to collect your garbage at the end of your stay - we use Javascript for that.#    See you on the next crawl! User-agent: GooglebotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: BingbotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: YandexBotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: BaiduspiderAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places… 283 lines not about AI bots User-agent: OAI-SearchbotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: ChatGPT-UserAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: meta-externalagentAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places User-agent: PerplexityBotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/placesDisallow: /rooms/Disallow: /experiences/ User-agent: anthropic-aiAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/placesDisallow: /rooms/Disallow: /experiences/ User-agent: cohere-aiAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/placesDisallow: /rooms/Disallow: /experiences/Disallow: /help/Disallow: /*/things-to-doDisallow: /*/stays/ User-agent: ImagesiftBotAllow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /  User-agent: GPTBotDisallow: /  User-agent: ClaudeBotDisallow: /  User-agent: Applebot-ExtendedDisallow: /  User-agent: Webzio-ExtendedDisallow: /  User-agent: AI2BotDisallow: /  User-agent: MistralAI-TrainingDisallow: / User-agent: *Allow: /calendar/ical/Allow: /.well-known/amphtml/apikey.pubDisallow: /.well-known/assetlinks.jsonDisallow: /*/skeletonDisallow: /*/sw_skeletonDisallow: /500Disallow: /accountDisallow: /alumniDisallow: /api/v1/trebuchetDisallow: /associates/clickDisallow: /book/Disallow: /calendar/Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribeDisallow: /embeddableDisallow: /experiences/*?*scheduled_idDisallow: /experiences/*?*modalDisallow: /experiences/*/bookDisallow: /external_link?Disallow: /fix-itDisallow: /fixitDisallow: /forgot_passwordDisallow: /google_place_photoDisallow: /api/v2/google_place_photosDisallow: /groupsDisallow: /guidebooksDisallow: /help/feedbackDisallow: /help/searchDisallow: /help/searchDisallow: /home/dashboardDisallow: /inboxDisallow: /login_with_redirectDisallow: /logoutDisallow: /manage-listingDisallow: /messaging/ajax_already_messaged/Disallow: /my_listingsDisallow: /oauth_connectDisallow: /payments/bookDisallow: /reservationDisallow: /rooms/*/amenitiesDisallow: /rooms/*/enhanced-cleaningDisallow: /rooms/*/house-rulesDisallow: /rooms/*/locationDisallow: /rooms/*/photosDisallow: /rooms/*/reviewsDisallow: /rooms/*/safetyDisallow: /rooms/*?viralityEntryPointDisallow: /rooms/*/cancellation-policyDisallow: /rooms/*/descriptionDisallow: /s/guidebooksDisallow: /signed_out_modal.jsonDisallow: /signup_modalDisallow: /storiesDisallow: /trips/upcomingDisallow: /trips/v1/Disallow: /update-your-browserDisallow: /users/*/listingsDisallow: /users/showDisallow: /users/profileDisallow: /api/v3/LandingPageStaysQueryDisallow: /s/*/*Disallow: /things-to-do/places Sitemap: https://www.airbnb.com/sitemap-master-index.xml.gz  … 68 lines not about AI bots

Notes about AI written in the file: "Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest." · "If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://careers.airbnb.com/positions/?_departments=eng"

Everything else we checked

Front doorLet in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.airbnb.com/ as WalledWebBot, from a residential connection in the United States
Anti-bot softwareNo sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIStructured data types: WebSite, SearchAction. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result.
Terms of Servicewww.airbnb.com/terms (found on the homepage)

Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context.

  • Do not scrape, hack, reverse engineer, compromise or impair the Airbnb Platform Do not use bots, crawlers, scrapers, or other automated means to access or collect data or other content from or otherwise interact with the Airbnb Platform.matched "scrape" · reads like a restriction
  • Do not scrape, hack, reverse engineer, compromise or impair the Airbnb Platform Do not use bots, crawlers, scrapers or other automated means to access or collect data or other content from or otherwise interact with the Airbnb Platform.matched "scrape" · reads like a restriction
  • If automated means have been used to review Content, and Airbnb has subsequently taken action with respect to that Content, Airbnb will notify the Member who posted the Content of the use of automated means to the extent that legitimate interests of Airbnb and its Members do not preclude such notice…matched "automated means" · reads like a restriction
  • Do not scrape, hack, reverse engineer, compromise or impair the Airbnb Platform.matched "scrape" · reads like a restriction
  • Airbnb is not acting as an agent in any capacity for any Member, except as specified in the Payments Terms of Service (“ Payment Terms ”).matched "agent" · reads like a restriction
Not foundNo llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags.

History

44 versions on record since 1 Jan 2023

OverallJan 2023: Overall allowedFeb 2023: Overall allowedMar 2023: Overall allowedApr 2023: Overall allowedMay 2023: Overall allowedJun 2023: Overall allowedJul 2023: Overall allowedAug 2023: Overall allowedSep 2023: Overall partialOct 2023: Overall partialNov 2023: Overall partialDec 2023: Overall partialJan 2024: Overall partialFeb 2024: Overall partialMar 2024: Overall allowedApr 2024: Overall allowedMay 2024: Overall allowedJun 2024: Overall allowedJul 2024: Overall allowedAug 2024: Overall allowedSep 2024: Overall allowedOct 2024: Overall allowedNov 2024: Overall allowedDec 2024: Overall allowedJan 2025: Overall allowedFeb 2025: Overall allowedMar 2025: Overall allowedApr 2025: Overall allowedMay 2025: Overall allowedJun 2025: Overall allowedJul 2025: Overall allowedAug 2025: Overall allowedSep 2025: Overall allowedOct 2025: Overall allowedNov 2025: Overall allowedDec 2025: Overall allowedJan 2026: Overall allowedFeb 2026: Overall allowedMar 2026: Overall allowedApr 2026: Overall allowedMay 2026: Overall allowedJun 2026: Overall allowedJul 2026: Overall allowedAug 2026: Overall allowedSep 2026: Overall allowedOct 2026: Overall partialTrainingJan 2023: Training AllowedFeb 2023: Training AllowedMar 2023: Training AllowedApr 2023: Training AllowedMay 2023: Training AllowedJun 2023: Training AllowedJul 2023: Training AllowedAug 2023: Training AllowedSep 2023: Training Some blockedOct 2023: Training Some blockedNov 2023: Training Some blockedDec 2023: Training Some blockedJan 2024: Training Some blockedFeb 2024: Training Some blockedMar 2024: Training AllowedApr 2024: Training AllowedMay 2024: Training AllowedJun 2024: Training AllowedJul 2024: Training AllowedAug 2024: Training AllowedSep 2024: Training AllowedOct 2024: Training AllowedNov 2024: Training AllowedDec 2024: Training AllowedJan 2025: Training AllowedFeb 2025: Training AllowedMar 2025: Training AllowedApr 2025: Training AllowedMay 2025: Training AllowedJun 2025: Training AllowedJul 2025: Training AllowedAug 2025: Training AllowedSep 2025: Training AllowedOct 2025: Training AllowedNov 2025: Training AllowedDec 2025: Training AllowedJan 2026: Training AllowedFeb 2026: Training AllowedMar 2026: Training AllowedApr 2026: Training AllowedMay 2026: Training AllowedJun 2026: Training AllowedJul 2026: Training AllowedAug 2026: Training AllowedSep 2026: Training AllowedOct 2026: Training Some blockedSearchJan 2023: Search AllowedFeb 2023: Search AllowedMar 2023: Search AllowedApr 2023: Search AllowedMay 2023: Search AllowedJun 2023: Search AllowedJul 2023: Search AllowedAug 2023: Search AllowedSep 2023: Search AllowedOct 2023: Search AllowedNov 2023: Search AllowedDec 2023: Search AllowedJan 2024: Search AllowedFeb 2024: Search AllowedMar 2024: Search AllowedApr 2024: Search AllowedMay 2024: Search AllowedJun 2024: Search AllowedJul 2024: Search AllowedAug 2024: Search AllowedSep 2024: Search AllowedOct 2024: Search AllowedNov 2024: Search AllowedDec 2024: Search AllowedJan 2025: Search AllowedFeb 2025: Search AllowedMar 2025: Search AllowedApr 2025: Search AllowedMay 2025: Search AllowedJun 2025: Search AllowedJul 2025: Search AllowedAug 2025: Search AllowedSep 2025: Search AllowedOct 2025: Search AllowedNov 2025: Search AllowedDec 2025: Search AllowedJan 2026: Search AllowedFeb 2026: Search AllowedMar 2026: Search AllowedApr 2026: Search AllowedMay 2026: Search AllowedJun 2026: Search AllowedJul 2026: Search AllowedAug 2026: Search AllowedSep 2026: Search AllowedOct 2026: Search AllowedAssistantsJan 2023: Assistants AllowedFeb 2023: Assistants AllowedMar 2023: Assistants AllowedApr 2023: Assistants AllowedMay 2023: Assistants AllowedJun 2023: Assistants AllowedJul 2023: Assistants AllowedAug 2023: Assistants AllowedSep 2023: Assistants AllowedOct 2023: Assistants AllowedNov 2023: Assistants AllowedDec 2023: Assistants AllowedJan 2024: Assistants AllowedFeb 2024: Assistants AllowedMar 2024: Assistants AllowedApr 2024: Assistants AllowedMay 2024: Assistants AllowedJun 2024: Assistants AllowedJul 2024: Assistants AllowedAug 2024: Assistants AllowedSep 2024: Assistants AllowedOct 2024: Assistants AllowedNov 2024: Assistants AllowedDec 2024: Assistants AllowedJan 2025: Assistants AllowedFeb 2025: Assistants AllowedMar 2025: Assistants AllowedApr 2025: Assistants AllowedMay 2025: Assistants AllowedJun 2025: Assistants AllowedJul 2025: Assistants AllowedAug 2025: Assistants AllowedSep 2025: Assistants AllowedOct 2025: Assistants AllowedNov 2025: Assistants AllowedDec 2025: Assistants AllowedJan 2026: Assistants AllowedFeb 2026: Assistants AllowedMar 2026: Assistants AllowedApr 2026: Assistants AllowedMay 2026: Assistants AllowedJun 2026: Assistants AllowedJul 2026: Assistants AllowedAug 2026: Assistants AllowedSep 2026: Assistants AllowedOct 2026: Assistants AllowedBotsblockedSep 2023: 1 of 24 AI bots blockedOct 2023: 1 of 24 AI bots blockedNov 2023: 1 of 24 AI bots blockedDec 2023: 1 of 24 AI bots blockedJan 2024: 1 of 24 AI bots blockedFeb 2024: 1 of 24 AI bots blockedOct 2026: 5 of 24 AI bots blocked52023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Oct 2026
    More blocking Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
    Found in the Internet Archive's copies: changed between 25 Aug 2026 and 1 Oct 2026. Evidence: before · after
    Show exactly what changed
    Disallow: /google_place_photo
    + Disallow: /api/v2/google_place_photos
    Disallow: /groups
    …
    Disallow: /users/show
    - Disallow: /s/*?
    - Disallow: /s/*/homes
    + Disallow: /users/profile
    + Disallow: /api/v3/LandingPageStaysQuery
    + Disallow: /s/*/*
    Disallow: /things-to-do/places
    …
    Disallow: /google_place_photo
    + Disallow: /api/v2/google_place_photos
    Disallow: /groups
    …
    Disallow: /users/show
    - Disallow: /s/*?
    - Disallow: /s/*/homes
    + Disallow: /users/profile
    + Disallow: /api/v3/LandingPageStaysQuery
    + Disallow: /s/*/*
    Disallow: /things-to-do/places
    …
    Disallow: /google_place_photo
    + Disallow: /api/v2/google_place_photos
    Disallow: /groups
    …
    Disallow: /users/show
    - Disallow: /s/*?
    - Disallow: /s/*/homes
    + Disallow: /users/profile
    + Disallow: /api/v3/LandingPageStaysQuery
    + Disallow: /s/*/*
    Disallow: /things-to-do/places
    …
    Disallow: /google_place_photo
    + Disallow: /api/v2/google_place_photos
    Disallow: /groups
    …
    Disallow: /users/show
    - Disallow: /s/*?
    - Disallow: /s/*/homes
    + Disallow: /users/profile
    + Disallow: /api/v3/LandingPageStaysQuery
    + Disallow: /s/*/*
    Disallow: /things-to-do/places
    …
    Disallow: /google_place_photo
    + Disallow: /api/v2/google_place_photos
    Disallow: /groups
    …
    Disallow: /users/show
    - Disallow: /s/*?
    - Disallow: /s/*/homes
    + Disallow: /users/profile
    + Disallow: /api/v3/LandingPageStaysQuery
    + Disallow: /s/*/*
    Disallow: /things-to-do/places
    …
    … 242 more lines
  • Nov 2024
    Signals Named GPTBot, ClaudeBot, Meta-ExternalAgent, anthropic-ai, OAI-SearchBot, PerplexityBot and 1 more explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Oct 2024 and 1 Nov 2024. Evidence: before · after
    Show exactly what changed
    + #
    # ///////
    …
    # Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.
    - # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://www.airbnb.com/careers/departments/engineering
    + # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://careers.airbnb.com/positions/?_departments=engineering
    # If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us.
    …
    Allow: /.well-known/amphtml/apikey.pub
    + Disallow: /.well-known/assetlinks.json
    + Disallow: /*/skeleton
    + Disallow: /*/sw_skeleton
    + Disallow: /500
    Disallow: /account
    Disallow: /alumni
    - Disallow: /associates/click
    Disallow: /api/v1/trebuchet
    + Disallow: /associates/click
    + Disallow: /book/
    Disallow: /calendar/
    + Disallow: /contact_host
    Disallow: /disaster/lookup
    Disallow: /email/unsubscribe
    + Disallow: /embeddable
    + Disallow: /experiences/*?*scheduled_id
    + Disallow: /experiences/*?*modal
    + Disallow: /experiences/*/book
    + Disallow: /external_link?
    Disallow: /fix-it
    …
    Disallow: /forgot_password
    + Disallow: /google_place_photo
    Disallow: /groups
    - Disallow: /help/search
    + Disallow: /guidebooks
    Disallow: /help/feedback
    + Disallow: /help/search
    + Disallow: /help/search
    Disallow: /home/dashboard
    Disallow: /inbox
    + Disallow: /login_with_redirect
    Disallow: /logout
    …
    Disallow: /my_listings
    - Disallow: /skeleton$
    - Disallow: /skeleton/
    + Disallow: /oauth_connect
    Disallow: /payments/book
    - Disallow: /signup_modal
    + Disallow: /reservation
    + Disallow: /rooms/*/amenities
    + Disallow: /rooms/*/enhanced-cleaning
    + Disallow: /rooms/*/house-rules
    + Disallow: /rooms/*/location
    + Disallow: /rooms/*/photos
    + Disallow: /rooms/*/reviews
    + Disallow: /rooms/*/safety
    + Disallow: /rooms/*?viralityEntryPoint
    + Disallow: /rooms/*/cancellation-policy
    + Disallow: /rooms/*/description
    … 866 more lines
  • Mar 2024
    Less blocking Now allowed: GPTBot.
    Found in the Internet Archive's copies: changed between 1 Feb 2024 and 1 Mar 2024. Evidence: before · after
    Show exactly what changed
    - User-agent: GPTBot
    - Disallow: /
    -
    User-Agent: *
  • Sep 2023
    More blocking Now blocked: GPTBot.
    Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Sep 2023. Evidence: before · after
    Show exactly what changed
    - # ///////
    - # // //
    - # // //
    - # // // /// /// ///
    - # // // /// ///
    - # // /// // //// /// /// /// //// /// //// /// //// /// ////
    - # // /// /// // ////////// /// ////////// /////////// ////////// ///////////
    - # // // // // /// /// /// /// /// /// /// /// /// ///
    - # // // // // /// /// /// /// /// /// /// /// /// ///
    - # // // // // /// /// /// /// /// /// /// /// /// ///
    - # // // // // ////////// /// /// ////////// /// /// //////////
    - # // ///// //
    - # // ///// //
    - # // /// /// //
    - # ////// //////
    + # ///////
    + # // //
    + # // //
    + # // // /// /// ///
    + # // // /// ///
    + # // /// // //// /// /// /// //// /// //// /// //// /// ////
    + # // /// /// // ////////// /// ////////// /////////// ////////// ///////////
    + # // // // // /// /// /// /// /// /// /// /// /// ///
    + # // // // // /// /// /// /// /// /// /// /// /// ///
    + # // // // // /// /// /// /// /// /// /// /// /// ///
    + # // // // // ////////// /// /// ////////// /// /// //////////
    + # // ///// //
    + # // ///// //
    + # // /// /// //
    + # ////// //////
    #
    #
    - # We thought you'd never make it!
    - # We hope you feel right at home in this file...unless you're a disallowed subfolder.
    - # And since you're here, read up on our culture and team: https://www.airbnb.com/careers/departments/engineering
    - # There's even a bring your robot to work day.
    -
    + # Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.
    + # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://www.airbnb.com/careers/departments/engineering
    + # If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us.
    + # Either way, thanks for stopping by! There's no need to collect your garbage at the end of your stay - we use Javascript for that.
    + # See you on the next crawl!
    …
    Disallow: /signed_out_modal.json
    + Disallow: /.well-known/assetlinks.json
    Disallow: /device_id_bev_map
    …
    Disallow: /reservation
    -
    - Disallow: /rooms/*/safety
    - Disallow: /rooms/*/reviews
    - Disallow: /rooms/*/photos
    - Disallow: /rooms/*/location
    - Disallow: /rooms/*/house-rules
    - Disallow: /rooms/*/enhanced-cleaning
    - Disallow: /rooms/*/amenities
    - Disallow: /users/show
    - Disallow: /users/*/listings
    - Disallow: /contact_host
    … 182 more lines

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.