All sites / Local, listings and reviews
Realtor.com
realtor.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 2 of the 24 AI bots we track from the whole site, including Meta-ExternalAgent and Meta-ExternalFetcher. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and 7 more can reach most of it. The homepage also turned our bot, which says who it is, away (HTTP 429).
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
1 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
1 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| ClaudeBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Allowed | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 320 | Suggests yes |
| BytespiderByteDance | Allowed | Not named; the catch-all rule applies | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | Not named; the catch-all rule applies | Says yes |
| Claude-SearchBotAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| PerplexityBotPerplexity | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Allowed | Not named; the catch-all rule applies | Says not always |
| Claude-UserAnthropic | Allowed | Not named; the catch-all rule applies | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 323 | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Named | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| GoogleOther | Blocked | line 314 | Documented by its operator |
| facebookbot | Blocked | line 317 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 329 | Documented by its operator |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
# LEGAL NOTICE: Per https://www.realtor.com's Terms of Service, scraping data from this website is unauthorized without the express written permission from Move Sales, Inc., operator of https://www.realtor.com. User-agent: bingbotCrawl-delay: 1 User-agent: *Allow: /news/*?cid=Allow: /advice/*?cid=Allow: /guides/*?cid=Allow: /living/*?cid=Allow: /realestateandhomes-search/*/price-na-Disallow: /news/draft/Disallow: /advice/draft/Disallow: /guides/draft/Disallow: /8058/Disallow: /addeliverymodule/Disallow: /advice/wp-content/themes/realtor/nobots/Disallow: /amenities?Disallow: /apartments/*?disp=2*Disallow: /apartments/*?view=map/pg-*Disallow: /apartments/*/*-rentalsDisallow: /apartments/*/cat-friendlyDisallow: /apartments/*/dog-friendlyDisallow: /apartments/*/*/pg-*Disallow: /apartments/near-me*Disallow: /contentDisallow: /ContentDisallow: /content/Disallow: /Content/Disallow: /diagnosticsDisallow: /DiagnosticsDisallow: /diagnostics/Disallow: /Diagnostics/Disallow: /foreseeDisallow: /ForeseeDisallow: /foresee/Disallow: /Foresee/Disallow: /home_advisorDisallow: /home_advisor/Disallow: /homes_aroundDisallow: /homemade/wp-content/plugins/Disallow: /homemade/wp-admin/Disallow: /homemade/readme.htmlDisallow: /homemade/refer/Disallow: /international/print/Disallow: /international/us/Disallow: /international/*?sort=distanceDisallow: /international/*?sort=dateDisallow: /*?lang=Disallow: /mapDisallow: /MapDisallow: /map/Disallow: /Map/Disallow: /*?map_layerDisallow: /market-insights/Disallow: /Market-Insights/Disallow: /MARKET-INSIGHTS/Disallow: /median_pricesDisallow: /median_prices?Disallow: /mortgage/home-loan/*?price=Disallow: /mortgage_paymentDisallow: /mortgage/rates/*property_price=Disallow: /moviesDisallow: /MoviesDisallow: /movies/Disallow: /Movies/Disallow: /musicDisallow: /MusicDisallow: /music/Disallow: /Music/Disallow: /myhome/*_MDisallow: /myhome/M*Disallow: /*?ncs=Disallow: /network-analysisDisallow: /newhomesconstruction/*Disallow: /newhomecommunities/near-me*Disallow: /news/feed/*/Disallow: /news/wp-content/themes/realtor/nobots/Disallow: /*not_foundDisallow: /*not-foundDisallow: /*not foundDisallow: /*not foundDisallow: /*not foundDisallow: /*Not_FoundDisallow: /*Not-FoundDisallow: /*Not FoundDisallow: /*Not FoundDisallow: /*Not FoundDisallow: /o/Disallow: /*pgsz=Disallow: /polaris-ldp/*Disallow: /polaris-ldpDisallow: /*?pos=Disallow: /printDisallow: /PrintDisallow: /print/Disallow: /Print/Disallow: /programsDisallow: /ProgramsDisallow: /programs/Disallow: /Programs/Disallow: /properties/topleadformDisallow: /property-overviewDisallow: /propertyrecord-search/*/beds-Disallow: /propertyrecord-search/*/type-Disallow: /propertyrecord-search/*/pg-*/pg-Disallow: /propertyrecord-search/*undefinedDisallow: /publicDisallow: /PublicDisallow: /public/Disallow: /Public/Disallow: /rdc_user_check.jsDisallow: /realestateagents/*agentname-Disallow: /realestateagents/58386a0cf70a250001177f2dDisallow: /realestateandhomes-detail/187-Harness-Downs-Rd_Port-Matilda_PA_16870_M47586-18534Disallow: /realestateandhomes-detail/4581-S-Valdai-Way_Aurora_CO_80015_M21324-97292Disallow: /realestateandhomes-detail/5845-Meadowlane-St_New-Port-Richey_FL_34652_M68583-60477Disallow: /realestateandhomes-detail/1100-Pinehurst-Dr_Chapel-Hill_NC_27517_M64159-81306Disallow: /realestateandhomes-detail/2027-Portsmouth-Dr_Richardson_TX_75082_M83039-98371Disallow: /realestateandhomes-detail/2137-W-Roosevelt-Dr_Milwaukee_WI_53209_M82248-14449Disallow: /realestateandhomes-detail/2137-W-Roosevelt-Dr_Milwaukee_WI_53209_M90548-95731Disallow: /realestateandhomes-detail/650-Church-Rd_Flourtown_PA_19031_M35761-59786Disallow: /realestateandhomes-detail/1132-Jason-Dr_Lompoc_CA_93436_M11599-85143Disallow: /realestateandhomes-detail/M1159985143Disallow: /realestateandhomes-detail/undefinedDisallow: /realestateandhomes-print/Disallow: /realestateandhomes-search/*/*/pg-Disallow: /realestateandhomes-search/*?from_ldp=trueDisallow: /*?cid=Disallow: /realestateandhomes-search/*/pg-*/show-recently-soldDisallow: /realestateandhomes-search/*/radius-Disallow: /realestateandhomes-search/undefinedDisallow: /realestateteam/bright-birch-real-estate__mn_1639730Disallow: /rentals/*?view=map*Disallow: /rentals/api/optimizely.jsDisallow: /restrictedDisallow: /RestrictedDisallow: /restricted/Disallow: /Restricted/Disallow: /*?row=Disallow: /similar_homesDisallow: /soldhomeprices/*/beds-Disallow: /soldhomeprices/*/type-Disallow: /srp_filter/Disallow: /*?view=ndDisallow: /wcfDisallow: /WCFDisallow: /wcf/Disallow: /WCF/Disallow: /webserviceDisallow: /WebServiceDisallow: /webservice/Disallow: /WebService/Disallow: /webusercontrolsDisallow: /WebUserControlsDisallow: /webusercontrols/Disallow: /WebUserControls/Disallow: /widgetsDisallow: /WidgetsDisallow: /widgets/Disallow: /Widgets/Disallow: /YfcuVaVBkfnzl2j6UbEcK0Oiqkv88jx6w/Disallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-43.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-42.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-41.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-40.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-39.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-38.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-37.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-36.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-35.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-34.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-33.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-32.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-31.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-30.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-29.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-28.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-27.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-26.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-us-25.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-17.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-16.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-15.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-14.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-fl-15.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-13.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-fl-14.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-fl-13.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-12.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ca-11.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-fl-12.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-13-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-Disallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-arkansas-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-andrews-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ss-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-state-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tx-10.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-xx-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-oregon-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tx,-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tx-9.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-123-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-94526-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ap-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-arizona-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nl-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ft-lauderdale-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-l3-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ny-6.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-in-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nn-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-az-4.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-brooklyn-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-office-state-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-puerto-rico-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tp-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-al-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-florida-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ga-5.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nf-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-on-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tp-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-72-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-dn-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-elmore-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-pu-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-[p-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-boston-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nc-4.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nc-5.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nt-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-37615-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-e-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nb-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nj-5.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-oh-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-sc-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ag-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-california-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-oc-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-os-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent---1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-27-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-mi-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ph-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-il-4.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ma-4.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-pe-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-pq-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-1-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ma-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ut-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-va-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-virginia-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ct-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-dr-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ha-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-co-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-nu-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ontario-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-pa-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-sfr-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-tn-3.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-mn-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-sp-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-mb-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-or-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-wi-2.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-georgia-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-memphis-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-woodbridge-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ahwd-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-f-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-san-ramon-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-96753-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-gainesville-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-naples-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-08759-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-addison-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-kerrville-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-palm-beach-gardens-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-uk-1.xml.gzDisallow: /realestateagents/sitemap-far-agent/sitemap-far-agent-ft.-myers-1.xml.gzDisallow: /realestateagents/*_$Disallow: /rdc_user_check/init.jsDisallow: /149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/*Disallow: /flyaroundDisallow: /realtors/Disallow: /ampDisallow: /news/search/Disallow: /advice/search/Disallow: /guides/search/Disallow: /living/search/Disallow: /realestateandhomes-detail/*/gallery/ User-agent: YandexDisallow: / User-agent: peer39_crawlerDisallow: / User-agent: ConnatixDisallow: / User-agent: SogouDisallow: / User-agent: AdsBot-GoogleDisallow:… 312 lines not about AI bots User-agent: GoogleOtherDisallow: / User-agent: facebookbotDisallow: / User-agent: Meta-ExternalAgentDisallow: / User-agent: Meta-ExternalFetcherDisallow: / User-agent: AmazonAdBotDisallow:… 2 lines not about AI bots User-agent: Google-CloudVertexBotDisallow: / User-agent: facebookexternalhitAllow: /realestateandhomes-detail/*?cid=Allow: /realestateandhomes-search/*?cid=Allow: /news/*?cid=Allow: /guides/*?cid=Allow: /advice/*?cid=Allow: /living/*?cid=Allow: /realestateagents/*?cid=Allow: /realestateteams/*?cid=Allow: /realestateagency/*?cid= User-agent: TwitterbotAllow: /realestateandhomes-detail/*?cid=Allow: /realestateandhomes-search/*?cid=Allow: /news/*?cid=Allow: /guides/*?cid=Allow: /advice/*?cid=Allow: /living/*?cid=Allow: /realestateagents/*?cid=Allow: /realestateteams/*?cid=Allow: /realestateagency/*?cid= User-agent: PinterestbotAllow: /realestateandhomes-detail/*?cid=Allow: /realestateandhomes-search/*?cid=Allow: /news/*?cid=Allow: /guides/*?cid=Allow: /advice/*?cid=Allow: /living/*?cid=Allow: /realestateagents/*?cid=Allow: /realestateteams/*?cid=Allow: /realestateagency/*?cid= User-agent: OAI-AdsBotDisallow: /Allow: /realestateandhomes-detail/*?cid=Allow: /realestateandhomes-search/*?cid=Allow: /news/*?cid=Allow: /guides/*?cid=Allow: /advice/*?cid=Allow: /living/*?cid=Allow: /realestateagents/*?cid=Allow: /realestateteams/*?cid=Allow: /realestateagency/*?cid=Allow: /myhome/*?cid= Sitemap: https://www.realtor.com/mortgage/sitemap/mortgagesitemap.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-rss-new/rss-new-index.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-rss-open-house/rss-open-house-index.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-rss-price/rss-price-index.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-rss-sold/rss-sold-index.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-rss-new-rental/rss-new-rental-index.xmlSitemap: https://www.realtor.com/realestateandhomes-detail/sitemap-spec/sitemap-new-homes-ldp-spec-1-index.xmlSitemap: https://www.realtor.com/local/sitemaps/district/sitemap-local-district-1-index.xmlSitemap: https://www.realtor.com/local/sitemaps/school/sitemap-local-school-1-index.xmlSitemap: https://www.realtor.com/sitemap_index.xml … 57 lines not about AI bots
Notes about AI written in the file: "LEGAL NOTICE: Per https://www.realtor.com's Terms of Service, scraping data from this website is unauthorized without the express written permission from Move S"
Everything else we checked
| Front door | Turned away The homepage turned our bot, which says who it is, away (HTTP 429).GET https://www.realtor.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| TDMRep | Refused (HTTP 403)/.well-known/tdmrep.json (W3C Community Group report) |
| Page tags | Homepage not read. |
| Anti-bot software | Served through Kasada, Amazon CloudFront. The homepage set a Kasada cookie (KP_UIDz-ssn, KP_UIDz), which anti-bot software uses.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Homepage not read.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | No Terms of Service page found. |
| Not found | No llms.txt, ai.txt, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402). |
History
42 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Nov 2024More blocking Now blocked: Meta-ExternalAgent and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Oct 2024 and 1 Nov 2024. Evidence: before · after
Show exactly what changed
- User-agent: *+ # LEGAL NOTICE: Per https://www.realtor.com's Terms of Service, scraping data from this website is unauthorized without the express written permission from Move Sales, Inc., operator of https://www.realtor.com.+ User-agent: bingbot+ Crawl-delay: 1++ User-agent: *Disallow: /8058/…Disallow: /apartments/*/*/pg-*- Disallow: /apartments/*/price-*+ Disallow: /apartments/near-me*Disallow: /content…Disallow: /international/us/+ Disallow: /international/*?sort=distance+ Disallow: /international/*?sort=dateDisallow: /*?lang=…Disallow: /newhomesconstruction/*+ Disallow: /newhomecommunities/near-me*Disallow: /news/wp-content/themes/realtor/nobots/…Disallow: /realestateandhomes-detail/M1159985143- Disallow: /realestateandhomes-detail/*?ex=+ Disallow: /realestateandhomes-detail/undefinedDisallow: /realestateandhomes-print/…Disallow: /realestateandhomes-search/*?from_ldp=true- Disallow: /realestateandhomes-search/*?cid=prt- Disallow: /realestateandhomes-search/*?cid=PRT- Disallow: /realestateandhomes-search/*?cid=pvs- Disallow: /realestateandhomes-search/*?cid=soc+ Disallow: /*?cid=Disallow: /realestateandhomes-search/*/pg-*/show-recently-sold- Disallow: /realestateandhomes-search/*/price-Disallow: /realestateandhomes-search/*/radius-+ Disallow: /realestateandhomes-search/undefinedDisallow: /realestateteam/bright-birch-real-estate__mn_1639730…Disallow: /Widgets/+ Disallow: /YfcuVaVBkfnzl2j6UbEcK0Oiqkv88jx6w/…- User-agent: bingbot- Crawl-delay: 1+ User-agent: peer39_crawler+ Disallow: /- Disallow: /8058/- Disallow: /addeliverymodule/- Disallow: /advice/wp-content/themes/realtor/nobots/- Disallow: /amenities?- Disallow: /apartments/*?disp=2*- Disallow: /apartments/*?view=map/pg-*- Disallow: /apartments/*/*-rentals- Disallow: /apartments/*/cat-friendly- Disallow: /apartments/*/dog-friendly- Disallow: /apartments/*/*/pg-*… 177 more lines
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.