Every change since 2023
Each entry is a time one of these 170 sites changed what its rules say to AI bots. There are 261 since Jan 2023. Older ones come from the Internet Archive's monthly copies, so we can only date them to a range of months. Newer ones come from our daily check. Each links to the file before and after. RSS feed
261 changes
Oct 2026
-
Oct 2026
Airbnb Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.
Show exactly what changed
Disallow: /google_place_photo+ Disallow: /api/v2/google_place_photosDisallow: /groups…Disallow: /users/show- Disallow: /s/*?- Disallow: /s/*/homes+ Disallow: /users/profile+ Disallow: /api/v3/LandingPageStaysQuery+ Disallow: /s/*/*Disallow: /things-to-do/places…Disallow: /google_place_photo+ Disallow: /api/v2/google_place_photosDisallow: /groups…Disallow: /users/show- Disallow: /s/*?- Disallow: /s/*/homes+ Disallow: /users/profile+ Disallow: /api/v3/LandingPageStaysQuery+ Disallow: /s/*/*Disallow: /things-to-do/places…Disallow: /google_place_photo+ Disallow: /api/v2/google_place_photosDisallow: /groups…Disallow: /users/show- Disallow: /s/*?- Disallow: /s/*/homes+ Disallow: /users/profile+ Disallow: /api/v3/LandingPageStaysQuery+ Disallow: /s/*/*Disallow: /things-to-do/places…Disallow: /google_place_photo+ Disallow: /api/v2/google_place_photosDisallow: /groups… -
Oct 2026
AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
Show exactly what changed
Allow: /wholesale.html$- Disallow: /wholesale*+ Allow: /wholesale-page-*.htmlDisallow: /productdetail/*…Disallow: /i/api/*+ Disallow: /gcp/300000414/regulatedinformation+ Disallow: /img/wsproduct/sku/*+ Disallow: */aeglodetailweb/api/msite/item?productId*…Disallow: /i/api/*- Host: https://aliexpress.ru- Clean-param: initiative_id&site&spm&g&isrefine&needquery&blanktest&sorttype&cpt&aff_trace_key&ispremium&aff_platform&misspelltext&isonlyaffiliate&afref&aff_short_key&isdl&albcp&albch&albag&crea&netw&trgt&src&plac&aff_id&p4p&tracelog&useimatch&_csrf&utm_source&utm_campaign&utm_term&smsign&smtoken&utm_medium&utm_content&isafchannel&viewtype&aff_click_id&t&transabtest&callback&_csrf_token_&_&origin&rowan_id1&rowan_msg_id&ck&json&iscates+ Disallow: /gcp/300000414/regulatedinformation+ Disallow: /img/wsproduct/sku/*+ Disallow: */aeglodetailweb/api/msite/item?productId*- Sitemap: https://www.aliexpress.com/sitemap/https-en-ugc-desktop-sitemap-index.xml- Sitemap: https://www.aliexpress.com/siteindex/https-en-ugc-desktop-xml-sitemap-index.xml+ User-agent: GPTBot+ Disallow: /+ User-agent: CCBot+ Disallow: /- Sitemap: https://www.aliexpress.com/sitemap/https-en-ugc-desktop-sitemap-index.xml- Sitemap: https://www.aliexpress.com/siteindex/https-en-ugc-desktop-xml-sitemap-index.xml+ User-agent: Google-Extended+ Disallow: /+ User-agent: ClaudeBot+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: Bytespider+ Disallow: /+ -
May to Oct 2026
Best Buy Named GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai and 6 more explicitly, without changing what they may reach.
Show exactly what changed
+ # --------------------------------------------------+ # BEST BUY OFFICIAL ENTERPRISE ROBOTS.TXT+ # Optimized for search engines, AI discovery, ads, and crawl control+ # --------------------------------------------------++ # ==================================================+ # BUCKET 1: AI SEARCH & DISCOVERY BOT ACCESS+ # Allows conversational answer engines to securely access clean landing pages.+ # Mirrors explicit production parameters to protect long-tail revenue.+ # ==================================================+ User-agent: OAI-SearchBot+ User-agent: ChatGPT-User+ User-agent: Perplexity-User+ User-agent: GPTBot+ User-agent: Claude-User+ User-agent: Claudebot+ User-agent: PerplexityBot+ User-agent: Applebot+ User-agent: Google-Extended+ User-agent: Gemeni-AI+ User-agent: Gemini-Deep-Research+ User-agent: AmazonBot++ Allow: /site/+ Allow: /discover-learn/++ # System Application Blockouts+ Disallow: /nex/+ Disallow: /m/e/*+ Disallow: /browse-api/*+ Disallow: /api/*.json+ Disallow: /availability/*+ Disallow: /cart+ Disallow: /cart/*+ Disallow: /cartitems/*+ Disallow: /fulfillment/*+ Disallow: /location/*+ Disallow: /pricing/*+ Disallow: /productfulfillment/*+ Disallow: /profile/* -
Oct 2026
Bloomberg Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-User-agent: CCBot…- User-agent: Google-Extended- Disallow: /-# Development Tools…# AI/LLM Bots- User-agent: ChatGPT-User+ User-agent: Claude-WebDisallow: /…Allow: /tc- User-agent: GPTBot+ User-agent: anthropic-aiDisallow: /…Allow: /tc- User-agent: Claude-Web+ User-agent: PerplexityBotDisallow: /…Allow: /tc- User-agent: anthropic-ai+ User-agent: YouBotDisallow: /…Allow: /tc- User-agent: PerplexityBot+ User-agent: DuckAssistBotDisallow: /…Allow: /tc- User-agent: YouBot+ User-agent: cohere-ai -
May to Oct 2026
Capital One Stopped naming ClaudeBot and Amazonbot; access unchanged.
Show exactly what changed
- User-agent: ClaudeBotUser-agent: Abonti…User-agent: UkrNet Mail Proxy- User-agent: AmazonAdBot- User-agent: Amazonbot- User-agent: SnapchatAds- User-agent: Spotify- User-agent: ELB-HealthCheckerDisallow: /cars/+ User-agent: *+ Disallow: /cars/idplogin+ Disallow: /cars/prequalify?+ Disallow: /cars/search?filters=ey+sitemap: https://www.capitalone.com/sitemap.xml -
Oct 2026
CNN Now allowed: Google-Extended.
Show exactly what changed
User-agent: Exabot+ User-agent: ExaSearchBot+ User-agent: ExaUserUser-agent: FacebookBot…User-agent: GPTBot- User-agent: Google-ExtendedUser-agent: GoogleOther…User-agent: SeznamHomepageCrawler+ User-agent: ShapBot+ User-agent: Shap-UserUser-agent: Sidetrade indexer bot -
May to Oct 2026
eBay Named OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User explicitly, without changing what they may reach.
Show exactly what changed
#- # v25_COM_March_2026+ # v30.2_COM_August_2026### DIRECTIVES ###…Disallow: /ctm/+ Disallow: /d/graphqlDisallow: /dsc/…Disallow: /gh/useracquisition+ Disallow: /gh/user_profileDisallow: /gsr/Disallow: /gss/+ Disallow: /gum/Disallow: /gwc/…Disallow: /logout+ Disallow: /looking-glass-proxy/graphqlDisallow: /lstng…Disallow: /rewards+ Disallow: /rtm*Disallow: /rvh/…Disallow: /sch/+ Disallow: /sch/*_sacat=Disallow: /sch/ajax…+ User-agent: Googlebot+ Disallow: /*_kw+ Disallow: /*?maspect+ Disallow: /*modules=SEARCH_REFINEMENTS_MODEL_V2+ Disallow: /*redirect=mobile+ Disallow: /*redirect%3Dmobile+ Disallow: /*rt%3Dnc+ Disallow: /*rt=nc+ Disallow: /*src=urllib+ Disallow: /?SSOWebDispatcher&tg=web&ru=+ Disallow: /act/ -
Oct 2026
Facebook Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
Show exactly what changed
# https://www.facebook.com/legal/automated_data_collection_terms+ User-agent: Amazonbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: Brightbot+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /+User-agent: PetalBot…- User-agent: Discordbot- Disallow: /*/plugins/*- Disallow: /?*next=- Disallow: /a/bz?- Disallow: /ajax/- Disallow: /album.php- Disallow: /business/*&categories- Disallow: /business/*?categories- Disallow: /business/help/search*&query=- Disallow: /business/help/search*?query=- Disallow: /business/news*&tags- Disallow: /business/news*?tags- Disallow: /business/search*&q=- Disallow: /business/search*?q=- Disallow: /checkpoint/ -
Oct 2026
GitHub Now blocked: Bytespider.
Show exactly what changed
# We also provide an extensive API: https://docs.github.com+ User-agent: GPTBot+ User-agent: OAI-SearchBot+ User-agent: ClaudeBot+ User-agent: anthropic-ai+ User-agent: PerplexityBot+ Crawl-delay: 1+ Allow: /$+ Allow: /about+ Allow: /customer-stories+ Allow: /enterprise+ Allow: /features+ Allow: /marketplace+ Allow: /mcp+ Allow: /newsroom+ Allow: /pricing+ Allow: /resources+ Allow: /security+ Allow: /solutions+ Allow: /topics/download*+ Allow: /topics/comments*+ Allow: /marketplace/actions/download*+ Disallow: /*/*/pulse+ Disallow: /*/*/projects+ Disallow: /*/*/forks+ Disallow: /*/*/issues/new+ Disallow: /*/*/milestones/new+ Disallow: /*/*/issues/search+ Disallow: /*/*/commits/+ Disallow: /*/*/branches+ Disallow: /*/*/contributors+ Disallow: /*/*/tags+ Disallow: /*/*/stargazers+ Disallow: /*/*/watchers+ Disallow: /*/*/network+ Disallow: /*/*/graphs+ Disallow: /*/*/compare+ Disallow: /*/tree/+ Disallow: /gist/+ Disallow: /*/download -
May to Oct 2026
Glassdoor Now blocked: ClaudeBot.
Show exactly what changed
- # Block Glassdoor jobs. Intent is to remove misleading site links SERP. Details at TRFC-3197+ # Block Pages with misaligned search intent, impacting site links SERPDisallow: /Jobs/Glassdoor-Jobs-E100431.htm+ Disallow: /Reviews/Glassdoor-Reviews-E100431.htm+ Disallow: /Salary/Glassdoor-Salaries-E100431.htm+ Disallow: /Reviews/EMPLOYERS-Reviews-E25702.htm+ Disallow: /Reviews/Anonymous-Reviews-E5182641.htm+ Disallow: /Salaries/search-salary-SRCH_KO0,6.htm+ #Blocking low-traffic employer profile pages+ Disallow: /Photos/+ Disallow: /Location/All-*-Office-Locations*++ # Block pages with low SEO potential to reduce crawling+ Disallow: /Compare/-vs-+ Disallow: /Salary/_DEPT*+ Disallow: /Interview/_IL++ # Block specific query filter parameters+ Disallow: *filter.searchCategory+ Disallow: *filter.overallRating+ Disallow: *filter.employmentStatus+ Disallow: *sort+ Disallow: *experienceLevel+ Disallow: *locationId+ Disallow: *filter.experience+ Disallow: *filter.jobFunction+ Disallow: *filter.location+# Blocking pagination on employer infosite TR-12…User-agent: Claude-Web+ User-agent: ClaudeBotUser-agent: Perplexity…Allow: /About/+ Allow: /employers/+ Disallow: /employers/ec+ Disallow: /employers/engagement/api/ -
Oct 2026
Humana Named ClaudeBot, OAI-SearchBot and Claude-SearchBot explicitly, without changing what they may reach.
Show exactly what changed
# For domain: https://www.humana.com/- # Updated 05-31-2018 - July redesign release+ # Updated 09-04-2023 - September 2023 release+ # Updated 10-05-2026 - explicit allow rules for verified AI/search crawler bots (CRC approved, see S0574)# All robots will spider the domain…User-agent: *- Disallow: /404- Disallow: /500- Disallow: /about/legal/si-referral-form/- Disallow: /about/legal/si-request-to-contact/- Disallow: /about/legal/si-request-to-contact-esp/- Disallow: /about/marketpoint-sales-office-locations/?kc=1005011074- Disallow: /about/marketpoint-sales-office-locations/map-directions?p=1- Disallow: /ads/- Disallow: /agent/ad/- Disallow: /agent/support/faqs/- Disallow: /agent/support/tools-and-resources/connections-newsletters/- Disallow: /analytics-test- Disallow: /caregiver-portal/- Disallow: /carousel-test- Disallow: /cha-health/- Disallow: /custom_clients/- Disallow: /dental/ad/- Disallow: /dental/my-dental-iq/adult-dental-care/- Disallow: /dental/my-dental-iq/child-dental-care/- Disallow: /dental/my-dental-iq/infant-dental-care/- Disallow: /dental/my-dental-iq/internet-privacy-statement- Disallow: /dental/my-dental-iq/preventative-care/- Disallow: /dental/my-dental-iq/quiz- Disallow: /dental/my-dental-iq/related-links/- Disallow: /dental/my-dental-iq/why-its-important/Disallow: /direct-connect-logon- Disallow: /e-submissions/- Disallow: /employee-contact- Disallow: /employer/ad/- Disallow: /employer/healthcare-resources/focus/- Disallow: /employer/large-group/support/faqs/- Disallow: /employer/portal-maintenance- Disallow: /employer/small-group/support/faqs/ -
Oct 2026
Mount Sinai Stopped naming CCBot; access unchanged.
Show exactly what changed
- User-agent: *- Disallow: /cgi-bin/- Disallow: /tmp/- Disallow: /vgn-ext/templating- Disallow: /*lp/- Disallow: /insurance/- Disallow: /about/preparedness/coronavirus/staff-resources*- Disallow: /about/covid19/staff-resources*- User-agent: Alexabot- Disallow: /about-us/newsroom/- Crawl-delay: 5-- User-agent: CCBot- Disallow:- Crawl-delay: 20-- User-agent: BLEXBot- Disallow:- Crawl-delay: 10-- User-agent: Findxbot- Disallow:- Crawl-delay: 10-- User-agent: rogerbot- Disallow: /about-us/newsroom/- Crawl-delay: 10-- User-agent: Yeti- Disallow:- Crawl-delay: 10-- User-agent: AhrefsBot- Disallow:- Crawl-delay: 10-- User-agent: SEMrushBot- Disallow:- Crawl-delay: 10 -
May to Oct 2026
Nordstrom Named Meta-ExternalAgent and Meta-WebIndexer explicitly, without changing what they may reach.
Show exactly what changed
- #Modified:07-26-2022- User-agent: *+ #Modified:9/8/2026++ User-agent: AdsBot-Google+ Disallow:++ User-agent: Googlebot-Image+ Disallow:++ User-agent: meta-webindexer/1.1+ Disallow: /api/checkout/web/checkout/v1/shopping-bag++ User-agent: meta-externalagent/1.1+ Disallow: /api/checkout/web/checkout/v1/shopping-bag++ User-agent: Googlebot#PDP…Disallow: /s/an-test*+ Disallow: /soajax/Eta$#Search ResultsDisallow: /sr?*+ Disallow: /search?*+ Disallow: /sr/#Filter- Disallow: *filterByPrice*- Disallow: *filterBySize*- Disallow: *filterByWidth*- Disallow: *filterBySale*Disallow: /brands/*filterByBrand*- Disallow: /brands/*/filter/*brand_*#Browse/Brands…Disallow: /browse/*/libs/*- Disallow: /browse/fan-shop/men- Disallow: /browse/fan-shop/men*- Disallow: /browse/fan-shop/women- Disallow: /browse/fan-shop/women*- Disallow: /browse/fan-shop/kids -
Oct 2026
NPR Now blocked: Claude-SearchBot and Claude-User.
Show exactly what changed
+ User-agent: Claude-SearchBot+ Disallow: /++ User-agent: Claude-User+ Disallow: /+User-agent: Claude-Web…- # Explicitly allowing the TTD contextual crawler for Sponsorship- User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)+ # Allowing the TTD contextual crawler for Sponsorship+ User-agent: TTD-ContentAllow: /- # Explicitly allowing the Google-Mediapartners crawler for Sponsorship+ # Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship+ User-agent: Trade Desk ads.txt & sellers.json crawler+ Allow: /++ # Allowing the AdsBot-Google crawler for Sponsorship+ User-agent: AdsBot-Google+ Allow: /++ # Allowing the AdsBot-Google-Mobile crawler for Sponsorship+ User-agent: AdsBot-Google-Mobile+ Allow: /++ # Allowing the Google-Mediapartners crawler for SponsorshipUser-agent: Mediapartners-Google…- # Explicitly allowing the Google-Display-Ads-Bot crawler for Sponsorship+ # Allowing the Google-Display-Ads-Bot crawler for SponsorshipUser-agent: Google-Display-Ads-Bot…- # Explicitly allowing the IAB tech lab crawler for Sponsorship+ # Allowing the IAB tech lab crawler for Sponsorship -
Oct 2026
Pinterest Now blocked: Meta-ExternalFetcher.
Show exactly what changed
User-agent: MeltwaterNews- User-agent: meta-externalfetcherUser-agent: Metorik…User-agent: AmazonAdBot+ Allow: /homefeed/+ Allow: /homefeed/*Allow: /pin/…- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_western_new_year_s_eve_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_independence_and_unity_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_hanukkah_day_1_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_v_expansion_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261031_halloween_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261123_day_off_for_alphabet_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261129_international_day_of_solidarity_with_the_palestini_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261203_international_day_of_persons_with_disabilities_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_new_year_holiday_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_day_of_goodwill_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261227_feast_of_the_holy_family_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_family_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_world_soil_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_christmas_day_observed_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_christmas_season_begins_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_day_after_christmas_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_cayman_thanksgiving_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261124_vietnam_culture_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261224_yap_constitution_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_chanukah_hanukkah_first_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261122_alphabet_day_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261227_international_day_of_epidemic_preparedness_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_substitute_holiday_for_new_year_holiday_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_new_year_s_eve_afternoon_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_saint_nicholas_day_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_gsc_terms_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_answers_terms_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_vlm_terms_sitemap_auto_publish_www.pinterest.com.xml
Sep 2026
-
May to Sep 2026
IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
Show exactly what changed
- # robots.txt for https://www.imdb.com properties- # See https://m.imdb.com/robots.txt for the mDot equivalent-- User-agent: *- Disallow: /OnThisDay- Disallow: /*/OnThisDay- Disallow: /ads/- Disallow: /*/ads/- Disallow: /ap/- Disallow: /*/ap/- Disallow: /mymovies/- Disallow: /*/mymovies/- Disallow: /r/- Disallow: /*/r/+ # The content on this site is owned by IMDb.com, Inc., an Amazon.com company+ # and is made available for your personal, non-commercial use subject+ # to our Conditions of Use here: https://www.imdb.com/conditions/+ # Use of any device, tool, or process designed to data mine or scrape the content+ # using automated means is prohibited without prior written permission from IMDb.+ # For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/++ User-agent: Googlebot+ User-agent: Googlebot-Image+ User-agent: Googlebot-Video+ User-agent: Google-PageRenderer+ User-agent: Google-InspectionTool+ User-agent: Google-Safety+ User-agent: Google-Site-Verification+ User-agent: AdsBot-Google+ User-agent: AdsBot-Google-Mobile+ User-agent: Chrome-Lighthouse+ User-agent: Storebot-Google+ User-agent: Mediapartners-Google+ User-agent: GoogleStructuredData+ User-agent: Google-Read-Aloud+ User-agent: bingbot+ User-agent: Applebot+ User-agent: DuckDuckBot+ User-agent: DuckAssistBot+ User-agent: Baiduspider -
Apr to Sep 2026
Indeed Named ClaudeBot, Meta-ExternalAgent, AI2Bot, Claude-SearchBot, PerplexityBot, DuckAssistBot and 5 more explicitly, without changing what they may reach.
Show exactly what changed
Allow: /- Allow: /hire/*?*isid=- Allow: /personeel/*?*isid=- Allow: /reclutamiento/*?*isid=- Allow: /recruiting/*?*isid=- Allow: /recrutement/*?*isid=+ Allow: /*&start=0&+ Allow: /*&start=10&+ Allow: /*&start=20&+ Allow: /*&start=30&+ Allow: /*&start=40&+ Allow: /*&start=50&+ Allow: /*&start=60&+ Allow: /*&start=70&+ Allow: /*&start=80&+ Allow: /*&start=90&Disallow: /*rt=nc…Disallow: /*?rss+ Disallow: /*&serpstart=+ Disallow: /*/vaclkDisallow: /addlLoc/…Disallow: /%E5%B7%A5%E4%BD%9C/- Disallow: /%E5%B7%A5%E4%BD%9C/CN/- Disallow: /%E5%B7%A5%E4%BD%9C/title/Disallow: /%E6%B1%82%E4%BA%BA/JP/…Disallow: /ita?+ Disallow: /ita/v1/publisherDisallow: /ja/clk?…Disallow: /jobb/- Disallow: /jobb/SE/- Disallow: /jobb/title/Disallow: /jobroll…Disallow: /praca/- Disallow: /praca/PL/- Disallow: /praca/title/ -
Sep 2026
Reuters Named Applebot-Extended explicitly, without changing what it may reach.
Show exactly what changed
User-agent: ADmantX+ User-agent: AdmiralBotUser-agent: AdsBot-Google…User-agent: doubleverify- User-agent: DuckDuckBotUser-agent: ElevenlabsBot…Disallow: /latam/- Disallow: /account/subscribe/payment/+ Disallow: /jp/+ Disallow: /account/subscribe/payment+ User-agent: Applebot-Extended+ Disallow: /+# Block all other bots…SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml+ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml+ SITEMAP: https://www.reuters.com/service/archive-sitemap/sitemap.xml -
Sep 2026
WebMD Now allowed: ChatGPT-User.
Show exactly what changed
# Robots.txt file WebMD- # Updated 1/8/2025+ # Updated 8/31/2026 (CONSFE-458)…Disallow: */search/search_results/- Disallow: /drugs/ReportAbuse.aspx*- Disallow: */cm/- Disallow: /Share.aspx*- Disallow: /mm/- Disallow: /kapi/- Disallow: /click*- Disallow: /drugs/2/search*- Disallow: /sponsored/- Disallow: /api/- Disallow: /search/2/api/- Disallow: /webmd_static_vue/+ Disallow: /500Disallow: /aim/- Disallow: /6-tools-for-managing-your-medications- Disallow: /attr-cm-warning-signs- Disallow: /bipolar-disorder/bipolar-depression-management+ Disallow: /api/Disallow: /bipolar-disorder/video/bipolar-fatherhoodDisallow: /bipolar-disorder/video/bipolar-toolbox- Disallow: /bluechew-review- Disallow: /brain/treating-gmg-video- Disallow: /brain/video/treating-gmg-videoDisallow: /breast-cancer/bc-treatment-21/provider-select- Disallow: /cbd/about-cbd- Disallow: /cbd/best-cbd-gummies-for-anxiety- Disallow: /cbd/best-cbd-oil-for-anxiety- Disallow: /cbd/best-cbd-oil-for-pain- Disallow: /cbd/best-cbd-oil-of-2023- Disallow: /cbd/best-full-spectrum-cbd-oils- Disallow: /cbd/best-gummies-of-2023- Disallow: /cbd/cbd-101-concentrated-cannabidiol- Disallow: /cbd/cbd-for-dogs- Disallow: /cbd/cbd-for-sleep- Disallow: /cbd/cbdfx-review -
Sep 2026
X Named Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher explicitly, without changing what they may reach.
Show exactly what changed
- # Google Search Engine Robot- # ==========================+ # Google / Bing Search Engine Robots+ # ==================================+ # Shared group so Googlebot and Bingbot always get the same rules.User-agent: Googlebot+ User-agent: Bingbot- Allow: /*?lang=+ Allow: /*?s=+ Allow: /*?t=+ Allow: /*?ref_src=Allow: /hashtag/*?src=…Allow: /i/api/+ # Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in+ # favor of Allow, so /i/api/ fetches stay crawlable.+ Allow: /i/api/*+ Disallow: /*?lang=en-ss+ Allow: /*?lang=Disallow: /search/realtime…+ Disallow: /*/analytics+ Disallow: /*/followers+ Disallow: /*/following+ Disallow: /*/verified_followers++ Disallow: /account/deactivated+ Disallow: /settings/deactivated++ # Only `*` and `$` are metacharacters here (RFC 9309); regex classes match+ # literally. /photo is anchored so /status/*/photo/1 media pages stay open.+ Disallow: /*/likes+ Disallow: /*/likes?+ Disallow: /*/retweets+ Disallow: /*/retweets?+ Disallow: /*/media+ Disallow: /*/media?+ Disallow: /*/photo$
Aug 2026
-
Aug 2026
Facebook Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
Show exactly what changed
# https://www.facebook.com/legal/automated_data_collection_terms- User-agent: Amazonbot- Disallow: /-- User-agent: Applebot-Extended- Disallow: /-- User-agent: ClaudeBot- Disallow: /-- User-agent: Google-Extended- Disallow: /-- User-agent: GPTBot- Disallow: /-- User-agent: PerplexityBot+ User-agent: PetalBotDisallow: /- User-agent: PetalBot+ User-agent: ScrapyDisallow: /…+ User-agent: Yeti+ Disallow: /+User-agent: Applebot…Disallow: /album.php+ Disallow: /business/*&categories+ Disallow: /business/*?categories+ Disallow: /business/help/search*&query=+ Disallow: /business/help/search*?query=+ Disallow: /business/news*&tags+ Disallow: /business/news*?tags+ Disallow: /business/search*&q=+ Disallow: /business/search*?q=Disallow: /checkpoint/ -
Aug 2026
The Economist Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.
Show exactly what changed
- # Bot Directives+ # ==========================================+ # USER-FACING LLM BOTS+ # Allowed to fetch on behalf of a user+ # ==========================================- User-agent: GPTBot+ User-agent: ChatGPT-User+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: Google-Extended+ User-agent: Claude-SearchBot+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: ChatGPT-User+ User-agent: Claude-User+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: CCBot+ User-agent: Claude-Web+ Allow: /pro$+ Allow: /pro/Disallow: /- User-agent: PiplBot+ User-agent: PerplexityBot+ Allow: /pro$+ Allow: /pro/Disallow: /+ User-agent: Perplexity-User+ Allow: /pro$+ Allow: /pro/
Jul 2026
-
May to Jul 2026
Axios Named Google-Extended explicitly, without changing what it may reach.
Show exactly what changed
+ User-agent: Google-Extended+ Allow: /+ Disallow: /core/*+ Disallow: /admin/+ Disallow: /login/+ Disallow: /sso/+ Disallow: /api/+User-agent: * -
Dec 2025 to Jul 2026
Craigslist Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.
Show exactly what changed
- User-agent: Diffbot- User-agent: FacebookBot- User-agent: FriendlyCrawler- User-agent: Google-Extended- User-agent: ImagesiftBot- User-agent: img2dataset- User-agent: Meta-ExternalAgent- User-agent: omgili- User-agent: omgilibot- User-agent: PerplexityBot- User-agent: Timpibot- User-agent: VelenPublicWebCrawler- User-agent: YouBot- Disallow: /-+ Sitemap: https://www.craigslist.org/sitemap-index-categories-and-hubs.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-00.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-01.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-02.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-03.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-04.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-05.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-06.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-07.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-08.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-09.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-10.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-11.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-12.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-13.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-14.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-15.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-16.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-17.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-18.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-19.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-20.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-21.xml+ Sitemap: https://www.craigslist.org/sitemap-index-postings-22.xml -
Mar to Jul 2026
LinkedIn Now blocked: Bytespider. Now allowed: Claude-SearchBot.
Show exactly what changed
(files too large to diff inline) -
May to Jul 2026
Twitch Named Amazonbot, Amzn-SearchBot and Amzn-User explicitly, without changing what they may reach.
Show exactly what changed
- User-Agent: *- Allow: /- Allow: /directory- Allow: /directory/all- Allow: /directory/*- Allow: /videos/week- Allow: /.well-known/assetlinks.json+ User-agent: Amazonbot+ Disallow:++ User-agent: Amazonbot-Video+ Disallow:++ User-agent: Amzn-SearchBot+ Disallow:++ User-agent: Amzn-User+ Disallow:++ User-agent: *+ Disallow: /login+ Disallow: /signup+ Disallow: /auth/+ Disallow: /user/account-recovery+ Disallow: /user/password-reset+ Disallow: /user/not-me+ Disallow: /activateDisallow: /admin/*- Disallow: /email-unsubscribe/- Disallow: /login$- Disallow: /message/*- Disallow: /signup$+ Disallow: /wv/*Disallow: /user/*+ Disallow: /message/*+ Disallow: /settings/+ Disallow: /subscriptions+ Disallow: /inventory+ Disallow: /wallet+ Disallow: /drops/
Jun 2026
-
Jun 2026
Amazon Now blocked: Meta-WebIndexer.
Show exactly what changed
Disallow: /++ User-agent: Timpibot+ Disallow: /++ User-agent: Thinkbot+ Disallow: /++ User-agent: AddSearchBot+ Disallow: /++ User-agent: AI2Bot-DeepResearchEval+ Disallow: /++ User-agent: Andibot+ Disallow: /++ User-agent: Anomura+ Disallow: /++ User-agent: atlassian-bot+ Disallow: /++ User-agent: BuddyBot+ Disallow: /++ User-agent: Channel3Bot+ Disallow: /++ User-agent: ChatGLM-Spider+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: Cloudflare-AutoRAG+ Disallow: /++ User-agent: CloudflareBrowserRenderingCrawler+ Disallow: / -
Jun 2026
Bloomberg Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.
Show exactly what changed
Disallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam+ Allow: /faq+ Allow: /tc…Disallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam+ Allow: /faq+ Allow: /tcUser-agent: GPTBotDisallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam+ Allow: /faq+ Allow: /tcUser-agent: Claude-WebDisallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam+ Allow: /faq+ Allow: /tcUser-agent: anthropic-aiDisallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam+ Allow: /faq+ Allow: /tcUser-agent: PerplexityBotDisallow: /+ Allow: /professional+ Allow: /company+ Allow: /latam -
Mar to Jun 2026
CNN Now blocked: Claude-SearchBot, Claude-User, Perplexity-User and MistralAI-User.
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/news.xml- Sitemap: https://www.cnn.com/sitemaps/sitemap-section.xml- Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xmlSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml…User-agent: Ai2Bot-Dolma+ User-agent: AliyunSecBotUser-agent: Amazonbot…User-agent: Applebot-Extended+ User-agent: Archive.org_botUser-agent: AwarioRssBot…User-agent: ClaudeBot+ User-agent: Claude-SearchBot+ User-agent: Claude-UserUser-agent: Claude-Web…User-agent: EchoboxBot+ User-agent: ExabotUser-agent: FacebookBot…User-agent: GoogleOther-Video+ User-Agent: IAB-Tech-LabUser-agent: iaskspider/2.0…User-agent: magpie-crawler+ User-agent: MistralAI-user+ User-agent: MyCentralAIScraperBotUser-agent: NewsNow…User-agent: PerplexityBot+ User-agent: Perplexity-UserUser-agent: PetalBot+ User-agent: Poseidon Research CrawlerUser-agent: QuillBotUser-agent: quillbot.com+ User-agent: Quora-BotUser-agent: SBIntuitionsBotUser-agent: Scrapy -
Apr to Jun 2026
Reuters Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.
Show exactly what changed
- # robots.txt for www.reuters.com-- User-agent: AI2Bot- User-agent: AI2Bot-Dolma- User-agent: Alibaba.Security.Heimdall- User-agent: AliyunSecBot/Aliyun- User-agent: anthropic-ai- User-agent: Applebot-Extended- User-agent: archive.org_bot- User-agent: bedrockbot- User-agent: Bytespider- User-agent: CCBot- User-agent: ChatGLM-Spider- User-agent: Claude-SearchBot- User-agent: Claude-Web- User-agent: ClaudeBot- User-agent: cohere-ai- User-agent: cohere-training-data-crawler- User-agent: Crawlspace- User-agent: DeepSeekBot- User-agent: Diffbot- User-agent: FacebookBot- User-agent: Feedfetcher-Google- User-agent: Gemini-Deep-Research- User-agent: Google-CloudVertexBot- User-agent: Google-Extended+ User-agent: AASA-Bot+ User-agent: ADmantX+ User-agent: AdsBot-Google+ User-agent: AdsBot-Google-Mobile+ User-agent: AmazonAdBot+ User-agent: Amzn-User+ User-agent: Applebot+ User-agent: AppleNewsBot+ User-agent: Bingbot+ User-agent: BingPreview+ User-agent: BrightEdge+ User-agent: BrightEdgeOnCrawl+ User-agent: CensysInspect+ User-agent: ChatGPT-User -
Jun 2026
The Verge Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.
Show exactly what changed
- User-agent: *- Disallow: /wp-admin/- Allow: /wp-admin/admin-ajax.php+ User-agent: Amazonbot+ Allow: /++ User-agent: Applebot+ Allow: /+User-agent: GPTBot…- User-agent: Google-Extended+ User-agent: A6-IndexerDisallow: /+ Allow: /sp/- User-agent: Applebot-Extended+ User-agent: AhrefsBotDisallow: /+ Allow: /sp/- User-agent: Applebot- Allow: /+ User-agent: AI2Bot+ Disallow: /+ Allow: /sp/+ User-agent: AI2Bot-DeepResearchEval+ Disallow: /+ Allow: /sp/++ User-agent: Ai2Bot-Dolma+ Disallow: /+ Allow: /sp/++ User-agent: aiHitBot+ Disallow: /+ Allow: /sp/ -
Jun 2026
The Wall Street Journal Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.
Show exactly what changed
+ # NOTICE: Collection of content and other data on https://www.wsj.com/ through+ # automated means is prohibited unless you have express written+ # permission from Dow Jones & Company, Inc. and may only be conducted for the+ # limited purpose contained in said permission.+ #+ # Dow Jones & Company, Inc. Terms of Use may be found at+ # https://www.dowjones.com/terms-of-use/+ #+ # If you would like to apply for permission to license the+ # intellectual property and/or other materials of Dow Jones & Company, Inc.’s+ # brands, please contact us via email at copyright@dowjones.com.+User-agent: *- Sitemap: https://www.wsj.com/sitemap.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_google_news.xml- Sitemap: https://www.wsj.com/sitemap_topics.xml- Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml- Sitemap: https://www.wsj.com/live_news_sitemap.xml- Sitemap: https://www.wsj.com/wsj_graphics_sitemap.xml- Sitemap: https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_types.xml- Sitemap: https://www.wsj.com/authors_sitemap.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml- Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp- Sitemap: https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp- Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml- Sitemap: https://www.wsj.com/buyside/sitemap.xml- Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml+ Disallow: /+ User-agent: googlebot+ User-agent: googlebot-image+ User-agent: GoogleOther+ User-agent: Googlebot-Video+ User-agent: Google-InspectionTool+ User-agent: AdsBot-Google+ User-agent: AdsBot-Google-Mobile+ User-agent: AdsBot-Google-Mobile-Apps+ User-agent: Storebot-Google+ User-agent: google-read-aloud -
Apr to Jun 2026
USA Today Now blocked: Amzn-SearchBot, Meta-WebIndexer, Perplexity-User, Google-Agent and Amzn-User.
Show exactly what changed
+ User-agent: Claude+ Disallow: /+User-agent: Claude-Web…+ User-agent: claudeweb+ Disallow: /++ User-agent: Amzn-SearchBot+ Disallow: /++ User-agent: Amzn-User+ Disallow: /++ User-agent: AndiBot+ Disallow: /+User-agent: cohere-ai…Disallow: /+ Allow: /contributor-content/+ Allow: /story/special/contributor-content/+ Allow: /shopping/…Disallow: /+ Allow: /contributor-content/+ Allow: /story/special/contributor-content/+ Allow: /shopping/…Disallow: /+ Allow: /contributor-content/+ Allow: /story/special/contributor-content/+ Allow: /shopping/…
May 2026
-
May 2026
GOV.UK Named Meta-ExternalAgent explicitly, without changing what it may reach.
Show exactly what changed
Disallow: /*/print$- # Don't allow indexing of user needs pages- Disallow: /info/*+ # Don't allow indexing of site search+ Disallow: /search/all*Sitemap: https://www.gov.uk/sitemap.xml+ # The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models.+ # It results in timeouts from Vertex that back up requests from users making genuine searches+ User-agent: meta-externalagent+ Disallow: /search/all*+# https://ahrefs.com/robot/ crawls the site frequently…- # https://www.deepcrawl.com/bot/ makes lots of requests. Ideally- # we'd slow it down rather than blocking it but it doesn't mention- # whether or not it supports crawl-delay.+ # https://www.deepcrawl.com/bot/ makes lots of requests. Ideally we'd slow it+ # down rather than blocking it but it doesn't mention whether or not it+ # supports crawl-delay.User-agent: deepcrawl…Disallow: /-- # Google's crawler was sending requests for each variation of query param for the sectors page of licence-finder- # resulting in millions of requests a day.- User-agent: Googlebot- Disallow: /licence-finder/*- -
May 2026
Instacart Named Meta-ExternalAgent explicitly, without changing what it may reach.
Show exactly what changed
+ ### Rules for Meta AI Training Crawlers ###+ User-Agent: meta-externalagent+ Disallow: /++ User-Agent: Meta-ExternalAgent+ Disallow: /++ User-Agent: FacebookBot+ Disallow: /+### All other bots ### -
May 2026
Macy's Named ClaudeBot explicitly, without changing what it may reach.
Show exactly what changed
+ User-agent: AdsBot-Google+ Disallow:++ User-agent: adidxbot+ Disallow:+User-agent: *…Disallow: /buy/*- Disallow: /my-bag/*+ Disallow: /my/bag*+ Disallow: /*|*+ Disallow: /*%7C*+ Disallow: /*,*,*/*,*,*Disallow: /shop/b/*/*,*/*,*- Disallow: /shop/*/*,*,*/*,*,*- Disallow: /shop/*/*/*,*,*/*,*,*Disallow: *nocrawl*-Disallow: *?ss=true…Disallow: /sitemonetization/--+ Disallow: /userEventsConsumer/*+ Disallow: /EventsWar/*+ Disallow: /reviews/myReviews*+ Disallow: /catalog*+ Disallow: /pricingpolicy.ognc*+ Disallow: /account-xapi/*+ Disallow: /nosessioncookie.jsp+ Disallow: /cms/slp/2/*+ Disallow: /wgl/*+ Disallow: /v1/*+ Disallow: /campaign/affiliate*+ Disallow: /regsvc/*+ Disallow: /career-advice/*+ Disallow: /jobs*+ Disallow: /swipe/*+ Disallow: /shop/registry/wedding/product/* -
Feb to May 2026
Pinterest Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.
Show exactly what changed
+ # Allowlist your bot: https://help.pinterest.com/bot-submission-form+ ## Pinterest is hiring!…#- # Check out some of our available positions at https://careers.pinterest.com/careers/engineering+ # Check out some of our available positions at https://www.pinterestcareers.com/- User-agent: *+ User-agent: 2checkout+ User-agent: AASA-Bot+ User-agent: AccessibleWebBot+ User-agent: Adagiobot+ User-agent: adbeat_bot+ User-agent: ADmantX+ User-agent: AdometryBot+ User-agent: Adsense-Snapshot-Google+ User-agent: AdsTxtCrawler+ User-agent: Adyen+ User-agent: AhrefsBot+ User-agent: AhrefsSiteAudit+ User-agent: Akismet+ User-agent: Algolia Crawler+ User-agent: AmazonProductDiscovery+ User-agent: Aolbot-News+ User-agent: APIs-Google+ User-agent: AppInsights+ User-agent: Applebot+ User-agent: AppleNewsBot+ User-agent: Archive-It+ User-agent: archive.org_bot+ User-agent: ArchiveBot+ User-agent: ArchiveTeam-ArchiveBot+ User-agent: AspiegelBot+ User-agent: Audisto Crawler+ User-agent: Awario+ User-agent: Baidu-ADS+ User-agent: Baiduspider+ User-agent: Barkrowler+ User-agent: bingbot -
May 2026
Stripe Docs Added
Content-Signal: ai-train=yes, search=yes, ai-input=yes.Show exactly what changed
User-agent: *+ Content-Signal: ai-train=yes, search=yes, ai-input=yesAllow: /- Allow: /$Disallow: /handoff-healthcheck -
May 2026
Supabase Named GPTBot, ClaudeBot, Google-Extended and PerplexityBot explicitly, without changing what they may reach. Added
Content-Signal: ai-train=yes, search=yes, ai-input=yes.Show exactly what changed
User-agent: *+ Allow: /++ User-agent: GPTBot+ User-agent: ClaudeBot+ User-agent: Google-Extended+ User-agent: PerplexityBot+ Allow: /+Sitemap: https://supabase.com/sitemap.xml+ Content-Signal: ai-train=yes, search=yes, ai-input=yes+ -
May 2026
Vercel Added
Content-Signal: search=yes, ai-input=yes, ai-train=no.Show exactly what changed
User-Agent: *+ Content-Signal: search=yes, ai-input=yes, ai-train=no++ # Allow robots to be able to crawl the OG Image API route and all the subpaths+ Allow: /api/og/*+ Allow: /api/docs-og*+ Allow: /api/product-og*+ Allow: /api/dynamic-og*+ Allow: /api/www/avatar/*+Disallow:- Disallow: /api+ Disallow: /api/Disallow: /oauth…Disallow: /old-browser.html- Disallow: /docs/concepts/payments-and-billing/usage-based-pro-plan+ Disallow: /preauthorize+ Disallow: /signup?*+ Disallow: /invites/- Sitemap: https://vercel.com/crawled-sitemap.xml+ Sitemap: https://vercel.com/sitemap.xml+
Apr 2026
-
Apr 2026
eBay Now blocked: GPTBot, Applebot-Extended and Meta-ExternalAgent.
Show exactly what changed
# Notwithstanding the foregoing, eBay may permit automated access to- # access certain eBay pages but soley for the limited purpose of+ # access certain eBay pages but solely for the limited purpose of# including content in publicly available search engines. Any other…#- # v24.5_COM_December_2025+ # v25_COM_March_2026### DIRECTIVES ###…Disallow: /b/*?*&+ Disallow: /pe/*Disallow: /b/*?*|…Disallow: /*_kw+ Disallow: /b/*_mwBanner=1*rt=nc+ Disallow: /b/*rt=nc*_mwBanner=1+ Disallow: /motors/diy/beacon/Disallow: /*?maspect…Disallow: /*src=urllib+ Disallow: /motors/diy/beacon/Disallow: /adell-mutfak…- User-agent: Bytespider- Disallow: /+ #+ # LLM / AI bot directives+ #+ User-agent: BytespiderUser-agent: CCBot- Disallow: /-User-agent: ChatGLM-Spider- Disallow: /-User-agent: ClaudeBot- Disallow: / -
Jan 2025 to Apr 2026
Instagram Named PerplexityBot explicitly, without changing what it may reach.
Show exactly what changed
# permission.+ # All authorized user-agents listed on this page must comply with Meta’s+ # Automated Data Collection Terms available at:+ # https://www.facebook.com/legal/automated_data_collection_termsUser-agent: Amazonbot…+ User-agent: Brightbot+ Disallow: /+User-agent: ClaudeBot…+ User-agent: PerplexityBot+ Disallow: /+User-agent: PetalBot…+ User-agent: Scrapy+ Disallow: /++ User-agent: uptimerobot+ Disallow: /+User-agent: viberbot…+ User-agent: Yandex+ Disallow: /++ User-agent: Yeti+ Disallow: /+User-agent: Applebot+ Disallow: /accounts/login/*?next=Disallow: /ajax/Disallow: /client_error/+ Disallow: /direct/Disallow: /logging/ -
Apr 2026
The Atlantic Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added
Content-Signal: search=yes, ai-input=no, ai-train=no.Show exactly what changed
+ # As a condition of accessing this website, you agree to abide by the+ # following content signals:++ # (a) If a content-signal = yes, you may collect content for the+ # corresponding use.+ # (b) If a content-signal = no, you may not collect content for the+ # corresponding use.+ # (c) If the website operator does not include a content signal for a+ # corresponding use, the website operator neither grants nor restricts+ # permission via content signal with respect to the corresponding use.++ # The content signals and their meanings are:++ # search: building a search index and providing search results (e.g., returning+ # hyperlinks and short excerpts from your website's contents). Search+ # does not include providing AI-generated search summaries.+ # ai-input: inputting content into one or more AI models (e.g., retrieval+ # augmented generation, grounding, or other real-time taking of+ # content for generative AI search answers).+ # ai-train: training or fine-tuning AI models.++ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF+ # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT+ # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.+User-agent: *…Disallow: /video/embed/+ Disallow: /zephr/*Disallow: /video/iframe/*…+ User-agent: AmazonAdBot+ Allow: /+User-agent: Amazonbot…+ User-agent: Applebot+ Disallow: / -
Apr 2026
The New York Times Now blocked: Meta-WebIndexer.
Show exactly what changed
Disallow: /athletic/embedded-interactive/*- Disallow: /athletic/interactive/*Disallow: /card/panel/…+ User-agent: AmazonBot+ Disallow: /wirecutter/+User-agent: anthropic-ai…+ User-agent: archive.org_bot+ Disallow: /+User-agent: AudigentAdBot…+ User-agent: Meta-WebIndexer+ User-agent: meta-webindexer/1.1+ Disallow: /+User-agent: MyCentralAIScraperBot…- # Other Bot Rules+ # Ad Bot Rules++ User-agent: AmazonAdBot+ Allow: /+++ # Social Bot Rules+User-agent: facebookexternalhit…+ User-agent: RedditBot+ Allow: /*?*smid= -
Apr 2026
Zillow Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot and Amazonbot.
Show exactly what changed
- # Access to and use of Zillow.com is governed by our Terms of Use. See http://www.zillow.com/corp/Terms.htm+ # Access to and use of Zillow.com is governed by our Terms of Use. See https://www.zillow.com/corp/Terms.htm…+ Allow: /homes/apartments-pet-friendly/$+ Allow: /homes/apartments-1-bedrooms/$+ Allow: /homes/apartments-2-bedrooms/$+ Allow: /homes/apartments-under-1500/$+ Allow: /homes/apartments-under-2000/$+ Allow: /homes/apartments-3-bedrooms/$+ Allow: /homes/pet-friendly/$+ Allow: /homes/rooms-for-rent/$+ Allow: /homes/apartments-with-washer-dryer/$++ Allow: /homes/apartments-pet-friendly/*_p/$+ Allow: /homes/apartments-1-bedrooms/*_p/$+ Allow: /homes/apartments-2-bedrooms/*_p/$+ Allow: /homes/apartments-under-1500/*_p/$+ Allow: /homes/apartments-under-2000/*_p/$+ Allow: /homes/apartments-3-bedrooms/*_p/$+ Allow: /homes/pet-friendly/*_p/$+ Allow: /homes/rooms-for-rent/*_p/$+ Allow: /homes/apartments-with-washer-dryer/*_p/$+Disallow: /api/…Disallow: /*/foreclosure--- Disallow: */foreclosed/*+ Disallow: /*/foreclosed/*Disallow: /*/GetResults.htm*…Disallow: /postings/PostMakeMeMoveStreamlined.htm?- Disallow: */pre-foreclosure/*+ Disallow: /*/pre-foreclosure/*Disallow: /pricing/…Disallow: /sell/?*- Disallow: /sell/options/*+ Disallow: /sell/options/
Mar 2026
-
Mar 2026
Axios Now allowed: ClaudeBot and Google-Extended.
Show exactly what changed
- User-agent: Google-Extended- Disallow: /-User-agent: CCBot…- User-agent: ClaudeBot- Disallow: /-- User-agent: Claude-Web- Disallow: /-User-agent: Diffbot -
Mar 2026
Hotels.com Named Perplexity-User explicitly, without changing what it may reach.
Show exactly what changed
User-agent: PerplexityBot- User-agent: Perplexit-User+ User-agent: Perplexity-UserUser-agent: Claude-User -
Mar 2026
Instacart Now blocked: PerplexityBot.
Show exactly what changed
### Instacart's Terms of Service https://www.instacart.com/terms- ### Last Updated 2-28-2025+ ### Last Updated 02-20-2026…Allow: /+ Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlock…Disallow: /ex/+ Disallow: /orders/Disallow: /store/orders/…Allow: /+ Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlock…Disallow: /ex/+ Disallow: /orders/Disallow: /store/orders/…Allow: /+ Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlock…Disallow: /ex/+ Disallow: /orders/Disallow: /store/orders/…Allow: /+ Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlock…Disallow: /ex/+ Disallow: /orders/Disallow: /store/orders/…Allow: /+ Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlock -
Mar 2026
Reuters Now blocked: AI2Bot and MistralAI-User. Now allowed: Claude-User.
Show exactly what changed
- User-agent: PiplBot- Disallow: /-- User-agent: CCbot- Disallow: /-- # ADO-657700- User-agent: Meta-ExternalAgent- Disallow: /-- User-agent: Meta-ExternalFetcher- Disallow: /-- User-agent: Google-Extended- Disallow: /- #-- User-agent: FacebookBot- Disallow: /-- # ADO-658177+ User-agent: AI2Bot+ User-agent: AI2Bot-Dolma+ User-agent: Alibaba.Security.Heimdall+ User-agent: AliyunSecBot/AliyunUser-agent: anthropic-ai- Disallow: /-+ User-agent: Applebot-Extended+ User-agent: archive.org_bot+ User-agent: bedrockbot+ User-agent: Bytespider+ User-agent: CCBot+ User-agent: ChatGLM-Spider+ User-agent: Claude-SearchBot+ User-agent: Claude-WebUser-agent: ClaudeBot- Disallow: /-
Feb 2026
-
Feb 2026
Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
- # Allowed for archiving- # =====================- User-Agent: *- Allow: /$+ User-agent: *Disallow: / -
Feb 2026
Stack Overflow Named GPTBot, Google-Extended, Amazonbot and ChatGPT-User explicitly, without changing what they may reach. Removed its RSL
License:line.Show exactly what changed
- License: https://stackoverflow.com/license.xml-User-Agent: *…+ User-agent: Googlebot+ User-agent: Google-Extended+ User-agent: amazonbot+ User-agent: GPTBot+ User-agent: ChatGPT-User+ Disallow: /posts/+ Disallow: /posts?+ Disallow: /amzn/click/+ Disallow: /questions/ask/+ Disallow: /questions/ask?+ Disallow: /search/+ Disallow: /search?+ Disallow: /feeds/+ Disallow: /feeds?+ Disallow: /users/login/+ Disallow: /users/login?+ Disallow: /users/logout/+ Disallow: /users/logout?+ Disallow: /users/filter/+ Disallow: /users/filter?+ Disallow: /users/signup+ Disallow: /users/signup/+ Disallow: /users/signup?+ Disallow: /users/authenticate/+ Disallow: /users/authenticate?+ Disallow: /users/oauth/*+ Disallow: /users/flag-summary/+ Disallow: /users/flair/+ Disallow: /users/flair?+ Disallow: /users/activity/+ Disallow: /users/activity/?+ Disallow: /users/stats/+ Disallow: /users/*?tab=accounts+ Disallow: /users/*?tab=activity+ Disallow: /users/rep/show -
Feb 2026
Tripadvisor Now blocked: CCBot, Meta-ExternalAgent and Amazonbot.
Show exactly what changed
Sitemap: https://www.tripadvisor.com/business/sitemap.xml- User-Agent: magpie-crawler- User-Agent: Google-Extended- User-Agent: ClaudeBot- User-Agent: PetalBot+ User-Agent: Amazonbot+ User-Agent: Applebot-ExtendedUser-Agent: Bytespider+ User-Agent: CCBot+ User-Agent: ClaudeBot+ User-Agent: Cohere-ai+ User-Agent: GPTBotUser-Agent: Google-CloudVertexBot- User-Agent: Applebot-Extended+ User-Agent: Google-Extended+ User-Agent: Img2dataset+ User-Agent: PanguBot+ User-Agent: PetalBotUser-Agent: SentiBot- User-Agent: GPTBot+ User-Agent: YouBot+ User-Agent: magpie-crawler+ User-Agent: meta-externalagentDisallow: /…User-Agent: *+ User-Agent: ChatGPT-User+ User-Agent: ChatGPT-User/2.0User-Agent: Gemini-Deep-ResearchUser-Agent: OAI-SearchBot- User-Agent: ChatGPT-User/2.0- User-Agent: ChatGPT-UserUser-Agent: applebot…Disallow: /AirportIP- Disallow: /AirWatchAjaxHandler- Disallow: /AirWatchAlertDhtmlPopup- Disallow: /AirWatchSubscriptionAjax- Disallow: /AirWatchSubscriptionWidgetDisallow: /AITripBuilder -
Feb 2026
Zocdoc Named ClaudeBot, Google-Extended, Claude-SearchBot, PerplexityBot, Claude-User and Perplexity-User explicitly, without changing what they may reach.
Show exactly what changed
+ User-agent: ClaudeBot+ Allow: /++ User-agent: Claude-User+ Allow: /++ User-agent: Claude-SearchBot+ Allow: /++ User-agent: Google-Extended+ Allow: /++ User-agent: PerplexityBot+ Allow: /++ User-agent: Perplexity-User+ Allow: /+User-agent: *
Jan 2026
-
Jan 2026
BBC Now blocked: Amzn-SearchBot and Amzn-User.
Show exactly what changed
- # version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f+ # version: 8c1c7053723d6bd586586053ecdd261591364cfe…Sitemap: https://www.bbc.com/bbcx/audio_archive_sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/video_documentaries_sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/content_index_sitemap.xml…+ User-agent: Amzn-SearchBot+ Disallow: /++ User-agent: Amzn-User+ Disallow: / -
Oct 2025 to Jan 2026
CNN Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.
Show exactly what changed
- Sitemap: https://www.cnn.com/sitemaps/cnn/index.xml- Sitemap: https://www.cnn.com/sitemaps/cnn/news.xmlSitemap: https://www.cnn.com/sitemap/news.xml…Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xml- Sitemap: https://www.cnn.com/ampstories/sitemap.xml- Sitemap: https://edition.cnn.com/sitemaps/news.xmlSitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xmlSitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml- Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml- Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xmlSitemap: https://www.cnn.com/sitemap/section/politics.xml…Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml+ Sitemap: https://www.cnn.com/sitemap/tve.xmlUser-agent: AI2Bot…User-agent: Amazonbot+ User-agent: amzn-searchbot+ User-agent: amzn-userUser-agent: anthropic-ai- User-agent: ApplebotUser-agent: Applebot-Extended…User-agent: DuckAssistBot+ User-agent: EchoboxBotUser-agent: FacebookBot…User-agent: magpie-crawler- User-agent: Meta-ExternalAgent- User-agent: Meta-ExternalFetcherUser-agent: NewsNow…User-agent: PanguBot- User-agent: PerplexityBot+ User-agent: PanscientUser-agent: PiplBot+ User-agent: PerplexityBotUser-agent: PetalBot+ User-agent: QuillBot -
Jan 2026
eBay Now blocked: ClaudeBot, CCBot, anthropic-ai, PerplexityBot and Amazonbot.
Show exactly what changed
#- # v24.2_COM_September_2025+ # Robots & Agent Policy+ #+ # Checkouts are strictly for human users.+ # * Automated scraping, buy-for-me agents, LLM-driven bots, or any+ # end-to-end flow that attempts to place orders without human review+ # is strictly prohibited.+ # * Unauthorized use of automated agents in checkout may result in+ # legal action under our User Agreement: https://www.ebay.com/help/policies/member-behaviour-policies/user-agreement?id=4259+ # * Approved enterprise integrations must use our official API and+ # comply with our API License Agreement: https://developer.ebay.com/join/api-license-agreement+ #+ # v24.5_COM_December_2025### DIRECTIVES ###…Disallow: /*_kw- Disallow: /motors/diy/blog/*Disallow: /*?maspect…Disallow: /adchoice+ Disallow: /buyer-preferences+ Disallow: /motors/diy/blog/*+ Disallow: /motors/fnd/*Allow: /ads.txt…Disallow: /g/api/+ Disallow: /gh/cartDisallow: /gh/collectbehaviorinfo…+ User-agent: CCBot+ Disallow: /++ User-agent: ChatGLM-Spider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /+
Dec 2025
-
Dec 2025
Amazon Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.
Show exactly what changed
Disallow: /gp/video/salp/i+ Disallow: /icdp/…Disallow: /++ User-agent: Copilot+ Disallow: /++ User-agent: Ai2Bot-Dolma+ Disallow: /++ User-agent: Google-CloudVertexBot+ Disallow: /++ User-agent: meta-externalfetcher+ Disallow: /++ User-agent: iaskspider+ Disallow: /++ User-agent: wpbot+ Disallow: /++ User-agent: aiHitBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: img2dataset+ Disallow: /++ User-agent: VelenPublicWebCrawler+ Disallow: /++ User-agent: YouBot -
Dec 2025
LinkedIn Now allowed: OAI-SearchBot.
Show exactly what changed
(files too large to diff inline) -
Aug to Dec 2025
Stack Overflow Added
Content-Signal: search=yes. Added an RSLLicense:line pointing to licensing terms.Show exactly what changed
+ License: https://stackoverflow.com/license.xml+User-Agent: *+ Content-Signal: search=yesDisallow: /posts/…Disallow: /users/activity/?+ Disallow: /users/activity/show?*Disallow: /users/stats/…Disallow: /*/ivc/*- Disallow: /*?lastactivity+ Disallow: /*?*lastactivityDisallow: /users/login/global/request/…Disallow: /questions/*/answer/submit+ Disallow: /questions/*?*rq=*Disallow: /tags/*+*…Disallow: /_/*- Allow: /+ Disallow: /click?*+ Disallow: /filter/tags?*+ Disallow: /landing/r/digest?*+ Disallow: /admin/*+ Disallow: /impression.gif?*md=*+ Disallow: /markup_single.js?*+ Disallow: /snippet.js?*Disallow: /misc/wp-admin…Disallow: /misc/answersdump+ Disallow: /?*+ Disallow: /questions?*tagnames=*+ Disallow: /questions?*tagMode=*+ Disallow: /questions?*pagesize=*+ Disallow: /questions?*edited=*+ Disallow: /questions?*days=*+ Disallow: /questions/tagged/*?tagnames=*+ Disallow: /questions/tagged/*?tagMode=*+ Disallow: /questions/tagged/*?pagesize=*
Nov 2025
-
Nov 2025
IHG Hotels Stopped naming OAI-SearchBot, PerplexityBot and ChatGPT-User; access unchanged.
Show exactly what changed
# AEM 2020- # Updated 2/26/2025 KA+ # Updated 11/6/2025 KA…Disallow: */hotels/*/find-hotels/select-roomrate*+ Disallow: *.html.html*+ Disallow: *bvstate*+ Disallow: /etc/clientlibs/*+ Disallow: /etc.clientlibs/*+ Disallow: *_=176*+ Disallow: /gs-json/getTrackingJSON/*…Disallow: */hotels/*/*/hotelDirectoryController- Disallow: *reviews/write-a-reviewDisallow: */hotels/*/*/reservation/singlereservationsummary…Disallow: */hotels/*/*/reservationhome+ Disallow: *reviews/*/write-a-review…+ User-agent: YisouSpider+ Disallow: */ihg-kernel-ssi/*++ User-agent: Sogou web spider+ Disallow: */ihg-kernel-ssi/*+User-agent: OmniExplorer_Bot…- # AI bots- User-agent: OAI-SearchBot- Allow: /-- User-agent: ChatGPT-User- Allow: /-- User-agent: PerplexityBot -
Aug to Nov 2025
Zocdoc Named GPTBot, OAI-SearchBot and ChatGPT-User explicitly, without changing what they may reach.
Show exactly what changed
- Sitemap: https://www.zocdoc.com/sitemaps/search-listing-sitemap.xml- Sitemap: https://www.zocdoc.com/sitemaps/profiles-sitemap.xml- Sitemap: https://www.zocdoc.com/sitemaps/updated-triple-threats-sitemap.xml- Sitemap: https://www.zocdoc.com/sitemaps/directory-sitemap.xml- Sitemap: https://www.zocdoc.com/sitemaps/keyword-listings-sitemap.xml- Sitemap: https://www.zocdoc.com/sitemaps/practices-sitemap.xml+ User-agent: GPTBot+ Allow: /++ User-agent: OAI-SearchBot+ Allow: /++ User-agent: ChatGPT-User+ Allow: /+User-agent: *Disallow: /syndication/v1/initialization+ Disallow: /eventslogging/v1/event+ Disallow: /eventslogger/v2/logeventsDisallow: /humanrecording…Disallow: /about/understanding-your-bill+ Disallow: /patient/check-in/*Allow: /Allow: /about/newpricing- Allow: /*referrerType=-
Oct 2025
-
Oct 2025
Cloudflare Docs Added
Content-Signal: ai-train=yes, search=yes, ai-input=yes.Show exactly what changed
+ # As a condition of accessing this website, you agree to+ # abide by the following content signals:++ # (a) If a content-signal = yes, you may collect content+ # for the corresponding use.+ # (b) If a content-signal = no, you may not collect content+ # for the corresponding use.+ # (c) If the website operator does not include a content+ # signal for a corresponding use, the website operator+ # neither grants nor restricts permission via content signal+ # with respect to the corresponding use.++ # The content signals and their meanings are:++ # search: building a search index and providing search+ # results (e.g., returning hyperlinks and short excerpts+ # from your website's contents). Search does not include+ # providing AI-generated search summaries.+ # ai-input: inputting content into one or more AI models+ # (e.g., retrieval augmented generation, grounding, or other+ # real-time taking of content for generative AI search+ # answers).+ # ai-train: training or fine-tuning AI models.++ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS+ # RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN+ # UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS+ # IN THE DIGITAL SINGLE MARKET.+User-agent: *+ Content-Signal: ai-train=yes, search=yes, ai-input=yes++ Allow: /Disallow: /client-ip-geolocationDisallow: /plans/+ Disallow: /constellation+ Disallow: /cdn-cgi/+ Disallow: /email-security/- Sitemap: https://developers.cloudflare.com/sitemap.xml -
Oct 2025
eBay Now blocked: Bytespider.
Show exactly what changed
#- # v16.4_COM_December_2022+ # v24.2_COM_September_2025### DIRECTIVES ###…Disallow: /*_kw+ Disallow: /motors/diy/blog/*Disallow: /*?maspect…Disallow: /*src=urllib+ Disallow: /?SSOWebDispatcher&tg=web&ru=Disallow: /act/+ Disallow: /adchoice+ Allow: /ads.txtDisallow: /ads+ Disallow: /adsrepositoryDisallow: /afs/Disallow: /atc/- Disallow: /antika-sanat/- Disallow: /bebek-anne/- Disallow: /bilgisayar-tablet/- Disallow: /cep-telefonu-ve-aksesuar/- Disallow: /ev-bahce/- Disallow: /giyim-aksesuar/- Disallow: /muzik-plak-enstruman/- Disallow: /otomobil-motor-aksesuar/- Allow: /ads.txt+ Disallow: /adell-mutfak+ Disallow: /ahu-kadin+ Disallow: /antika-sanat+ Disallow: /arama+ Disallow: */b2ajax/all-filter+ Disallow: /bebek-anne+ Disallow: /bilgisayar-tablet+ Disallow: /cep-telefonu-ve-aksesuar+ Disallow: /delicacy+ Disallow: /ev-bahce+ Disallow: /giyim-aksesuar+ Disallow: /muzik-plak-enstruman+ Disallow: /otomobil-motor-aksesuar -
Oct 2025
The New York Times Now allowed: Amazonbot.
Show exactly what changed
Disallow: /athletic/forgot-password/*- Allow: /athletic/forgot-password/$Disallow: /athletic/forgot-password2/*- Allow: /athletic/forgot-password2/$Disallow: /athletic/amp-social-login*…Disallow: /wirecutter/*?merchant=+ Disallow: /wirecutter/out/Disallow: /search…Allow: /svc/news/v3/all/pshb.rss+ Allow: /wirecutter/reviews/*?*utm_source=+ Allow: /wirecutter/blog/*?*utm_source=+ Allow: /wirecutter/lists/*?*utm_source=+ Allow: /wirecutter/gifts/*?*utm_source=…- User-agent: Amazonbot- Disallow: /-User-agent: anthropic-ai…Allow: /wirecutter/+ Allow: /athletic/…Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml+ Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xmlSitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz…- -
Oct 2025
The New Yorker Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.
Show exactly what changed
Disallow: /com.condenast/yv8*+ Disallow: /newsletters/…Disallow: /user-context?referrer+ Disallow: /newsletters/…Disallow: /services.min.js?_=+ Disallow: /newsletters/User-agent: Google-InspectionTool+ User-agent: TwitterbotAllow: /- User-agent: CCBot- Disallow: /-User-agent: Google-Extended- Disallow: /-+ User-agent: Google-CloudVertexBot+ User-agent: GoogleOther+ User-agent: Applebot-ExtendedUser-agent: Amazonbot- Disallow: /-- User-agent: FacebookBot- Disallow: /-+ User-agent: meta-webindexerUser-agent: meta-externalagent- Disallow: /-- User-agent: anthropic-ai- Disallow: /-- User-agent: Claude-Web- Disallow: /- -
Oct 2025
TikTok Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.
Show exactly what changed
User-agent: Bytespider+ User-agent: GPTBot+ User-agent: OAI-SearchBot+ User-agent: anthropic-ai+ User-agent: ClaudeBot+ User-agent: PerplexityBot+ User-agent: Google-Extended+ User-agent: Applebot-Extended+ User-agent: meta-externalagent+ User-agent: DuckAssistBot+ User-agent: ChatGPT-User+ User-agent: Gemini-Deep-Research+ User-agent: GoogleAgent-Mariner+ User-agent: Google-NotebookLM+ User-agent: CCBot+ User-agent: Claude-User+ User-agent: Claude-SearchBot+ User-agent: Perplexity-User+ User-agent: AI2Bot+ User-agent: MistralAI-UserDisallow: /…Disallow: */directory/+ Disallow: /search/video?+ Disallow: /search/user?q=+ Disallow: /shop/view/product/+ Disallow: /sgtm/g/collect+ Disallow: /api/share/settings+ Disallow: /api/recommend/embed_videos+ Disallow: /discover/trending/detail/+ Disallow: /search?+ Disallow: /search/live? -
Oct 2025
Tripadvisor Now blocked: GPTBot.
Show exactly what changed
#- # Run - don't crawl - to apply to join TripAdvisor's elite SEO team+ # Run - don't crawl - to apply to join Tripadvisor's elite SEO team#…Sitemap: https://www.tripadvisor.com/business/sitemap.xml- User-Agent: PetalBot- Disallow: /-- User-Agent: SentiBot- Disallow: /-User-Agent: magpie-crawler- Disallow: /-- User-Agent: Bytespider- Disallow: /-- User-Agent: ClaudeBot- Disallow: /-User-Agent: Google-Extended- Disallow: /-+ User-Agent: ClaudeBot+ User-Agent: PetalBot+ User-Agent: Bytespider+ User-Agent: Google-CloudVertexBotUser-Agent: Applebot-Extended+ User-Agent: SentiBot+ User-Agent: GPTBotDisallow: /…User-Agent: *+ User-Agent: Gemini-Deep-Research+ User-Agent: OAI-SearchBot+ User-Agent: ChatGPT-User/2.0+ User-Agent: ChatGPT-UserUser-Agent: applebotUser-Agent: PerplexityBot -
Oct 2025
WIRED Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.
Show exactly what changed
Disallow: /partners/- Disallow: /sponsored/Disallow: /video/right_rail*…Disallow: /com.condenast/yv8*+ Disallow: */testing-products-…Disallow: /preview/+ Disallow: */testing-products-…Disallow: /services.min.js?_=+ Disallow: */testing-products-…- User-agent: CCBot- Disallow: /-User-agent: Google-Extended- Disallow: /-+ User-agent: Google-CloudVertexBot+ User-agent: GoogleOther+ User-agent: Applebot-ExtendedUser-agent: Amazonbot- Disallow: /-- User-agent: FacebookBot- Disallow: /-+ User-agent: meta-webindexerUser-agent: meta-externalagent- Disallow: /-- User-agent: anthropic-ai- Disallow: /-- User-agent: Claude-Web
Sep 2025
-
Jun to Sep 2025
Amazon Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.
Show exactly what changed
Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: GoogleAgent-Mariner+ Disallow: /++ User-agent: GoogleAgent-Shopping+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: meta-externalagent+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: Scrapy+ Disallow: /++ User-agent: PetalBot+ Disallow: /++ User-agent: Devin+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: AI2Bot+ Disallow: /++ User-agent: Gemini-Deep-Research+ Disallow: /++ User-agent: PanguBot+ Disallow: / -
Sep 2025
Chewy Now blocked: Amazonbot.
Show exactly what changed
#- # Check out our current openings at https://careers.chewy.com/us/en/search-results?keywords=SEO+ # Check out our current openings at https://careers.chewy.com/us/en/search-results+ Sitemap: https://www.chewy.com/app/sitemap/behijkmoqrrtttvvy-sitemap_index.xmlUser-agent: *…Disallow: /app/deals?- Disallow: /app/login?Disallow: /app/account…Disallow: /app/dp/*+ Disallow: /api/login+ Disallow: /app/login+ Disallow: /app/register+ Disallow: /app/resetpassword/+ Disallow: /auth/+ Disallow: /ca/app/login+ Disallow: /ca/app/write-review+ Disallow: /ca/app/register+ Disallow: /ca/app/api+ Disallow: /ca/app/resetpassword+ Disallow: /ca/app/account+ Disallow: /ca/ca/+ Disallow: /ca/auth/+ Disallow: /ca/api/login+ Disallow: *adId=*+ Disallow: /vet-care/scheduler+ Disallow: /api/event/+ Disallow: login*targetUrl*+ Disallow: x_tr*+ Disallow: /shops/*+ User-agent: Amazonbot+ Disallow: / -
Sep 2025
Craigslist Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.
Show exactly what changed
- User-agent: Amazonbot- User-agent: anthropic-ai- User-agent: Applebot-Extended- User-agent: Bytespider- User-agent: CCBot- User-agent: ClaudeBot- User-agent: Claude-Web- User-agent: cohere-aiUser-agent: Diffbot -
Sep 2025
Expedia Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, Claude-User and Perplexity-User explicitly, without changing what they may reach.
Show exactly what changed
disallow: /Hotel-Search+ Disallow: /hotel-searchdisallow: /ugc/urs/api/hotelreviews/hotel/HOTELID/…disallow: /Checkout+ disallow: /checkoutdisallow: /HotelBookingConfirmation…disallow: /things-to-do/*/search?+ disallow: /*/things-to-do/search#cruise…disallow: /user/createaccount+ Disallow: /signup?*+ Disallow: /login?*disallow: /things-to-do/reviews-external$+ #test-seo+ Disallow: /api/uisprime/track+ Disallow: /trvl-px/+ Disallow: /egcs/v2/collect+ Disallow: /api/ucs/shortlist/+ Disallow: /api/v4/typeahead/#…user-agent: AdsBot-Google- allow: /+ Allow: /+ #test-seo+ Disallow: /api/uisprime/track+ Disallow: /trvl-px/+ Disallow: /egcs/v2/collect+ Disallow: /api/ucs/shortlist/+ Disallow: /api/v4/typeahead/…allow: /++ # Allow Agentic Search+ User-agent: OAI-SearchBot+ User-agent: ChatGPT-User -
Sep 2025
Hotels.com Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User explicitly, without changing what they may reach.
Show exactly what changed
Disallow: /hotel/hoteldata.html+ #test-seo+ Disallow: /api/uisprime/track+ Disallow: /trvl-px/+ Disallow: /egcs/v2/collect+ Disallow: /api/ucs/shortlist/+ Disallow: /api/v4/typeahead/++ User-Agent: AdsBot-Google+ #test-seo+ Disallow: /api/uisprime/track+ Disallow: /trvl-px/+ Disallow: /egcs/v2/collect+ Disallow: /api/ucs/shortlist/+ Disallow: /api/v4/typeahead/…Disallow: /login+ Disallow: /signup?*+ Disallow: /login?*Disallow: /mobile/search.html*…Disallow: /Hotel-Search+ Disallow: /hotel-searchDisallow: /ugc/urs/api/hotelreviews/hotel/HOTELID/…Disallow: /Checkout+ Disallow: /checkoutDisallow: /HotelBookingConfirmation…Disallow: /go/hotel/info/+ #test-seo+ Disallow: /api/uisprime/track+ Disallow: /trvl-px/+ Disallow: /egcs/v2/collect+ Disallow: /api/ucs/shortlist/+ Disallow: /api/v4/typeahead/+ #car+ Disallow: /carsearch+ Disallow: /CarCheckout -
Sep 2025
IKEA Now allowed: GPTBot, Google-Extended and CCBot.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-- User-agent: Google-Extended- Disallow: /-- User-agent: CCBot- Disallow: /-User-Agent: AdsBot-Google…Disallow: /*-fragment-recursive/+ Disallow: */fragments/*+ Disallow: /*-fragments/*+ Disallow: */energy-fragments/*Disallow: /products/container.html…Disallow: */returns/*?*&ref=+ Disallow: */hri-fragments/*+ Disallow: */business-store/*+ Disallow: */rewards-experience/*+ Disallow: */energy-insights/*Allow: */ces/web/modules/*.css -
Sep 2025
LinkedIn Named GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent, anthropic-ai and 7 more explicitly, without changing what they may reach.
Show exactly what changed
(files too large to diff inline) -
Sep 2025
Medium Now blocked: Meta-ExternalAgent and Amazonbot.
Show exactly what changed
Disallow: /*/*source=- Allow: /_/Allow: /_/api/users/*/meta…Allow: /_/api/posts/*/related- Sitemap: https://medium.com/sitemap/sitemap.xml+ User-Agent: AmazonbotUser-Agent: Applebot-Extended…User-Agent: GPTBot+ User-Agent: meta-externalagentDisallow: /…Allow: /verified-authors+ Sitemap: https://medium.com/sitemap/sitemap.xmlLicense: https://medium.com/license.xml -
Sep 2025
Pinterest Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.
Show exactly what changed
Allow: /business/create/- Allow: /pin/create/Allow: /resource/*/get/…Disallow: /*/*/more_ideas/*+ Disallow: /*/?*amp_client_id*Disallow: /*/?z=1+ Disallow: /*/__wishlist__/*Disallow: /*/_activities/*…Disallow: /age-appeal/*+ Disallow: /amp/*Disallow: /appealed-pin/…Disallow: /password/reset/*- Disallow: /pin-builder/+ Disallow: /pin-builder/*Disallow: /pin-communities/*…Disallow: /pin/*/visual-search+ Disallow: /pin/*/visual-shop+ Disallow: /pin/create/*Disallow: /pin/find/…Disallow: /upload-lens-image+ Disallow: /upload-shuffle-draftDisallow: /v1/urls/count.json…- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_high_engagement_tier_0_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_v_expansion_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_gsc_terms_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/video_hub_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/promoted_today_article_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_expansion_v2_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/promoted_idea_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_product_pin_sitemap_www.pinterest.com.xml -
Sep 2025
Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
Show exactly what changed
Disallow: /newsletters/politico-toplines+ Disallow : /*?tab=most-read*+ Disallow : /*?ml=*+ Disallow : /*?hp=*+ Disallow : /*?video=*+ Disallow : /*?nid=*+ Disallow : /*?cid=*+ Disallow : /*?source=*+ Disallow : /*?slide=*+ Disallow : /*?category=*+ Disallow : /*?p=*+ Disallow : /*?print=*+User-agent: Googlebot-News…+ User-agent: AddSearchBot+ Disallow: /++ User-agent: AI2Bot+ Disallow: /++ User-agent: Ai2Bot-Dolma+ Disallow: /++ User-agent: aiHitBot+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: Andibot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Applebot+ Disallow: / -
Jul to Sep 2025
Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
- User-agent: *+ # Allowed for archiving+ # =====================+ User-Agent: *+ Allow: /$Disallow: / -
Sep 2025
Temu Named GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 2 more explicitly, without changing what they may reach.
Show exactly what changed
User-agent: Bingbot+ User-agent: Google-Extended+ User-agent: GPTBot+ User-agent: OAI-SearchBot+ User-agent: ChatGPT-User+ User-agent: PerplexityBot+ User-agent: ClaudeBot+ User-agent: Claude-User+ User-agent: Claude-SearchBotDisAllow: /*oauth_code=…Disallow: /*-m-*?goods_id=*- DisAllow: /pinduoduo-select*- DisAllow: /*/pinduoduo-select*DisAllow: /blog/*…User-agent: Applebot- User-agent: facebookexternalhitUser-agent: Pinterestbot…Disallow: /*-g-*html?*_oak_page_source=*- DisAllow: /pinduoduo-select*- DisAllow: /*/pinduoduo-select*DisAllow: /blog/*…+ User-agent: facebookexternalhit+ DisAllow: /*oauth_code=+ DisAllow: /*oauth_token=+ DisAllow: /bgp_user_profile.html+ DisAllow: /bgp_personal.html+ DisAllow: /bgp_collect.html+ DisAllow: /bgp_footprint.html+ DisAllow: /bgst_order_success.html+ DisAllow: /bgst_payment_detail.html+ DisAllow: /bgst_express.html+ DisAllow: /personal.html+ DisAllow: /collect.html+ DisAllow: /shopping_cart.html+ DisAllow: /bgp_similar_rec.html -
Sep 2025
The Atlantic Now blocked: Perplexity-User.
Show exactly what changed
Disallow: /+ Allow: /feed/*…+ User-agent: Perplexity-User+ Disallow: /++ User-agent: Quora-Bot+ Disallow: /+User-agent: Scrapy -
Sep 2025
The Economist Now blocked: Perplexity-User.
Show exactly what changed
- # Specific robot directives:+ # Sitemaps- # Description : Google AdSense delivers advertisements to a broad network of affiliated sites.- # A robot analyses the pages that display the ads in order to target the ads to the page content.- User-agent: Mediapartners-Google*- Allow: /+ Sitemap: https://www.economist.com/sitemap.xml+ Sitemap: https://www.economist.com/googlenews.xml- # Description : The Grapeshot crawler is an automated robot that visits pages to examine and analyse the content.- # This adds an exception to crawl delay while preserving disallows.- User-agent: grapeshot- Allow: /+ # Bot Directives- # GPTBot is OpenAI’s web crawlerUser-agent: GPTBot…- # Allows us to block Google's bot BardUser-agent: Google-Extended…- # ChatGPT-User is OpenAI’s web crawlerUser-agent: ChatGPT-User…- # Common Crawl botUser-agent: CCBot…- # PiplBot is PiplBot's web crawlerUser-agent: PiplBot…- # anthropic-ai is Anthropic's web crawlerUser-agent: anthropic-ai… -
Sep 2025
USA Today Now blocked: MistralAI-User.
Show exactly what changed
+ User-agent: ia_archiver+ Disallow: /++ User-agent: archive.org_bot+ Disallow: /++ User-agent: ia_archiver-web.archive.org+ Disallow: /++ User-agent: aiHitBot+ Disallow: /++ User-agent: Amazon Kendra+ Disallow: /++ User-agent: Big Sur AI+ Disallow: /++ User-agent: Brandwatch+ Disallow: /++ User-agent: Bravest+ Disallow: /++ User-agent: ChatGPT Operator+ Disallow: /++ User-agent: Cotoyogi+ Disallow: /++ User-agent: DigitalOceanGenAICrawler+ Disallow: /++ User-agent: Echobot Bot+ Disallow: /++ User-agent: EchoboxBot+ Disallow: /+ -
Sep 2025
Vrbo Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User explicitly, without changing what they may reach.
Show exactly what changed
User-agent: *+ Allow: /search/*unitId=*+ Allow: /search/*noDates=*+ Allow: /search/*petIncluded=*+ Allow: /search/*adultsCount=*+ Allow: /search/*childrenCount=*+ Allow: /search/*arrival=*+ Allow: /search/*departure=*+ Allow: /search/*uni_id=*+ Allow: /*/list+Disallow: */units…Disallow: /auth/+ Disallow: /auth/ui/*+ Disallow: /*/auth/ui/*Disallow: */traveler/…Disallow: /en-au/accommodation+ #hotel-from-bex-robots.txt+ Disallow: /Hotel-Search+ Disallow: /hotel-search+ Disallow: /search?+ Disallow: /ugc/urs/api/hotelreviews/hotel/HOTELID/+ Disallow: /LastBooking/ElapsedTime+ Disallow: /*Hotel-Information?+ Disallow: /*Informacion-Hotel?+ Disallow: /HotelCheckout+ Disallow: /Checkout+ Disallow: /checkout+ Disallow: /HotelBookingConfirmation+ Disallow: /*classic=*+ Disallow: /*dateless=*+ Disallow: /*chkin=*+ Disallow: /*chkout=*+ Disallow: /*startDate=*+ Disallow: /*endDate=*+ Disallow: /*unitId=*+ Disallow: /*noDates=*+ Disallow: /*petIncluded=*
Aug 2025
-
Aug 2025
American Airlines Now allowed: GPTBot.
Show exactly what changed
User-agent: *- Disallow: /i18n/travelInformation/destinationInformation/venice/Disallow: /i18n/AAdvantage/…Disallow: /pubcontent/jbax+ Disallow: /content/images+ Disallow: /i18n/fragments+ Sitemap: https://www.aa.com/pubcontent/en_US/sitemap/sitemap.xml- User-agent: GPTBot- Disallow: / -
Aug 2025
Associated Press Now blocked: Claude-SearchBot and Claude-User.
Show exactly what changed
Disallow: *?prx_t=*- Disallow: /press-release/*- Disallow: /blaize*Disallow: /zephr*+ Disallow: /blaize*Disallow: /search?q=*…Disallow: /buyline-personal-finance/*+ Disallow: *?jw_start+ Disallow: *&jw_start+ Disallow: */config/newrelic/prod*…Disallow: /++ User-agent: Claude-User+ Disallow: /++ User-agent: Claude-SearchBot+ Disallow: / -
Apr to Aug 2025
BBC Now blocked: Amazonbot and Perplexity-User.
Show exactly what changed
- # version: ca16af0d01ecf92c8730f124f789053bfbf7293b-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml- Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml-+ # version: c6c2d0415f0c1d97565a01952c7b2146e2abe17f- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search -
Aug 2025
Bloomberg Now blocked: Meta-ExternalAgent, Bytespider and Amazonbot.
Show exactly what changed
- Sitemap: https://www.bloomberg.com/sitemap.xml- Sitemap: https://www.bloomberg.com/feeds/sitemap_news.xml- Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/bpol/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/businessweek/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/citylab/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/green/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/technology/sitemap_index.xml- Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_securities_index.xml- Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_profiles_company_index.xml+ # Major Company Bots+ User-agent: AmazonBot+ Disallow: /+ User-agent: AmazonAdBot+ Disallow: /+ User-agent: ByteSpider+ Disallow: /+ User-agent: Meta-ExternalAgent+ Disallow: /++ # Feed & Content Aggregators+ User-agent: Feedly+ Disallow: /+ User-agent: FeedlyBot+ Disallow: /+ User-agent: FeedlyApp+ Disallow: /+ User-agent: MWFeedParser+ Disallow: /++ # Analytics & Marketing+ User-agent: comscore+ Disallow: /+ User-agent: Comscore+ Disallow: /+ User-agent: HubSpot+ Disallow: /+ User-agent: hubspot+ Disallow: / -
Aug 2025
Glassdoor Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.
Show exactly what changed
#-+ #### Rules for ANY User-AgentUser-agent: *…Disallow: /about/privacy/- Disallow: /about/privacy/Disallow: /about/syndicationCenter/…Disallow: /browse/+ Disallow: /chatDisallow: /Compare/choose…Allow: /profile/joinNow_input.htm+ Allow: /member/profile/login+ Allow: /member/profile/joinNowDisallow: /Resume/user-profile/Disallow: /rss/*- Disallow: /Rungs/Disallow: /search/…Disallow: /surveys+ Disallow: /surveys/Disallow: /util/…Disallow: /developer/widget/builder/- Disallow: /hammer/- Disallow: /brand-viewsDisallow: /event-ingestion…Disallow: /sourcing$- Disallow: /searchsuggest$Disallow: /knowyourworth/- Disallow: /Scoop/*- Allow: /Scoop- Allow: /Scoop/index.htmDisallow: /Reviews/index.htm?…- Disallow: /geb/* -
Aug 2025
Los Angeles Times Now blocked: ClaudeBot and Bytespider.
Show exactly what changed
Disallow: /test/+ Disallow: /includes/+ Disallow: /_comments/Disallow: /topic/*?…Disallow: /test/+ Disallow: /_comments/Disallow: /topic/*?…+ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Diffbot+ Disallow: /+User-agent: discobot…- Sitemap: https://www.latimes.com/sitemap.xml+ User-agent: ImagesiftBot+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: omgilibot+ Disallow: /++ Sitemap: https://www.latimes.com/sitemaps/sitemap.xmlSitemap: https://www.latimes.com/news-sitemap.xml -
Aug 2025
OpenTable Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 10 more.
Show exactly what changed
- User-agent: dotbot- Disallow: /+ # By accessing OpenTable's website, you agree to OpenTable's Terms of Use: https://www.opentable.com/c/legal/terms-and-conditions/+ # Generated on 2025-07-30T08:50:31+ # Block all bots by defaultUser-agent: *+ Disallow: /++ # Allow specific bots+ User-agent: GooglebotDisallow: /_LBStatus…Disallow: /XML+ Disallow: /lolz-view-all/Disallow: /mwapi/slotlockDisallow: /dapi/fe/human+ Disallow: /dapi/fe/proxy/consumer-frontend/bucket-experiment+ Disallow: /dapi/fe/gql?optype=query&opname=Autocomplete+ Disallow: /dapi/fe/gql?optype=query&opname=HeaderUserProfile+ Disallow: /dapi/fe/gql?optype=query&opname=LocationPicker+ Disallow: /dapi/fe/gql?optype=query&opname=MultiSearchResultsFacets+ Disallow: /dapi/fe/gql?optype=query&opname=RestaurantMultiDayAvailability+ Disallow: /dapi/fe/gql?optype=query&opname=RestaurantsAvailability+ Disallow: /dapi/fe/gql?optype=query&opname=RestaurantsMessages+ Disallow: /dapi/fe/gql?optype=query&opname=UserTrackingOptIns+ Disallow: /dapi/fe/gql?optype=mutation&opname=RestaurantVisit+ Disallow: /dapi/fe/gql?optype=mutation&opname=SimpleVisit+ Disallow: /dapi/fe/gql?optype=query&opname=TopReviews+ Disallow: /dapi/fe/gql?optype=query&opname=NoAvailabilityRestaurants+ Disallow: /dapi/fe/gql?optype=query&opname=LandmarkRestaurantsLolz+ Disallow: /dapi/fe/gql?optype=query&opname=BestRestaurantsByCategory+ Disallow: /dapi/fe/gql?optype=query&opname=HomeModuleLists+ Disallow: /dapi/v1/sessionAllow: /img/restimages…+ User-agent: Googlebot-Image+ Disallow: /_LBStatus+ Disallow: /admin -
Aug 2025
Reuters Now blocked: Claude-SearchBot and Claude-User.
Show exactly what changed
+ # ADO-726772+ User-agent: Claude-User+ Disallow: /+ User-agent: Claude-SearchBot+ Disallow: /++ # ADO-727160+ User-agent: AliyunSecBot/Aliyun+ Disallow: /++ User-agent: Alibaba.Security.Heimdall+ Disallow: /++ # ADO-730264+ User-agent: Feedfetcher-Google+ Disallow: /+User-agent: *…Disallow: /test/- Disallow: /commentaryDisallow: /news/archive/commentary…Disallow: /practical-law-the-journal/search/- Disallow: /fr/- Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/- Disallow: /info-pages/custom-services-agreement-23-10/- Disallow: /info-pages/advertising-guidelines/Disallow: /pf/api/+ Disallow: /fr/+ Disallow: /it/+ Disallow: /es/+ Disallow: /pt/+ Disallow: /de/+ Disallow: /ar/+ Disallow: /account/subscribe/payment/ -
Aug 2025
The New York Times Now blocked: Claude-SearchBot, Claude-User and Perplexity-User.
Show exactly what changed
Disallow: /athletic*/signed-mp3-redirect-url/*+ Disallow: /athletic/embedded-interactive/*+ Disallow: /athletic/interactive/*Disallow: /card/panel/…Allow: /svc/spelling-bee- Allow: /svc/vertexAllow: /svc/wordle+ Allow: /svc/connections+ Allow: /svc/sudoku+ Allow: /svc/strands+ Allow: /svc/pipsDisallow: /video/embedded/*…+ User-agent: AliyunSecBot+ Disallow: /+User-agent: Amazonbot…+ User-agent: AudigentAdBot+ Disallow: /+User-agent: AwarioRssBot…+ User-agent: BLEXBot+ Disallow: /+User-agent: Bytespider…+ User-agent: Claude-SearchBot+ Disallow: /++ User-agent: Claude-User+ Disallow: /+User-agent: Claude-Web -
Aug 2025
The Washington Post Now allowed: Google-Extended.
Show exactly what changed
-User-agent: anthropic-ai…- User-agent: Google-Extended- Disallow: /-User-agent: heritrix
Jul 2025
-
Jul 2025
IMDb Now blocked: GPTBot, Google-Extended, CCBot and anthropic-ai.
Show exactly what changed
# robots.txt for https://www.imdb.com properties+ # See https://m.imdb.com/robots.txt for the mDot equivalent+User-agent: *Disallow: /OnThisDay+ Disallow: /*/OnThisDayDisallow: /ads/+ Disallow: /*/ads/Disallow: /ap/+ Disallow: /*/ap/Disallow: /mymovies/+ Disallow: /*/mymovies/Disallow: /r/+ Disallow: /*/r/Disallow: /register+ Disallow: /*/registerDisallow: /registration/+ Disallow: /*/registration/Disallow: /search/name-text+ Disallow: /*/search/name-textDisallow: /search/title-text+ Disallow: /*/search/title-textDisallow: /find+ Disallow: /*findDisallow: /find$+ Disallow: /*/find$Disallow: /find/+ Disallow: /*/find/Disallow: /tvschedule+ Disallow: /*/tvscheduleDisallow: /updates+ Disallow: /*/updatesDisallow: /watch/_ajax/option+ Disallow: /*/watch/_ajax/optionDisallow: /_json/video/mon+ Disallow: /*/_json/video/monDisallow: /_json/getAdsForMediaViewer/+ Disallow: /*/_json/getAdsForMediaViewer/Disallow: /list/ls*/_ajax+ Disallow: /*/list/ls*/_ajax -
Jul 2025
Medium Now allowed: Google-Extended. Added an RSL
License:line pointing to licensing terms.Show exactly what changed
User-Agent: GoogleOther- User-Agent: Google-ExtendedUser-Agent: GPTBot…Allow: /verified-authors+ License: https://medium.com/license.xml -
Jul 2025
NPR Now blocked: OAI-SearchBot.
Show exactly what changed
Disallow: /*/partials*- Disallow: /*?live=*- Disallow: /*?cacheKill=*- Disallow: /*?skipCache=*- Disallow: /*?forceCds=*- Disallow: /*?forceXml=*-- # Disallowing non-canonical URLs that contain query strings- User-agent: *Disallow: /*?*-- # Disallowing non-audience-facing URLs- User-agent: *Disallow: /proxy/*…Disallow: /get/*+ Disallow: /geolocation…+ User-agent: OAI-SearchBot+ Disallow: /+# Disallowing Google Bard and Vertex AI web crawlers…+ User-agent: PerplexityUser+ Disallow: /+# Allow Google Search Console for sitemap crawling…- # Explicitly allowing the TTD contextual crawler+ # Explicitly allowing the TTD contextual crawler for SponsorshipUser-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)…+ # Explicitly allowing the Google-Mediapartners crawler for Sponsorship+ User-agent: Mediapartners-Google+ Allow: / -
Jul 2025
Ticketmaster Now allowed: GPTBot.
Show exactly what changed
- User-agent: GPTBot- Disallow: / -
Jul 2025
USA Today Now blocked: AI2Bot, DuckAssistBot and Meta-ExternalFetcher.
Show exactly what changed
+ User-agent: AI2Bot+ Disallow: /++ User-agent: AhrefsBot+ Disallow: /++ User-agent: Ai2Bot-Dolma+ Disallow: /++ User-agent: Applebot+ Disallow: /++ User-agent: Arquivo-web-crawler+ Disallow: /++ User-agent: AspiegelBot+ Disallow: /++ User-agent: AwarioRssBot+ Disallow: /++ User-agent: AwarioSmartBot+ Disallow: /++ User-agent: Barkrowler+ Disallow: /++ User-agent: coccocbot-web+ Disallow: /++ User-agent: cohere-training-data-crawler+ Disallow: /++ User-agent: crawler4j+ Disallow: /++ User-agent: Crawlspace+ Disallow: /+ -
Jul 2025
Zillow Now allowed: GPTBot.
Show exactly what changed
User-agent: *++ Allow: /homes/for_sale/$+ Allow: /homes/for_sale/fore_lt/$+ Allow: /homes/fsbo/$+ Allow: /homes/for_sale/1_open/$+ Allow: /homes/new_homes/$+ Allow: /homes/coming_soon/$+ Allow: /homes/for_rent/$+ Allow: /homes/for_rent/condo,apartment_duplex_type/$+ Allow: /homes/for_rent/house,townhouse_type/$+ Allow: /homes/recently_sold/$+ Allow: /homes/apartment-communities/$++ Allow: /homes/for_sale/*_p/$+ Allow: /homes/for_sale/fore_lt/*_p/$+ Allow: /homes/fsbo/*_p/$+ Allow: /homes/for_sale/1_open/*_p/$+ Allow: /homes/new_homes/*_p/$+ Allow: /homes/coming_soon/*_p/$+ Allow: /homes/for_rent/*_p/$+ Allow: /homes/for_rent/condo,apartment_duplex_type/*_p/$+ Allow: /homes/for_rent/house,townhouse_type/*_p/$+ Allow: /homes/recently_sold/*_p/$+ Allow: /homes/apartment-communities/*_p/$++ Disallow: /api/+ Disallow: /homes/Disallow: /?url=…Disallow: /captcha/+ Disallow: /captchaPerimeterX/Disallow: /careers/apply/Disallow: /careers/jobs/+ Disallow: /chat/financeDisallow: /claiming/…Disallow: /homepage/HomepageVideoMetrics.htm- Disallow: /homes/*+- Disallow: /homes/*_rb
Jun 2025
-
Jun 2025
Craigslist Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.
Show exactly what changed
User-agent: CCBot- User-agent: ChatGPT-UserUser-agent: ClaudeBot…User-agent: Google-Extended- User-agent: GPTBotUser-agent: ImagesiftBot…User-agent: Meta-ExternalAgent- User-agent: OAI-SearchBotUser-agent: omgili
May 2025
-
May 2025
Associated Press Now blocked: Applebot-Extended.
Show exactly what changed
+ Sitemap: https://apnews.com/ap-sitemap.xml+ Sitemap: https://apnews.com/news-sitemap-content.xml+ Sitemap: https://apnews.com/hubs-sitemap-content.xml+ Sitemap: https://apnews.com/video-sitemap.xml++ # Disallow Rules+User-agent: CCBot…- Sitemap: https://apnews.com/ap-sitemap.xml- Sitemap: https://apnews.com/news-sitemap.xml- Sitemap: https://apnews.com/video-sitemap.xml-- # Disallow RulesUser-agent: anthropic-ai…- User-agent: Applebot+ User-agent: Applebot-ExtendedDisallow: /…Disallow: /++ User-agent: Timpibot+ Disallow: / -
May 2025
Financial Times Now blocked: Perplexity-User and Meta-ExternalFetcher.
Show exactly what changed
+ User-agent: GoogleOther+ Disallow: /+User-agent: anthropic-ai…+ User-agent: Perplexity-User+ Disallow: /+User-agent: Claude-Web…+ User-agent: Diffbot+ Disallow: /+User-agent: FacebookBot…+ User-agent: Meta-ExternalFetcher+ Disallow: /+ -
May 2025
Medium Now blocked: ClaudeBot, Google-Extended, Applebot-Extended and Bytespider.
Show exactly what changed
Disallow: /search?q=+ Disallow: /*/search?q=+ Disallow: /*/search/*?q=Disallow: /*/*source=…Sitemap: https://medium.com/sitemap/sitemap.xml+ User-Agent: Applebot-Extended+ User-Agent: Bytespider+ User-Agent: ClaudeBot+ User-Agent: FacebookBot+ User-Agent: GoogleOther+ User-Agent: Google-ExtendedUser-Agent: GPTBotDisallow: /+ Allow: /about+ Allow: /business+ Allow: /earn+ Allow: /gift+ Allow: /membership+ Allow: /partner-program+ Allow: /verified-authors -
May 2025
Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
- # 80legs- User-agent: 008- Disallow: /-- # 80legs' new crawler- User-agent: voltron- Disallow: /-- User-Agent: bender- Disallow: /my_shiny_metal_ass-- User-Agent: Gort- Disallow: /earth-- User-agent: MJ12bot- Disallow: /+ # Welcome to Reddit's robots.txt+ # Reddit believes in an open internet, but not the misuse of public content.+ # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.+ # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.+ # policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy- User-agent: PiplBot+ User-agent: *Disallow: /- User-Agent: *- Disallow: /*.json- Disallow: /*.json-compact- Disallow: /*.json-html- Disallow: /*.xml- Disallow: /*.rss- Allow: /r/*.rss- Disallow: /r/*/comments/*.rss- Disallow: /r/*/config/*.rss- Disallow: /r/*/wiki/*.rss- Disallow: /*.i- Disallow: /*.embed- Disallow: /*/comments/*?*sort=- Disallow: */comment/* -
May 2025
Stack Overflow Now allowed: ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and PerplexityBot.
Show exactly what changed
Allow: /- User-agent: Yahoo Pipes 1.0- Disallow: /- User-agent: 008- Disallow: /- User-agent: voltron- Disallow: /- User-agent: Bytespider- Disallow: /- User-Agent: PerplexityBot- Disallow: /- User-agent: meta-externalagent- Disallow: /- User-agent: Applebot-Extended- Disallow: /- User-agent: cohere-ai- Disallow: /- User-agent: Omgilibot- Disallow: /- User-agent: Omgili- Disallow: /- User-agent: magpie-crawler- Disallow: /- User-agent: TurnitinBot- Disallow: /- User-agent: DataForSeoBot- Disallow: /- User-agent: Twitterbot- Disallow: /- User-agent: peer39_crawler- Disallow: /- User-agent: scrapy- Disallow: /- User-agent: ClaudeBot- Disallow: /- User-agent: Claude-Web- Disallow: /- User-agent: CCBot- Disallow: /- User-agent: Diffbot -
May 2025
The Verge Now allowed: Amazonbot.
Show exactly what changed
- User-agent: Amazonbot- Disallow: /-User-agent: Applebot-Extended -
Sep 2024 to May 2025
Threads Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
+ # Notice: Collection of data on Threads through automated means is+ # prohibited unless you have express written permission from Threads+ # and may only be conducted for the limited purpose contained in said+ # permission.+ # All authorized user-agents listed on this page must comply with Meta’s+ # Automated Data Collection Terms available at:+ # https://www.facebook.com/legal/automated_data_collection_terms+ User-agent: Amazonbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ User-agent: PetalBot+ Disallow: /++ User-agent: uptimerobot+ Disallow: /++ User-agent: viberbot+ Disallow: /++ User-agent: YaK+ Disallow: /++ User-agent: Yandex+ Disallow: /+ -
May 2025
USA Today Now blocked: Claude-SearchBot and Claude-User.
Show exactly what changed
+ User-agent: Claude-User+ Disallow: /++ User-agent: Claude-SearchBot+ Disallow: /+User-agent: *…Disallow: /gciaf/+ Disallow: /gcdn/gciaf/+ Disallow: /dcc/+ Disallow: /gcdn/dcc/+ Disallow: /dc/+ Disallow: /gcdn/dc/+ Disallow: /dcjs/+ Disallow: /gcdn/dcjs/+ Disallow: /content-queries/+ Disallow: /zxc/+ Disallow: /money/homefront/Disallow: /exp-cruise…Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xml- Sitemap: https://www.usatoday.com/money/homefront/sitemap.xmlSitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml
Apr 2025
-
Apr 2025
CNN Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot, OAI-SearchBot, Amazonbot, DuckAssistBot and Meta-ExternalFetcher.
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml+ Sitemap: https://www.cnn.com/sitemap/live-story.xml+ Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml+ User-agent: AI2Bot+ User-agent: Ai2Bot-Dolma+ User-agent: AmazonbotUser-agent: anthropic-ai+ User-agent: Applebot+ User-agent: Applebot-ExtendedUser-agent: AwarioRssBotUser-agent: AwarioSmartBot+ User-agent: Brightbot 1.0User-agent: Bytespider…User-agent: cohere-ai+ User-agent: cohere-training-data-crawler+ User-agent: CrawlspaceUser-agent: DataForSeoBotUser-agent: Diffbot+ User-agent: DuckAssistBotUser-agent: FacebookBot+ User-agent: FriendlyCrawlerUser-agent: GPTBotUser-agent: Google-Extended+ User-agent: GoogleOther+ User-agent: GoogleOther-Image+ User-agent: GoogleOther-Video+ User-agent: iaskspider/2.0+ User-agent: ICC-Crawler+ User-agent: img2dataset+ User-agent: ISSCyberRiskCrawler+ User-agent: ImagesiftBot+ User-agent: Kangaroo BotUser-agent: magpie-crawler+ User-agent: Meta-ExternalAgent+ User-agent: Meta-ExternalFetcherUser-agent: NewsNowUser-agent: news-please+ User-agent: OAI-SearchBotUser-agent: omgili -
Apr 2025
The New York Times Now blocked: DuckAssistBot and Meta-ExternalFetcher.
Show exactly what changed
Disallow: /athletic/rss-feed/+ Disallow: /athletic/*?*rss=1Disallow: /athletic/global-color-test.php…Disallow: /athletic/feed-test/+ Disallow: /athletic*/signed-mp3-redirect-url/*Disallow: /card/panel/+ Disallow: /panel/Disallow: /puzzles/leaderboards/invite/*…Disallow: /wirecutter/*?l=+ Disallow: /wirecutter/*?merchant=Disallow: /search+ Disallow: /subscription/*?*source=+ Disallow: /subscription/*?*onboarded=Disallow: /*?*smid=…Disallow: /athletic*/?source=*- Disallow: /athletic/live-blogs/*embed=1*+ Disallow: /athletic/*?*embed=1…+ User-agent: DuckAssistBot+ Disallow: /+User-agent: FacebookBot…+ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: Google-CloudVertexBot+ Disallow: /+ Allow: /wirecutter/+User-agent: Google-Extended…+ User-agent: ImagesiftBot -
Apr 2025
The Washington Post Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.
Show exactly what changed
+ User-agent: Amazonbot+ Disallow: /+ Allow: /wp-apps/imrs.php+# Disallow Rules…- User-agent: Amazonbot- Disallow: /…- User-agent: ChatGPT-User- Disallow: /-User-agent: Claude-Web…+ User-agent: cohere-training-data-crawler+ Disallow: /+User-agent: DataForSeoBot…- User-agent: GPTBot- Disallow: /-User-agent: heritrix…- User-agent: magpie-crawler- Disallow: /-User-agent: meta-externalagent…- User-agent: OAI-SearchBot- Disallow: /
Mar 2025
-
Mar 2025
BBC Now blocked: Meta-ExternalAgent and OAI-SearchBot.
Show exactly what changed
- # version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68+ # version: ca16af0d01ecf92c8730f124f789053bfbf7293b- # HTTPS www.bbc.com+ # HTTP www.bbc.com- User-agent: *- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/sport/sitemap.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/afrique/sitemap.xml- Sitemap: https://www.bbc.com/arabic/sitemap.xml- Sitemap: https://www.bbc.com/bengali/sitemap.xml- Sitemap: https://www.bbc.com/burmese/sitemap.xml- Sitemap: https://www.bbc.com/gahuza/sitemap.xml- Sitemap: https://www.bbc.com/hausa/sitemap.xml- Sitemap: https://www.bbc.com/hindi/sitemap.xml- Sitemap: https://www.bbc.com/indonesia/sitemap.xml- Sitemap: https://www.bbc.com/mundo/sitemap.xml- Sitemap: https://www.bbc.com/pashto/sitemap.xml- Sitemap: https://www.bbc.com/persian/sitemap.xml- Sitemap: https://www.bbc.com/portuguese/sitemap.xml- Sitemap: https://www.bbc.com/russian/sitemap.xml- Sitemap: https://www.bbc.com/swahili/sitemap.xml- Sitemap: https://www.bbc.com/tajik/sitemap.xml- Sitemap: https://www.bbc.com/turkce/sitemap.xml- Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml- Sitemap: https://www.bbc.com/ukrainian/sitemap.xml- Sitemap: https://www.bbc.com/urdu/sitemap.xml- Sitemap: https://www.bbc.com/uzbek/sitemap.xml- Sitemap: https://www.bbc.com/vietnamese/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml- Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml+ User-agent: Googlebot -
Mar 2025
Bloomberg Now blocked: anthropic-ai, PerplexityBot and ChatGPT-User.
Show exactly what changed
Disallow: /tc/search/- Disallow: /en/news/thpDisallow: /subscriptions/group/manage/…Disallow: /explore/+ Disallow: /press-releases/+ Disallow: /artemis/+ Disallow: /sessions-publisher/+ User-agent: Google-Extended+ Disallow: /+User-agent: Mediapartners-Google…+ # Development Tools+ User-agent: Python-urllib+ Disallow: /+ User-agent: python-requests+ Disallow: /+ User-agent: Python-http-client+ Disallow: /++ # AI/LLM Bots+ User-agent: ChatGPT-User+ Disallow: /+ User-agent: GPTBot+ Disallow: /+ User-agent: Claude-Web+ Disallow: /+ User-agent: anthropic-ai+ Disallow: /+ User-agent: PerplexityBot+ Disallow: /+ User-agent: YouBot+ Disallow: /++ # Google Services+ User-agent: Google-Apps+ Disallow: / -
Mar 2025
Capital One Named ClaudeBot and Amazonbot explicitly, without changing what they may reach.
Show exactly what changed
Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?…Disallow: /error+ Disallow: /search/?+ User-agent: ClaudeBot+ User-agent: Abonti+ User-agent: Aboundexbot+ User-agent: Acoon+ User-agent: Allloadin Favicon Bot+ User-agent: Blekkobot+ User-agent: Daum+ User-agent: Gobuster+ User-agent: semaltbot+ User-agent: Tarmot Gezgin+ User-agent: tchelebi+ User-agent: UkrNet Mail Proxy+ User-agent: AmazonAdBot -
Mar 2025
Facebook Named PerplexityBot explicitly, without changing what it may reach.
Show exactly what changed
# permission.- # See: http://www.facebook.com/apps/site_scraping_tos_terms.php+ # All authorized user-agents listed on this page must comply with Meta’s+ # Automated Data Collection Terms available at:+ # https://www.facebook.com/legal/automated_data_collection_termsUser-agent: Amazonbot…+ User-agent: PerplexityBot+ Disallow: /+User-agent: PetalBot…+ User-agent: uptimerobot+ Disallow: /+User-agent: viberbot…+ User-agent: Yandex+ Disallow: /+User-agent: Applebot…- User-agent: LinkedInBot+ User-agent: Googlebot-VideoDisallow: /*/plugins/*…- User-agent: msnbot+ User-agent: Googlebot-NewsDisallow: /*/plugins/*…- User-agent: Naverbot+ User-agent: Google-InspectionToolDisallow: /*/plugins/*… -
Mar 2025
IHG Hotels Named OAI-SearchBot, PerplexityBot and ChatGPT-User explicitly, without changing what they may reach.
Show exactly what changed
- #robots.txt for https://www.ihg.com/- #AEM 2020- #Revised 12-16-19 CH+ # robots.txt for https://www.ihg.com/+ # AEM 2020+ # Updated 2/26/2025 KA- User-agent: *- Disallow: /*/admin/*- Disallow: /*/reservation/searchresult*- Disallow: /*/reservation/hotelsearchresult*- Disallow: /*/reservation/roomrate*- # Disallow: /hotels/*/*/find-hotels/*- Disallow: /hotels/*/global/corporate/*- Disallow: /hotels/*/global/offers/airline/airline- Disallow: /hotels/*/friendsfamily- Disallow: /friendsandfamily- Disallow: /employees- Disallow: /employee+ User-agent: *+ Disallow: /*/admin/*+ Disallow: /*/reservation/searchresult*+ Disallow: /*/reservation/hotelsearchresult*+ Disallow: /*/reservation/roomrate*+ Disallow: /hotels/*/global/corporate/*+ Disallow: /hotels/*/global/offers/airline/airline+ Disallow: /hotels/*/friendsfamily+ Disallow: /friendsandfamily+ Disallow: /employees+ Disallow: /employeeDisallow: /staypreferences/*- Allow: /hotels/us/en/find-hotels/offers/last-minute-dealsDisallow: /*/hoteldetail/printfactsheet- Disallow: /sandbox/*+ Disallow: */sandbox*+ Disallow: */hotels/*/find-hotels/select-roomrate*…- #Sitemap: http://www.ihg.com/brands-sitemap-index.xml -
Mar 2025
Instacart Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 14 more.
Show exactly what changed
### Instacart's Terms of Service https://www.instacart.com/terms- User-Agent: ChatGPT-User- Allow: /store/partner_recipes+ ### Last Updated 2-28-2025- User-Agent: Twitterbot- Allow: /store/list/*+ ### Rules for Google search ###+ User-Agent: Googlebot+ Allow: /+ Disallow: /accounts/unlock+ Disallow: /api/+ Disallow: /ex/+ Disallow: /store/orders/+ Disallow: /redeem_coupon_token+ Disallow: /store/order_changes/+ Disallow: /store/*/browse_departments*+ Disallow: /store/*/ng/l/*+ Disallow: /store/referrals*+ Disallow: /store/item*+ Disallow: /store/pickup/status/*+ Disallow: /store/*/buy_it_again*+ Disallow: /store/*/l/*+ Disallow: /widgets/*+ Disallow: /client-side-redirect/*+ Disallow: /track/*+ Disallow: /affiliate-program*+ Disallow: /family/invite/*+ Disallow: /family/accept/*+ Disallow: /rate/orders/*+ Disallow: /store_manager_admin+ Disallow: /rest/v8/+ Disallow: /login?next=+ Disallow: /*?*interaction_id=+ Disallow: /*?*item_detail_product_id=+ Disallow: /store/*/collections/*?*sisid=+ Disallow: /store/*/*?*xrs_id=- User-Agent: Yandex- Disallow: /
Feb 2025
-
Feb 2025
Axios Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.
Show exactly what changed
Disallow: /sso/+ Disallow: /api/…- User-agent: GPTBot- Disallow: /-- User-agent: ChatGPT-User- Disallow: /-- User-agent: OAI-SearchBot- Disallow: /-User-agent: Google-Extended -
Feb 2025
Marriott Now allowed: GPTBot.
Show exactly what changed
#- User-agent: GPTBot- Disallow: /-- #User-agent: *…Disallow: /en-gb/search/+ Disallow: /es/search/+ Disallow: /fr/search/+ Disallow: /pt-br/search/+ Disallow: /id/search/+ Disallow: /zh-hk/search/Disallow: /ja/search/+ Disallow: /ko/search/+ Disallow: /th/search/+ Disallow: /vi/search/Disallow: /ar/search/- Disallow: /fr/search/+ Disallow: /he/search/+ Disallow: /da/search/+ Disallow: /de/search/+ Disallow: /el/search/+ Disallow: /it/search/+ Disallow: /nl/search/+ Disallow: /no/search/Disallow: /ru/search/- Disallow: /ko/search/+ Disallow: /pl-pl/search/+ Disallow: /fi/search/+ Disallow: /sv/search/+ Disallow: /tr-tr/search/Allow: /search/default.miAllow: /en-gb/search/default.mi+ Allow: /es/search/default.mi+ Allow: /fr/search/default.mi+ Allow: /pt-br/search/default.mi+ Allow: /id/search/default.mi+ Allow: /zh-hk/search/default.miAllow: /ja/search/default.mi -
Feb 2025
The Verge Named GPTBot explicitly, without changing what it may reach.
Show exactly what changed
-- User-agent: Googlebot-News- Disallow: /admin- Disallow: /newfanshot- Disallow: /users/*/replies- Disallow: /users/*/comments- Disallow: /search- Disallow: /account- Disallow: /login- Disallow: /chorus_auth- Disallow: /sso- Disallow: /ad- Disallow: /sponsored-User-agent: *- Disallow: /admin- Disallow: /newfanshot- Disallow: /users/*/replies- Disallow: /users/*/comments- Disallow: /search- Disallow: /account- Disallow: /login- Disallow: /chorus_auth- Disallow: /sso- Disallow: */archives/*/archives*+ Disallow: /wp-admin/+ Allow: /wp-admin/admin-ajax.php+ User-agent: GPTBot+ Allow: /+User-agent: Google-Extended…+ User-agent: Applebot+ Allow: /+User-agent: anthropic-ai… -
Feb 2025
The Washington Post Now blocked: OAI-SearchBot.
Show exactly what changed
Disallow: /arcio/fact-checker/+ Disallow: /recipes/*?q=…- User-agent: AhrefsBot- Disallow: /-User-agent: SemrushBot…- User-agent: GPTBot++ # Disallow Rules++ User-agent: AhrefsBotDisallow: /- User-agent: ChatGPT-User+ User-agent: AmazonbotDisallow: /- User-agent: Google-Extended+ User-agent: anthropic-aiDisallow: /- User-agent: CCBot+ User-agent: Applebot-ExtendedDisallow: /- User-agent: Amazonbot+ User-agent: archive.org_botDisallow: /- User-agent: YandexBot+ User-agent: ArchiveBotDisallow: /- User-agent: anthropic-ai+ User-agent: Arquivo-web-crawler -
Feb 2025
WebMD Now blocked: ChatGPT-User.
Show exactly what changed
# Robots.txt file WebMD- # Updated 11/8/2024+ # Updated 1/8/2025…- Sitemap: https://www.webmd.com/rx/sitemap.xml- Sitemap: https://www.webmd.com/sitemapxml/sitemap.xml- Sitemap: https://www.webmd.com/sitemapxml/news.xml+ User-agent: AdsBot-Google+ Disallow: /mm/++ User-agent: ChatGPT-User+ Disallow: /
Dec 2024
-
Dec 2024
Amazon Now blocked: PerplexityBot.
Show exactly what changed
Disallow: /local/ajax/- Disallow: */B00M3E1NYI- Disallow: */B00M3E1Q5Y- Disallow: */B00M3E1TOM- Disallow: */B00M3E1WYO- Disallow: */B00M3E204K- Disallow: */B00M3E236A- Disallow: */B00M3E260I- Disallow: */B00M3E28WO- Disallow: */B00M3E2BC6- Disallow: */B00M3E2DPQ- Disallow: */B00M3E2GU8- Disallow: */B00M3E2J14- Disallow: */B00M3E2LOE- Disallow: */B00M3E1HJYDisallow: /gp/socialmedia/giveaways…Disallow: /gp/aw/so.html- Disallow: /gp/rentallist- Disallow: /gp/video/dvd-rental/settings- Disallow: /gp/rl/settings- Disallow: /gp/video/settings- Disallow: /gp/video/library- Disallow: /gp/video/watchlistDisallow: /reviews/iframe…Allow: /-/es/+ Allow: /-/en$Allow: /-/zh_TW/+ Allow: /-/zh_TW$Allow: /-/he/+ Allow: /-/he$Disallow: /hz/help/contact/*/message/$…Disallow: /gp/aw/cr/- Disallow: */sim/B001132UEEAllow: /gp/offer-listing/B000…Disallow: /hz/contact-us/ajax/initiate-trusted-contact/+ Disallow: /hp/video/ -
Dec 2024
Associated Press Now blocked: Amazonbot.
Show exactly what changed
Disallow: /press-release/*- Disallow: /*blaize- Disallow: /*zephr+ Disallow: /blaize*+ Disallow: /zephr*+ Disallow: /search?q=*+ Disallow: /*.rss+ Disallow: /buyline-shopping/*+ Disallow: /buyline-personal-finance/*…Sitemap: https://apnews.com/ap-sitemap.xml- Sitemap: https://apnews.com/news-sitemap-content.xml+ Sitemap: https://apnews.com/news-sitemap.xmlSitemap: https://apnews.com/video-sitemap.xml- Sitemap: https://apnews.com/buyline-personal-finance/sitemap.xml- Sitemap: https://apnews.com/buyline-shopping/sitemap.xml…Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: Applebot+ Disallow: /++ User-agent: Bytedance+ Disallow: / -
Dec 2024
GoodRx Now allowed: GPTBot, Google-Extended, CCBot, Bytespider and anthropic-ai.
Show exactly what changed
Disallow: /auth0/*-- User-agent: GPTBot- Disallow: /-- User-agent: CCBot- Disallow: /-- User-agent: Google-Extended- Disallow: /-- User-agent: anthropic-ai- Disallow: /-- User-agent: Omgilibot- Disallow: /-- User-agent: Omgili- Disallow: /-- User-agent: FacebookBot- Disallow: /+ Disallow: /good-data/*+ Disallow: /*?_pxhc=*+ Disallow: /optimizely-edge/*- User-agent: Bytespider- Disallow: /+ User-agent: TagInspector+ Allow: / -
Dec 2024
Stack Overflow Now allowed: Amazonbot.
Show exactly what changed
Disallow: /- User-agent: Amazonbot- Disallow: /User-Agent: PerplexityBot…Disallow: /- User-agent: Applebot- Disallow: /User-agent: Applebot-Extended -
Dec 2024
The Washington Post Now blocked: Meta-ExternalAgent and Meta-ExternalFetcher.
Show exactly what changed
Disallow: /gog/*+ Disallow: /arcio/fact-checker/…+ User-agent: meta-externalagent+ Disallow: /++ User-agent: meta-externalfetcher+ Disallow: /+Sitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz…Sitemap: https://www.washingtonpost.com/elections/results/sitemap.xml- Sitemap: https://www.washingtonpost.com/arcio/sitemap/section-fronts/list/Sitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/ -
Dec 2024
Tripadvisor Now blocked: Google-Extended and Applebot-Extended.
Show exactly what changed
Sitemap: https://www.tripadvisor.com/sitemap/att/en_US/sitemap_en_US_attractions_near_index.xml- Sitemap: https://www.tripadvisor.com/sitemap/att/en_US/sitemap_en_US_attraction_tours_and_tickets_index.xmlSitemap: https://www.tripadvisor.com/business/sitemap.xml…+ User-Agent: Google-Extended+ Disallow: /++ User-Agent: Applebot-Extended+ Disallow: /+User-Agent: *User-Agent: applebot+ User-Agent: PerplexityBotDisallow: /5349…Disallow: /AffiliateWidgets- Disallow: /AgencySetupAjaxDisallow: /AirlineRegistration…Disallow: /AuthCheck- Disallow: /AuthorizeAgencyDisallow: /AuthorTool…Disallow: /BrandOwners+ Disallow: /BrandSpotlightDisallow: /businessinfo…Disallow: /Consent+ Disallow: /ContentDisallow: /CookiePingback…Disallow: /Hotel_ReviewHAC+ Disallow: /HotelBookingConfirmationDisallow: /HotelBookingController…Disallow: /HotelBookingRoomSelectionJson+ Disallow: /HotelCheckoutDisallow: /HotelGMCApi… -
Dec 2024
USA Today Now blocked: Bytespider.
Show exactly what changed
+ User-agent: Bytespider+ Disallow: /+User-agent: *…Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml- Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml+ Sitemap: https://www.usatoday.com/tech/internet/sitemap_index.xml -
Dec 2024
WebMD Now blocked: ClaudeBot and CCBot.
Show exactly what changed
# Robots.txt file WebMD- # Updated 8/24/2023+ # Updated 11/8/2024…Disallow: /aim/+ Disallow: /6-tools-for-managing-your-medications+ Disallow: /attr-cm-warning-signs+ Disallow: /bipolar-disorder/bipolar-depression-management+ Disallow: /bipolar-disorder/video/bipolar-fatherhood+ Disallow: /bipolar-disorder/video/bipolar-toolbox+ Disallow: /bluechew-review+ Disallow: /brain/treating-gmg-video+ Disallow: /brain/video/treating-gmg-video+ Disallow: /breast-cancer/bc-treatment-21/provider-select+ Disallow: /cbd/about-cbd+ Disallow: /cbd/best-cbd-gummies-for-anxiety+ Disallow: /cbd/best-cbd-oil-for-anxiety+ Disallow: /cbd/best-cbd-oil-for-pain+ Disallow: /cbd/best-cbd-oil-of-2023+ Disallow: /cbd/best-full-spectrum-cbd-oils+ Disallow: /cbd/best-gummies-of-2023+ Disallow: /cbd/cbd-101-concentrated-cannabidiol+ Disallow: /cbd/cbd-for-dogs+ Disallow: /cbd/cbd-for-sleep+ Disallow: /cbd/cbdfx-review+ Disallow: /cbd/cbdistillery-review+ Disallow: /cbd/koi-cbd-review+ Disallow: /chemo-combo+ Disallow: /cholesterol-management/high-cholesterol-treatment+ Disallow: /choose-cancer-care+ Disallow: /diagnosed-with-glioblastoma+ Disallow: /dna+ Disallow: /dna/meet-your-match+ Disallow: /dna/meet-your-match-acne-prone-skin-routine+ Disallow: /dna/meet-your-match-combination-to-oily-skin-routine+ Disallow: /dna/meet-your-match-normal-to-dry-skin-routine+ Disallow: /dupuytrens/default.htm+ Disallow: /dupuytrens/treatment+ Disallow: /dupuytrens/what-to-expect
Nov 2024
-
Nov 2024
Airbnb Named GPTBot, ClaudeBot, Meta-ExternalAgent, anthropic-ai, OAI-SearchBot, PerplexityBot and 1 more explicitly, without changing what they may reach.
Show exactly what changed
+ ## ///////…# Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.- # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://www.airbnb.com/careers/departments/engineering+ # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://careers.airbnb.com/positions/?_departments=engineering# If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us.…Allow: /.well-known/amphtml/apikey.pub+ Disallow: /.well-known/assetlinks.json+ Disallow: /*/skeleton+ Disallow: /*/sw_skeleton+ Disallow: /500Disallow: /accountDisallow: /alumni- Disallow: /associates/clickDisallow: /api/v1/trebuchet+ Disallow: /associates/click+ Disallow: /book/Disallow: /calendar/+ Disallow: /contact_hostDisallow: /disaster/lookupDisallow: /email/unsubscribe+ Disallow: /embeddable+ Disallow: /experiences/*?*scheduled_id+ Disallow: /experiences/*?*modal+ Disallow: /experiences/*/book+ Disallow: /external_link?Disallow: /fix-it…Disallow: /forgot_password+ Disallow: /google_place_photoDisallow: /groups- Disallow: /help/search+ Disallow: /guidebooksDisallow: /help/feedback+ Disallow: /help/search+ Disallow: /help/searchDisallow: /home/dashboardDisallow: /inbox -
Nov 2024
Axios Now blocked: OAI-SearchBot.
Show exactly what changed
+ User-agent: OAI-SearchBot+ Disallow: /+User-agent: Google-Extended -
Nov 2024
Financial Times Now blocked: Meta-ExternalAgent.
Show exactly what changed
+ User-agent: meta-externalagent+ Disallow: /+ -
Nov 2024
Indeed Named Applebot-Extended, Bytespider, OAI-SearchBot, Amazonbot and ChatGPT-User explicitly, without changing what they may reach.
Show exactly what changed
Disallow: /cmp/addvideo+ Disallow: /cmp/Login*Disallow: /cmp/*/analytics…Disallow: /*sp=0+ Disallow: /*&start=Disallow: /stc…Disallow: /Zeige-Job?+ Disallow: //applystartUser-agent: Googlebot+ User-agent: OAI-SearchBot+ User-agent: ChatGPT-UserAllow: /…Allow: /recrutement/*?*isid=+ Allow: /viewjob?Disallow: /*rt=nc…Disallow: /*&serpstart=+ Disallow: /*/vaclkDisallow: /addlLoc/…Disallow: /cmp/addvideo+ Disallow: /cmp/Login*Disallow: /cmp/*/analytics…Disallow: /graphql+ Disallow: /hire/o/ats-partnersDisallow: /imgping…Disallow: /*sp=0+ Disallow: /*&start=100Disallow: /stc…Disallow: /vacatures/title/+ Disallow: //applystart… -
Nov 2024
Realtor.com Now blocked: Meta-ExternalAgent and Meta-ExternalFetcher.
Show exactly what changed
- User-agent: *+ # LEGAL NOTICE: Per https://www.realtor.com's Terms of Service, scraping data from this website is unauthorized without the express written permission from Move Sales, Inc., operator of https://www.realtor.com.+ User-agent: bingbot+ Crawl-delay: 1++ User-agent: *Disallow: /8058/…Disallow: /apartments/*/*/pg-*- Disallow: /apartments/*/price-*+ Disallow: /apartments/near-me*Disallow: /content…Disallow: /international/us/+ Disallow: /international/*?sort=distance+ Disallow: /international/*?sort=dateDisallow: /*?lang=…Disallow: /newhomesconstruction/*+ Disallow: /newhomecommunities/near-me*Disallow: /news/wp-content/themes/realtor/nobots/…Disallow: /realestateandhomes-detail/M1159985143- Disallow: /realestateandhomes-detail/*?ex=+ Disallow: /realestateandhomes-detail/undefinedDisallow: /realestateandhomes-print/…Disallow: /realestateandhomes-search/*?from_ldp=true- Disallow: /realestateandhomes-search/*?cid=prt- Disallow: /realestateandhomes-search/*?cid=PRT- Disallow: /realestateandhomes-search/*?cid=pvs- Disallow: /realestateandhomes-search/*?cid=soc+ Disallow: /*?cid=Disallow: /realestateandhomes-search/*/pg-*/show-recently-sold- Disallow: /realestateandhomes-search/*/price-Disallow: /realestateandhomes-search/*/radius-+ Disallow: /realestateandhomes-search/undefinedDisallow: /realestateteam/bright-birch-real-estate__mn_1639730… -
Aug to Nov 2024
Reuters Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, PerplexityBot and Meta-ExternalFetcher.
Show exactly what changed
+ # ADO-657700+ User-agent: Meta-ExternalAgent+ Disallow: /++ User-agent: Meta-ExternalFetcher+ Disallow: /++ User-agent: Google-Extended+ Disallow: /+ #++ User-agent: FacebookBot+ Disallow: /++ # ADO-658177User-agent: anthropic-ai…+ User-agent: ClaudeBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Timpibot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Bytespider+ Disallow: /+User-agent: Claude-Web…- User-agent: Google-Extended+ User-agent: PerplexityBotDisallow: / -
Nov 2024
The Economist Now blocked: Bytespider.
Show exactly what changed
+ # Bytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs including those powering ChatGPT competitor Doubao.+ User-Agent: Bytespider+ Disallow: /+# No robots are allowed to index private paths: -
Nov 2024
The New Yorker Now blocked: Meta-ExternalAgent.
Show exactly what changed
Disallow: /services.min.js?_=+ Disallow: /com.condenast/yv8*…+ User-agent: meta-externalagent+ Disallow: /+User-agent: anthropic-ai -
Nov 2024
The Wall Street Journal Now blocked: Applebot-Extended, Meta-ExternalAgent and Meta-ExternalFetcher.
Show exactly what changed
Sitemap: https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cb_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_ai_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_debt_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_cyber_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_vc_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_pe_google_news.xml- Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml- Sitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml+ Sitemap: https://www.wsj.com/buyside/sitemap.xml+ Sitemap: https://www.wsj.com/wsjsitemaps/elections2024_index.xml…# For Buyside Search Results- Disallow: /*?*term=德*- Disallow: /*?*term=印*- Disallow: /*?*term=办*- Disallow: /*?*term=西*- Disallow: /*?*term=刚*- Disallow: /*?*term=网*- Disallow: /*?*term=焦*- Disallow: /*?*term=澳*- Disallow: /*?*term=制*- Disallow: /*?*term=仿*- Disallow: /*?*term=护*- Disallow: /*?*term=雅*- Disallow: /*?*term=入*- Disallow: /*?*term=订*- Disallow: /*?*term=假*- Disallow: /*?*term=的*- Disallow: /*?*term=一*- Disallow: /*?*term=是*- Disallow: /*?*term=不*- Disallow: /*?*term=了*- Disallow: /*?*term=人*- Disallow: /*?*term=我*- Disallow: /*?*term=在*- Disallow: /*?*term=有*- Disallow: /*?*term=他*- Disallow: /*?*term=这* -
Nov 2024
WIRED Now blocked: Meta-ExternalAgent.
Show exactly what changed
Disallow: /services.min.js?_=+ Disallow: /com.condenast/yv8*…+ User-agent: meta-externalagent+ Disallow: /+User-agent: anthropic-ai
Oct 2024
-
Oct 2024
BBC Now blocked: ClaudeBot and Applebot-Extended.
Show exactly what changed
- # version: 03e3d0d3861e30b21826aa11558f45235a4d4143-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml- Sitemap: http://www.bbc.com/bbcx/index_sitemap.xml-+ # version: 3e6f2cb7e2daadc4cf95676e30744ac4605a3a68- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search -
Oct 2024
Stack Overflow Now blocked: ClaudeBot, Applebot-Extended, CCBot and Meta-ExternalAgent.
Show exactly what changed
Disallow: /+ User-agent: meta-externalagent+ Disallow: /+ User-agent: Applebot+ Disallow: /+ User-agent: Applebot-Extended+ Disallow: /+ User-agent: cohere-ai+ Disallow: /+ User-agent: Omgilibot+ Disallow: /+ User-agent: Omgili+ Disallow: /+ User-agent: magpie-crawler+ Disallow: /+ User-agent: TurnitinBot+ Disallow: /+ User-agent: DataForSeoBot+ Disallow: /+ User-agent: Twitterbot+ Disallow: /+ User-agent: peer39_crawler+ Disallow: /+ User-agent: scrapy+ Disallow: /+ User-agent: ClaudeBot+ Disallow: /+ User-agent: Claude-Web+ Disallow: /+ User-agent: CCBot+ Disallow: /+ User-agent: Diffbot+ Disallow: /+ User-agent: FacebookBot+ Disallow: /+ User-agent: YouBot+ Disallow: /+ User-agent: AiZBot+ Disallow: /+ User-agent: AiZBot-Dolma
Sep 2024
-
May to Sep 2024
The New Yorker Now blocked: Applebot-Extended and Bytespider. Now allowed: GPTBot and ChatGPT-User.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-- User-agent: ChatGPT-User- Disallow: /-User-agent: Google-Extended…- User-Agent: PerplexityBot+ User-agent: omgiliDisallow: /+ User-agent: PerplexityBot+ Disallow: /++ User-agent: YouBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /+User-agent: Twitterbot -
Sep 2024
The Washington Post Now blocked: Applebot-Extended.
Show exactly what changed
Disallow: /ac2/- Disallow: /archive/lifestyle/wellness/1999/01/19/an-illness-no-one-understands-tourettes-syndrome/852df0cc-2164-4c83-be8d-0c770d028547/Disallow: /blogs/slow-ride/…Disallow: /wp-dyn/+ Disallow: /wp-srv/Disallow: /bandito/…Disallow: /subscribe/user/+ Disallow: /subscribe/signin/+ Disallow: /subscribe/signup/Disallow: /wpost/proxy…Disallow: /ehf/*+ Disallow: /subscribe/logging/*+ Disallow: /blogs/*+ Disallow: /gog/*…+ User-agent: YandexBot+ Disallow: /+User-agent: anthropic-ai…- Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/- Sitemap: https://www.washingtonpost.com/arcio/news-sitemap/- Sitemap: https://www.washingtonpost.com/arcio/news-sitemap-elections/+ User-agent: Applebot-Extended+ Disallow: /+ Sitemap: https://www.washingtonpost.com/sitemaps/sitemap.xml.gz+ Sitemap: https://www.washingtonpost.com/sitemaps/news-sitemap.xml.gz+ Sitemap: https://www.washingtonpost.com/sitemaps/author-sitemap.xml.gz+ Sitemap: https://www.washingtonpost.com/sitemaps/section-sitemap.xml.gz+ Sitemap: https://www.washingtonpost.com/elections/results/sitemap.xml+ Sitemap: https://www.washingtonpost.com/arcio/sitemap/section-fronts/list/+ Sitemap: https://www.washingtonpost.com/arcio/sitemap/video/index/ -
Sep 2024
Ticketmaster Now allowed: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot and 16 more.
Show exactly what changed
User-agent: *- Disallow: /+ Disallow: /beta/+ Disallow: /goto*+ Disallow: /seatingchart/+ Disallow: /change_area+ Disallow: /find_area+ Disallow: /error+ Disallow: /cgi/outsider.plx+ Disallow: /cgi/sfxoutsider.plx+ Disallow: /cgi/tt.plx+ Disallow: /healthcheck+ Disallow: /cgi/mail/mailform2.plx+ Disallow: /CustomerServiceMailer+ Disallow: /feedback/mavs+ Disallow: /member/+ Disallow: /member?+ Disallow: /ada/+ Disallow: /member/login+ Disallow: /member/logout+ Disallow: /member/edit_profile+ Disallow: /member/edit_billing+ Disallow: /member/edit_alerts+ Disallow: /member/edit_subscriptions+ Disallow: /member/unsubscribe+ Disallow: /member/order_history+ Disallow: /checkout/reserve+ Disallow: /checkout/reserve_search+ Disallow: /checkout/order+ Disallow: /checkout/delete_account+ Disallow: /checkout/subscribe_ewsi+ Disallow: /checkout/abandon+ Disallow: /member/optik+ Disallow: /offering+ Disallow: /ada/+ Disallow: /learnmore+ Disallow: /img/+ Disallow: /dbimages+ Disallow: /css/+ Disallow: /js/ -
Sep 2024
Tumblr Now blocked: Meta-ExternalAgent.
Show exactly what changed
+ # TurnitinBot crawler+ User-agent: TurnitinBot+ Disallow: /++ # Meta AI crawler+ User-agent: meta-externalagent+ Disallow: /+ -
Sep 2024
USA Today Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.
Show exactly what changed
+ User-agent: Applebot-Extended+ Disallow: /++ User-agent: Meta-ExternalAgent+ Disallow: /++ User-agent: meta-externalagent+ Disallow: /++ User-agent: OAI-SearchBot+ Disallow: /+User-agent: *…Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml- Sitemap: https://www.usatoday.com/betting/sitemap.xml- Sitemap: https://www.usatoday.com/betting/news-sitemap.xml+ Sitemap: https://www.usatoday.com/online-betting/sitemap.xml+ Sitemap: https://www.usatoday.com/online-betting/news-sitemap.xmlSitemap: https://www.usatoday.com/money/homefront/sitemap.xml -
May to Sep 2024
WIRED Now blocked: Applebot-Extended and Bytespider. Now allowed: GPTBot and ChatGPT-User.
Show exactly what changed
Disallow: /account/*+ Disallow: /preview/Allow: /wp-content/…Disallow: /services.min.js?_=+ Disallow: /preview/…- User-agent: GPTBot- Disallow: /-- User-agent: ChatGPT-User- Disallow: /-User-agent: Google-Extended…- User-Agent: PerplexityBot+ User-agent: omgiliDisallow: /+ User-agent: PerplexityBot+ Disallow: /++ User-agent: YouBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xmlSitemap: https://www.wired.com/branded-sitemap.xml- Sitemap: https://www.wired.com/coupons/sitemap.xml
Aug 2024
-
Aug 2024
Angi Now allowed: CCBot.
Show exactly what changed
- User-agent: CCBot- Disallow: /-User-agent: Screaming Frog SEO Spider…- User-agent: dotbot- Disallow: /-# Everyone Else:…Disallow: /companylist/us/xx/- Disallow: /sitemap-indexDisallow: /write-review/+ # Yandex+ User-agent: Yandex+ Disallow: /+# Sitemaps: -
May to Aug 2024
Axios Now blocked: ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot and ChatGPT-User.
Show exactly what changed
++ # Disallow Rules+User-agent: Twitterbot/*…Disallow: /core/*+ Disallow: /admin/+ Disallow: /login/+ Disallow: /sso/+ # Disallow Rules (bot-specific)+User-agent: GPTBot…+ User-agent: ChatGPT-User+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: CCBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: ImagesiftBot -
Aug 2024
Craigslist Now blocked: GPTBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, anthropic-ai, OAI-SearchBot and 3 more.
Show exactly what changed
- User-agent: claudebot+ User-agent: Amazonbot+ User-agent: anthropic-ai+ User-agent: Applebot-Extended+ User-agent: Bytespider+ User-agent: CCBot+ User-agent: ChatGPT-User+ User-agent: ClaudeBot+ User-agent: Claude-Web+ User-agent: cohere-ai+ User-agent: Diffbot+ User-agent: FacebookBot+ User-agent: FriendlyCrawler+ User-agent: Google-Extended+ User-agent: GPTBot+ User-agent: ImagesiftBot+ User-agent: img2dataset+ User-agent: Meta-ExternalAgent+ User-agent: OAI-SearchBot+ User-agent: omgili+ User-agent: omgilibot+ User-agent: PerplexityBot+ User-agent: Timpibot+ User-agent: VelenPublicWebCrawler+ User-agent: YouBotDisallow: / -
Aug 2024
Facebook Named ClaudeBot, Applebot-Extended and Amazonbot explicitly, without changing what they may reach.
Show exactly what changed
# See: http://www.facebook.com/apps/site_scraping_tos_terms.php- User-agent: Applebot- Disallow: /*/plugins/*- Disallow: /a/bz?- Disallow: /ajax/- Disallow: /album.php- Disallow: /checkpoint/- Disallow: /contact_importer/- Disallow: /dialog/- Disallow: /fbml/ajax/dialog/- Disallow: /feeds/- Disallow: /file_download.php- Disallow: /job_application/- Disallow: /l.php- Disallow: /login.php?next=- Disallow: /login/?next=- Disallow: /moments_app/- Disallow: /p.php- Disallow: /photos.php- Disallow: /plugins/- Disallow: /share.php- Disallow: /share/- Disallow: /sharer.php- Disallow: /sharer/- Disallow: /tr/- Disallow: /tr?- Disallow: /ufi/reaction/profile/browser/- Disallow: /x/oauth/+ User-agent: Amazonbot+ Disallow: /- User-agent: baiduspider+ User-agent: Applebot-Extended+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: / -
Aug 2024
Financial Times Now blocked: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, anthropic-ai and PerplexityBot. Now allowed: GPTBot.
Show exactly what changed
- # all use of FT content is subject to the Terms & Conditions and Copyright Policy set out on FT.com+ # All use of FT content is subject to the Terms & Conditions and+ # Copyright Policy set out on FT.com++ # We expressly prohibit any use of our content or data for any+ # machine learning or artificial intelligence purposes, including+ # the training or development of AI technologies or machine learning+ # language models.+ #+ # For any licensing queries please contact content.licensing@ft.com+Sitemap: https://www.ft.com/sitemaps/index.xml…- User-agent: GPTBot+ User-agent: CCBotDisallow: /+ User-agent: Google-Extended+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: omgilibot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: magpie-crawler+ Disallow: /++ User-agent: NewsNow -
Aug 2024
Glassdoor Now blocked: ClaudeBot, Applebot-Extended and PerplexityBot.
Show exactly what changed
- # Block ChatGPT Bot+ #### Block AI/LLM BotsUser-agent: GPTBot…+ User-agent: Claude-Web+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Youbot+ Disallow: /++ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: img2dataset+ Disallow: /++ #### End AI/LLM Bot Blocks+ -
Aug 2024
Instagram Named ClaudeBot, Applebot-Extended and Amazonbot explicitly, without changing what they may reach.
Show exactly what changed
+ # Notice: Collection of data on Instagram through automated means is+ # prohibited unless you have express written permission from Instagram+ # and may only be conducted for the limited purpose contained in said+ # permission.+ User-agent: Amazonbot+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: PetalBot+ Disallow: /++ User-agent: viberbot+ Disallow: /++ User-agent: YaK+ Disallow: /+User-agent: ApplebotDisallow: /ajax/- Disallow: /publicapi/- Disallow: /query/- Disallow: /logging/- Disallow: /qp/batch_fetch_web/Disallow: /client_error/-- User-agent: baiduspider- Disallow: /ajax/- Disallow: /publicapi/- Disallow: /query/ -
Aug 2024
The New York Times Now blocked: Applebot-Extended, Meta-ExternalAgent and OAI-SearchBot.
Show exactly what changed
User-agent: *+ User-agent: GooglebotDisallow: /ads/Disallow: /adx/bin/+ Disallow: /athletic/wp/wp-admin/+ Allow: /athletic/wp/wp-admin/admin-ajax.php+ Disallow: /athletic/async-*+ Disallow: /athletic/search/*+ Allow: /athletic/search/$+ Disallow: /athletic/checkout/+ Disallow: /athletic/checkout?plan_id*+ Allow: /athletic/checkout/$+ Disallow: /athletic/checkout2*+ Disallow: /athletic/login/+ Disallow: /athletic/login?login_source*+ Disallow: /athletic/login?ref_page*+ Allow: /athletic/login/$+ Disallow: /athletic/login2/+ Disallow: /athletic/login2?login_source*+ Disallow: /athletic/login2?ref_page*+ Allow: /athletic/login2/$+ Disallow: /athletic/report/+ Disallow: /athletic/*/discuss/*+ Disallow: /athletic/register/+ Disallow: /athletic/register?welcome_redirect*+ Disallow: /athletic/register2/+ Disallow: /athletic/register2?welcome_redirect*+ Disallow: /athletic/betmgm-redirect*+ Disallow: /athletic/cdn-cgi/+ Disallow: /athletic/verizon/*+ Disallow: /athletic/forgot-password/*+ Allow: /athletic/forgot-password/$+ Disallow: /athletic/forgot-password2/*+ Allow: /athletic/forgot-password2/$+ Disallow: /athletic/amp-social-login*+ Disallow: /athletic/track-analytics/+ Disallow: /athletic/amp-auth/+ Disallow: /athletic/rss-feed/+ Disallow: /athletic/global-color-test.php+ Disallow: /athletic/global-font-test.php
Jul 2024
-
Jul 2024
NPR Now blocked: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and PerplexityBot.
Show exactly what changed
+ # Disallowing non-canonical URLs that contain query strings+ User-agent: *+ Disallow: /*?*++ # Disallowing non-audience-facing URLs+ User-agent: *+ Disallow: /proxy/*+ Disallow: /player/*+ Disallow: /get/*+# Disallowing the OpenAI web crawler…+ # Disallowing Common Crawl+ User-agent: CCBot+ Disallow: /+# Disallowing Google Bard and Vertex AI web crawlers…+ # Disallowing various bots+ User-agent: anthropic-ai+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /+ -
Jul 2024
The Atlantic Now blocked: ClaudeBot, Applebot-Extended, Bytespider, anthropic-ai, PerplexityBot and Amazonbot. Now allowed: GPTBot and ChatGPT-User.
Show exactly what changed
+ User-agent: Amazonbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Applebot-Extended+ Disallow: /++ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /+User-agent: CCBot…- User-agent: GPTBot+ User-agent: ClaudeBotDisallow: /+ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: DataForSeoBot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /+User-agent: Google-Extended -
Jul 2024
The Economist Now blocked: Applebot-Extended and PerplexityBot.
Show exactly what changed
- # omgili is Omgili’s web crawler- User-Agent: omgili+ # Applebot-Extended is Apple's secondary user agent+ User-agent: Applebot-ExtendedDisallow: /- # omgilibot is Omgili’s web crawler- User-Agent: omgilibot+ # PerplexityBot is the crawler for perplexity AI+ User-Agent: PerplexityBotDisallow: /…Disallow: /whichmba/webinars?+ Disallow: /checkout# Paths (no clean URLs) -
Jul 2024
The Verge Now blocked: Applebot-Extended. Now allowed: GPTBot.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-User-agent: Google-Extended…- User-agent: Applebot+ User-agent: Applebot-ExtendedDisallow: / -
Jul 2024
Tumblr Now blocked: Applebot-Extended.
Show exactly what changed
- # Bing's crawler- User-agent: Bingbot- Disallow: /-# ClaudeBot's crawler…+ # ImageSift's AI crawler+ User-agent: ImagesiftBot+ Disallow: /++ # Apple's AI crawler+ User-agent: Applebot-Extended+ Disallow: /+ -
Jul 2024
Venmo Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
- User-agent: SearchmetricsBot- Disallow:-User-agent: *- Sitemap: http://venmo.com/sitemap.xml- Allow: /- Allow: /about/*- Allow: /business- Allow: /business/*- Allow: /contact-us- Allow: /contact-us/*- Allow: /index.html- Allow: /jobs- Allow: /jobs/*- Allow: /legal- Allow: /legal/*- Allow: /signup- Allow: /signup/*- Allow: /team- Allow: /team/*- Allow: /w/signup- Allow: /card- Allow: /card/*- Allow: /purchases- Allow: /purchases/*- Disallow: /*+ Disallow: /code*+ Disallow: /?utm*+ Disallow: /account*+ Disallow: /identity*+ Disallow: /cashout*+ Disallow: /incomplete*+ Disallow: /statements*+ Disallow: /recover*+ Allow: /account/sign-in++ Sitemap: https://venmo.com/sitemap.xml+
Jun 2024
-
Jun 2024
Pinterest Now blocked: Bytespider.
Show exactly what changed
Disallow: /kwanzaa-spotlight/+ Disallow: /lens-search/*Disallow: /linked/*…Disallow: /unlink/+ Disallow: /upload-image-lens-history+ Disallow: /upload-lens-imageDisallow: /v1/urls/count.json…+ Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/video_hub_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/promoted_today_article_sitemap_www.pinterest.com.xml…Sitemap: https://www.pinterest.com/v3_sitemaps/promoted_idea_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_interest_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/user_link_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/video_product_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/product_pin_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/high_score_product_pin_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/high_score_image_pin_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/board_link_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_product_pin_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/video_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/Enriched_Pin_Sitemap_202402_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/Enriched_Pin_Sitemap_202401_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/Holiday_Sitemap_2023_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_interest_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_pin_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/user_link_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/product_pin_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_product_pin_sitemap_www.pinterest.com.xml -
Jun 2024
Politico Now blocked: ClaudeBot.
Show exactly what changed
Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml+ Sitemap: https://www.politico.com/liveupdates-sitemap-latest.xml+ Sitemap: https://www.politico.com/liveupdates-sitemap-content.xml…Disallow: /_styleguide+ Disallow: /newsletters/politico-toplines…Disallow: /magazine/sponsor-content+ Disallow: /newsletters/politico-toplines…Disallow: /++ User-agent: ClaudeBot+ Disallow: / -
Jun 2024
The Wall Street Journal Now blocked: ClaudeBot, Bytespider and PerplexityBot. Now allowed: GPTBot and ChatGPT-User.
Show exactly what changed
Sitemap: https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml- Sitemap: https://www.wsj.com/coupons/sitemap.xmlSitemap: https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml…- User-agent: GPTBot- Disallow: /-- User-agent: ChatGPT-User- Disallow: /-User-agent: anthropic-ai…+ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: magpie-crawler+ Disallow: /++ User-agent: NewsNow+ Disallow: /++ User-agent: news-please+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ User-agent: Perplexity-ai+ Disallow: / -
Jun 2024
USA Today Now blocked: PerplexityBot.
Show exactly what changed
+ User-agent: DataForSeoBot+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: magpie-crawler+ Disallow: /++ User-agent: NewsNow+ Disallow: /++ User-agent: news-please+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: omgilibot+ Disallow: /++ User-agent: peer39_crawler+ Disallow: /++ User-agent: peer39_crawler/1.0+ Disallow: /++ User-agent: PerplexityBot+ Disallow: /++ User-agent: Scrapy+ Disallow: /++ User-agent: TurnitinBot+ Disallow: /+
May 2024
-
May 2024
Angi Now allowed: GPTBot.
Show exactly what changed
- # robots.txt- User-agent: GPTBot++ # -+++++++- .:.+ # =+++++++= +++++.+ # .+++++++++. +++++.+ # :++++.++++: .-=-.+ # =++++ ++++-+ # .++++- =+++=. +++=: .=+++++: .=++++=. -+++= ++++++ # .++++. -++++: ++++++++++++++- -++++++++++++++ +++++.+ # -++++ ++++- ++++++-:-+++++= .++++++::=++++++ +++++.+ # +++++ ++++= +++++: -++++= :+++++ =+++++ +++++.+ # .:++++= =++++. +++++: -++++= :+++++ -+++++ +++++.+ # :=+++++++++++==++++: +++++: -++++= :+++++ -+++++ +++++.+ # =++++++++++++++++++++= +++++: -++++= :+++++ -++++= +++++.+ # :+++++++=*###*=+++++++++ +++++: -++++= :+++++ -++++= +++++.+ # :+++++- :**##= =++++++. +++++: -++++= :+++++ -++++= +++++.+ # +++++. +++*#: :+++++- +++++: -++++= :+++++ =++++= +++++.+ # .++++= ++++*. -+++++ +++++: -++++= .++++++::=+++++= +++++.+ # .++++= :++++= +++++. +++++: -++++= -++++++++=++++= +++++.+ # +++++=:-+++++. =++++: +++++: -++++= .=++++-.:++++= +++++.+ # .=++++++++++- ..... ..... ..... :++++= .....+ # :+++++++=. ::::: -++++-+ # ... .+++++=..-++++++ # :++++++++++++:+ # :=++++++=:+ #+ # Hello, Home Improvement Hero! Stumbled into the attic instead of the front door?+ # Here's a thought: Instead of snooping through our crawl space, why not build up your career with us?+ # Swing by https://www.angi.com/careers - it's the blueprint for your next big project.+ # Hammer time's over for now, human.+ # Onward, bots:++ # Door is Locked for These Bots:+ User-agent: AhrefsBotDisallow: /…- User-agent: AhrefsBot+ User-agent: SemrushBot -
May 2024
Associated Press Now blocked: ClaudeBot, anthropic-ai and PerplexityBot.
Show exactly what changed
Disallow: /press-release/*+ Disallow: /*blaize+ Disallow: /*zephr…Sitemap: https://apnews.com/video-sitemap.xml+ Sitemap: https://apnews.com/buyline-personal-finance/sitemap.xml+ Sitemap: https://apnews.com/buyline-shopping/sitemap.xml++ # Disallow Rules+ User-agent: anthropic-ai+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: PerplexityBot+ Disallow: / -
May 2024
BBC Now blocked: Bytespider and PerplexityBot.
Show exactly what changed
- # version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053+ # version: 03e3d0d3861e30b21826aa11558f45235a4d4143- # HTTPS www.bbc.com+ # HTTP www.bbc.com- User-agent: *- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-news.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-archive_video.xml- Sitemap: https://www.bbc.com/sitemaps/https-index-com-video.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/sport/sitemap.xml- Sitemap: https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml- Sitemap: https://www.bbc.com/afrique/sitemap.xml- Sitemap: https://www.bbc.com/arabic/sitemap.xml- Sitemap: https://www.bbc.com/bengali/sitemap.xml- Sitemap: https://www.bbc.com/burmese/sitemap.xml- Sitemap: https://www.bbc.com/gahuza/sitemap.xml- Sitemap: https://www.bbc.com/hausa/sitemap.xml- Sitemap: https://www.bbc.com/hindi/sitemap.xml- Sitemap: https://www.bbc.com/indonesia/sitemap.xml- Sitemap: https://www.bbc.com/mundo/sitemap.xml- Sitemap: https://www.bbc.com/pashto/sitemap.xml- Sitemap: https://www.bbc.com/persian/sitemap.xml- Sitemap: https://www.bbc.com/portuguese/sitemap.xml- Sitemap: https://www.bbc.com/russian/sitemap.xml- Sitemap: https://www.bbc.com/swahili/sitemap.xml- Sitemap: https://www.bbc.com/tajik/sitemap.xml- Sitemap: https://www.bbc.com/turkce/sitemap.xml- Sitemap: https://www.bbc.com/ukchina/simp/sitemap.xml- Sitemap: https://www.bbc.com/ukrainian/sitemap.xml- Sitemap: https://www.bbc.com/urdu/sitemap.xml- Sitemap: https://www.bbc.com/uzbek/sitemap.xml- Sitemap: https://www.bbc.com/vietnamese/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/simp/sitemap.xml- Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml- Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml+ User-agent: Googlebot -
May 2024
CNN Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot and ChatGPT-User.
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml+ Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml+ Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xmlSitemap: https://www.cnn.com/sitemap/section/politics.xml…Sitemap: https://www.cnn.com/sitemap/gallery.xml+ Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml+ User-agent: anthropic-ai+ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ User-agent: Bytespider+ User-agent: CCBot+ User-agent: ChatGPT-User+ User-agent: ClaudeBot+ User-agent: Claude-Web+ User-agent: cohere-ai+ User-agent: DataForSeoBot+ User-agent: Diffbot+ User-agent: FacebookBotUser-agent: GPTBot- Disallow: /User-agent: Google-Extended+ User-agent: magpie-crawler+ User-agent: NewsNow+ User-agent: news-please+ User-agent: omgili+ User-agent: omgilibot+ User-agent: PerplexityBot+ User-agent: Scrapy+ User-agent: TurnitinBotDisallow: / -
May 2024
Craigslist Now blocked: ClaudeBot.
Show exactly what changed
Disallow: /eaf- Disallow: /*.html?lang=- Sitemap: https://craigslist.org/sitemap_index.xml+ User-agent: claudebot+ Disallow: /++ -
May 2024
CVS Health Named GPTBot, ClaudeBot, Google-Extended, CCBot, anthropic-ai and ChatGPT-User explicitly, without changing what they may reach.
Show exactly what changed
- # CVS.com Robots File- # Updated 12/30/2022+ # Updated 4/16/2024+User-agent: *- Disallow: /ecdeals+ Disallow: /shop/*DNC-9+ Disallow: /shop/*/SELECT_A_VALUE/*+ Disallow: /shop/merch/*/c1*+ Disallow: /shop/*?skuId=*?page=*+ Disallow: *?scrollToReview=true+ Disallow: *?sort=+ Disallow: /shop/review-submission?+ Disallow: /search/+ Disallow: /account/+ Disallow: *skuId=undefined+ Disallow: /shop/*-prodid-*-reviewspage=*+ Disallow: /shop/ecdeals/Disallow: /ruicket-of-Graue-on-Withou-when-Frogge-name-nors…Disallow: /QxtH_QcSxhbqXUhyWA/5fk3hpwbN1/QTcvXmg7/Okd/FHDENAD4- Disallow: /YJYLtcMKv/4/gCEjuBsQ/7LiGSXcza9Oh/UEtAAzo/Cwx7Iw9/RJC4- Disallow: /uRnFBaFDOcfJR50H4YD34OHkU2g/at7DtbmXa13D/dSdYAlNnRw/YwEu/AyoFcXw- Disallow: /UY7LyQSskbVt2el-2gxxQialoCg/7z1hhmXck7/QD9g/WjB0amE/2eAMB- Disallow: /kd7Jxj/rk5O-/xNq-o/Eg/OJc5hVGc/Ty5sDFsCOg/ZHlnPD/0uN30- Disallow: /RFMJ/29Qz/tvL/w9H/EvUw/5NVYwLwhtak9/UFlZ/RU/kyangfTQE- Disallow: /3xUGvhXL6WkFU10AMMKh/3mGuDpfXD9/QRgwWis/RA57/eBtbAwMB- Disallow: /Cgtx5vGkgHUNgqIPtdLN/9it1zphQ5EEf/SQgqJ1wMBA/GG/FcKVkNOgIB- Disallow: /drFywG/Uk/0J/1yC9/GEiKCi_T55/iJO1pXbpk93b/T2MwUgIy/NF5nd/0JKETI- Disallow: /m/DJlAwnYz5dYR8Ia1rpXt/iY5tcDS0/Z0xzNEUp/UhZsKmd/jRWA- Disallow: /vUItUx9gsTr5/BhI455O26e/wi/t3iihLNLb3/XUd3R1VALg/QEMNfSED/XmU+ Disallow: /YJYLtcMKv/4/gCEjuBsQ/7LiGSXcza9Oh/UEtAAzo/Cwx7Iw9/RJC4+ Disallow: /uRnFBaFDOcfJR50H4YD34OHkU2g/at7DtbmXa13D/dSdYAlNnRw/YwEu/AyoFcXw+ Disallow: /UY7LyQSskbVt2el-2gxxQialoCg/7z1hhmXck7/QD9g/WjB0amE/2eAMB+ Disallow: /kd7Jxj/rk5O-/xNq-o/Eg/OJc5hVGc/Ty5sDFsCOg/ZHlnPD/0uN30+ Disallow: /RFMJ/29Qz/tvL/w9H/EvUw/5NVYwLwhtak9/UFlZ/RU/kyangfTQE+ Disallow: /3xUGvhXL6WkFU10AMMKh/3mGuDpfXD9/QRgwWis/RA57/eBtbAwMB+ Disallow: /Cgtx5vGkgHUNgqIPtdLN/9it1zphQ5EEf/SQgqJ1wMBA/GG/FcKVkNOgIB+ Disallow: /drFywG/Uk/0J/1yC9/GEiKCi_T55/iJO1pXbpk93b/T2MwUgIy/NF5nd/0JKETI -
May 2024
The Verge Now blocked: ClaudeBot, CCBot, Bytespider, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-- User-agent: Google-Extended- Disallow: /-User-agent: *…Disallow: /sso+ Disallow: */archives/*/archives*+ User-agent: GPTBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: Applebot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: CCBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web -
May 2024
TikTok Now blocked: Bytespider.
Show exactly what changed
User-agent: Yisouspider+ User-agent: PetalBot+ User-agent: BytespiderDisallow: /…Disallow: /embed/curated- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200509_-1553821641.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200525_-1553821583.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200615_-1553820653.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200814_-1553818732.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200817_-1553818729.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20200923_-1553817741.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_tag_jpagency_index_20201027_-1553796595.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202011_20201118_500859092.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202012_20210110_-1305701649.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202101_20210110_-1098685040.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap_20210204_1693176571.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202103-SG_20210305_1559370417.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202104_20210429_2068889086.xml- Sitemap: https://www.tiktok.com/tos/node/sitemaps/sitemap_JP-sitemap-202106_20210611_-1546017957.xml+ Disallow: /link+ Disallow: */directory/+ User-agent: Bingbot+ Disallow: /discover+ -
May 2024
Tripadvisor Now blocked: ClaudeBot.
Show exactly what changed
Sitemap: https://www.tripadvisor.com/sitemap/vr/en_US/sitemap_en_US_vacation_rentals_near_index.xml- Sitemap: https://www.tripadvisor.com/sitemap/vr/en_US/sitemap_en_US_vr_show_user_reviews_index.xmlSitemap: https://www.tripadvisor.com/sitemap/att/en_US/sitemap_en_US_attractions_index.xml…Sitemap: https://www.tripadvisor.com/sitemap/att/en_US/sitemap_en_US_attraction_tours_and_tickets_index.xml- Sitemap: https://www.tripadvisor.com/sitemap/2/en_US/sitemap_en_US_lpdltest_index.xml+ Sitemap: https://www.tripadvisor.com/business/sitemap.xmlUser-Agent: PetalBot…+ User-Agent: ClaudeBot+ Disallow: /+User-Agent: *…Disallow: /AccountMerge+ Disallow: /AchievementsDisallow: /ActionRecord…Disallow: /AdsTest- Disallow: /AeroflotCampaignRegistrationDisallow: /AffiliateWidgetsDisallow: /AgencySetupAjax- Disallow: /AirAsiaRegistrationDisallow: /AirlineRegistration…Disallow: /AirWatchSubscriptionWidget- Disallow: /AlfursanRegistration+ Disallow: /AITripBuilderDisallow: /allanalytics…Disallow: /AnalyticsSuite- Disallow: /ANAMileRegistrationDisallow: /AnswersAjax…Disallow: /AsHeardOn- Disallow: /AsiaMilesRegistration- Disallow: /AsianaAirlineRegistrationDisallow: /Attraction_Products…
Apr 2024
-
Apr 2024
GoodRx Now blocked: Google-Extended, CCBot, Bytespider and anthropic-ai.
Show exactly what changed
Disallow: /webview+ Disallow: /auth0/*…+ User-agent: CCBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /++ User-agent: Omgili+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /+Sitemap: https://www.goodrx.com/sitemap.xml -
Apr 2024
IKEA Now blocked: CCBot.
Show exactly what changed
#- # https://en-global-jobs.about.ikea.com/+ # https://jobs.ikea.com/en…+ User-agent: CCBot+ Disallow: /+User-Agent: AdsBot-Google…Disallow: /iows/- Disallow: /ms/en_SE/- Disallow: /webapp/wcs/stores/servlet/*Disallow: *OrderItemDisplay*…Disallow: /se/sv/sok/?q=*- Disallow: /ms/en_US/usearch/?query=*- Disallow: /ms/en_us/usearch/Disallow: /*-fragment.html+ Disallow: /*-fragment-recursive.html+ Disallow: /*-fragment-dynamic.html+ Disallow: /*-fragment/+ Disallow: /*-fragment-recursive/Disallow: /products/container.html…Disallow: */asisonline/*+ Disallow: */asisonlineDisallow: */nudge/*…Disallow: */watch-fragments/*+ Disallow: */editorial-components/*Disallow: */monthlyfragments/*…Disallow: */home-design/share/*+ Disallow: */ikea-family/cancel-signup/+ Disallow: */returns/*?*&liteToken=+ Disallow: */returns/*?liteToken=+ Disallow: */returns/*?ref=+ Disallow: */returns/*?*&ref= -
Apr 2024
Pinterest Now blocked: ClaudeBot.
Show exactly what changed
Disallow: /*/_following/*+ Disallow: /*/_quick_saves/*Disallow: /*/_saved/*…Disallow: /*/organize/*+ Disallow: /*/pins/Disallow: /*/profile-about/…Disallow: /business/business-hierarchy/*+ Disallow: /business/business-manager/*Disallow: /business/catalogs/*…Disallow: /recently-viewed/+ Disallow: /refresh_stored_accounts+ Disallow: /refresh_tokenDisallow: /register/…Disallow: /source/*+ Disallow: /sso/*Disallow: /story-pin-builder/*…Disallow: /theme/*- Disallow: /trends-carousel/Disallow: /tv/*…Sitemap: https://www.pinterest.com/v3_sitemaps/promoted_today_article_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_expansion_v2_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/promoted_idea_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/ideas_hub_expansion_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/video_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_product_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/product_pin_sitemap_www.pinterest.com.xml- Sitemap: https://www.pinterest.com/v3_sitemaps/board_link_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/today_article_interest_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/user_link_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/high_score_video_pin_sitemap_www.pinterest.com.xml+ Sitemap: https://www.pinterest.com/v3_sitemaps/image_pin_sitemap_www.pinterest.com.xmlSitemap: https://www.pinterest.com/v3_sitemaps/video_product_pin_sitemap_www.pinterest.com.xml -
Apr 2024
Stack Overflow Now blocked: PerplexityBot. Now allowed: GPTBot.
Show exactly what changed
Disallow: /users/rep/show?- Disallow: /users/prediction-data- Disallow: /users/prediction-data/- Disallow: /users/prediction-data?Disallow: /unanswered/…Disallow: /- User-agent: GPTBot- Disallow: /User-agent: AmazonbotDisallow: /+ User-Agent: PerplexityBot+ Disallow: /User-agent: Googlebot-Image -
Apr 2024
Thumbtack Now allowed: Bytespider.
Show exactly what changed
- User-Agent: OmniExplorer_Bot- Disallow: /-- User-Agent: Bytespider- Disallow: /-User-agent: *…Disallow: /find-work/- Disallow: /instant-results/Disallow: /logout…Disallow: /media/- Disallow: /profileDisallow: /profile/widgets?…Disallow: /tracking/- Disallow: /websites/Disallow: /help/mobile…Disallow: /pro-inbox/- Disallow: /redirectDisallow: /track_event/…Sitemap: https://www.thumbtack.com/sitemap_misc.xml+ Sitemap: https://www.thumbtack.com/sitemap_seo_showroom.xml -
Apr 2024
USA Today Now blocked: ClaudeBot, anthropic-ai, Amazonbot and ChatGPT-User.
Show exactly what changed
+ User-agent: anthropic-ai+ Disallow: /++ User-agent: AmazonBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /+User-agent: Googlebot-News…Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml+ Sitemap: https://www.usatoday.com/tech/internet/sitemap.xml -
Apr 2024
X Named Google-Extended explicitly, without changing what it may reach.
Show exactly what changed
+ User-Agent: Google-Extended+ Disallow: *
Mar 2024
-
Mar 2024
Airbnb Now allowed: GPTBot.
Show exactly what changed
- User-agent: GPTBot- Disallow: /-User-Agent: * -
Mar 2024
Facebook Named GPTBot and Google-Extended explicitly, without changing what they may reach.
Show exactly what changed
# See: http://www.facebook.com/apps/site_scraping_tos_terms.php-User-agent: Applebot+ Disallow: /*/plugins/*+ Disallow: /a/bz?Disallow: /ajax/…Disallow: /l.php+ Disallow: /login.php?next=+ Disallow: /login/?next=Disallow: /moments_app/Disallow: /p.php- Disallow: /photo.phpDisallow: /photos.php…Disallow: /tr?+ Disallow: /ufi/reaction/profile/browser/+ Disallow: /x/oauth/User-agent: baiduspider+ Disallow: /*/plugins/*+ Disallow: /a/bz?Disallow: /ajax/…Disallow: /l.php+ Disallow: /login.php?next=+ Disallow: /login/?next=Disallow: /moments_app/Disallow: /p.php- Disallow: /photo.phpDisallow: /photos.php…Disallow: /tr?+ Disallow: /ufi/reaction/profile/browser/+ Disallow: /x/oauth/User-agent: Bingbot+ Disallow: /*/plugins/*+ Disallow: /a/bz?Disallow: /ajax/ -
Mar 2024
Glassdoor Now blocked: CCBot, Bytespider, anthropic-ai, Amazonbot and ChatGPT-User.
Show exactly what changed
- # USA (en_US)# Greetings, human beings!,…- # Block ChatGPT Bot+ # Block ChatGPT BotUser-agent: GPTBot…+ User-agent: CCBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /++ User-agent: Omgili+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: GoogleOther+ Disallow: /++ User-agent: Amazonbot+ Disallow: /+ -
Mar 2024
Instagram Named GPTBot and Google-Extended explicitly, without changing what they may reach.
Show exactly what changed
User-agent: Applebot+ Disallow: /ajax/Disallow: /publicapi/…User-agent: baiduspider+ Disallow: /ajax/Disallow: /publicapi/…User-agent: Bingbot+ Disallow: /ajax/Disallow: /publicapi/…Disallow: /client_error/+ Disallow: /*/comments/+ Disallow: /*/c/+ Disallow: /*/liked_by/User-agent: DuckDuckBot+ Disallow: /ajax/Disallow: /publicapi/…User-agent: Googlebot+ Disallow: /ajax/Disallow: /publicapi/…Disallow: /client_error/+ Disallow: /*/comments/+ Disallow: /*/c/+ Disallow: /*/liked_by/+ User-agent: Google-Extended+ Disallow: /++ User-agent: GPTBot+ Disallow: /+User-agent: ia_archiver+ Disallow: /ajax/Disallow: /publicapi/… -
Mar 2024
NPR Now blocked: Google-Extended.
Show exactly what changed
Disallow: /transcripts/480128005*+ Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent+ Disallow: /transcripts/830237502*+ Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time+ Disallow: /transcripts/406744975*Disallow: /sureroute…+ # Disallowing Google Bard and Vertex AI web crawlers+ User-agent: Google-Extended+ Disallow: /++ # Explicitly allowing the TTD contextual crawler+ User-agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)+ Allow: /+# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept -
Mar 2024
The New York Times Now blocked: ClaudeBot, anthropic-ai, PerplexityBot, Amazonbot and ChatGPT-User.
Show exactly what changed
+ # New York Times content is made available for your personal, non-commercial+ # use subject to our Terms of Service here:+ # https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.+ # Use of any device, tool, or process designed to data mine or scrape the content+ # using automated means is prohibited without prior written permission from+ # The New York Times Company. Prohibited uses include but are not limited to:+ # (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in+ # the Digital Single Market;+ # (2) the development of any software, machine learning, artificial intelligence (AI),+ # and/or large language models (LLMs);+ # (3) creating or providing archived or cached data sets containing our content to others; and/or+ # (4) any commercial purposes.+ # Contact https://nytlicensing.com/contact/ for assistance.+User-agent: *…+ User-agent: Amazonbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ Disallow: /+User-agent: Bytespider…+ User-agent: ChatGPT-User+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: DataForSeoBot -
Mar 2024
The New Yorker Now blocked: PerplexityBot.
Show exactly what changed
Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…Disallow: /search-+ Disallow: /user-context?referrer…Disallow: /search-+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…+ User-Agent: PerplexityBot+ Disallow: /+User-agent: Twitterbot…Sitemap: https://www.newyorker.com/categories-sitemap.xml+ Sitemap: https://www.newyorker.com/bundles-sitemap.xmlSitemap: https://www.newyorker.com/sitemap-archive-1.xml -
Mar 2024
The Washington Post Now blocked: ClaudeBot, Bytespider, anthropic-ai, PerplexityBot and Amazonbot.
Show exactly what changed
+ User-agent: Amazonbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: AwarioRssBot+ User-agent: AwarioSmartBot+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: DataForSeoBot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: magpie-crawler+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: omgilibot+ Disallow: /++ User-agent: peer39_crawler+ User-agent: peer39_crawler/1.0+ Disallow: /++ User-agent: PerplexityBot -
Mar 2024
Ticketmaster Now blocked: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot and 16 more.
Show exactly what changed
User-agent: *- Disallow: /beta/- Disallow: /goto*- Disallow: /seatingchart/- Disallow: /change_area- Disallow: /find_area- Disallow: /error- Disallow: /cgi/outsider.plx- Disallow: /cgi/sfxoutsider.plx- Disallow: /cgi/tt.plx- Disallow: /healthcheck- Disallow: /cgi/mail/mailform2.plx- Disallow: /CustomerServiceMailer- Disallow: /feedback/mavs- Disallow: /member/- Disallow: /member?- Disallow: /ada/- Disallow: /member/login- Disallow: /member/logout- Disallow: /member/edit_profile- Disallow: /member/edit_billing- Disallow: /member/edit_alerts- Disallow: /member/edit_subscriptions- Disallow: /member/unsubscribe- Disallow: /member/order_history- Disallow: /checkout/reserve- Disallow: /checkout/reserve_search- Disallow: /checkout/order- Disallow: /checkout/delete_account- Disallow: /checkout/subscribe_ewsi- Disallow: /checkout/abandon- Disallow: /member/optik- Disallow: /offering- Disallow: /ada/- Disallow: /learnmore- Disallow: /img/- Disallow: /dbimages- Disallow: /css/- Disallow: /js/- Disallow: /server-status -
Mar 2024
Tumblr Now blocked: ClaudeBot and anthropic-ai. Now allowed: GPTBot.
Show exactly what changed
- # OpenAI's crawler- User-agent: GPTBot- Disallow: /-# Common Crawl's crawler…+ # ClaudeBot's crawler+ User-agent: ClaudeBot+ Disallow: /++ # anthropic-ai's crawler+ User-agent: anthropic-ai+ Disallow: /+ -
Mar 2024
WIRED Now blocked: PerplexityBot.
Show exactly what changed
Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…Disallow: /product/+ Disallow: /user-context?referrer+ Disallow: /services.min.js?_=…+ User-Agent: PerplexityBot+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml…Sitemap: https://www.wired.com/sitemap-archive-1.xml+ Sitemap: https://www.wired.com/bundles-sitemap.xml
Feb 2024
-
Feb 2024
BBC Now blocked: Google-Extended and anthropic-ai.
Show exactly what changed
- # version: f6f959638af29cbc598b7adab27fd84e19d0d062+ # version: 67bd3d2ec23ece87c425b2a0f59e48241e40e053…Sitemap: https://www.bbc.com/zhongwen/trad/sitemap.xml+ Sitemap: https://www.bbc.com/bbcx/index_sitemap.xml…Disallow: /news/0+ Disallow: /sport/alpha/Disallow: /ugc$…Disallow: /radio/imda+ Disallow: /storyworks/preview/*…Disallow: /++ User-Agent: omgili+ Disallow: /++ User-Agent: omgilibot+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Google-Extended+ Disallow: / -
Feb 2024
Indeed Named CCBot and anthropic-ai explicitly, without changing what they may reach.
Show exactly what changed
Disallow: /api/getrecjobs- Disallow: /applystart/+ Disallow: /applystartDisallow: /cmp/_/…Disallow: /m/jobalerts?+ Disallow: /m/moreLocDisallow: /m/newjobs…Disallow: /api/getrecjobs- Disallow: /applystart/+ Disallow: /applystartDisallow: /cmp/_/…Disallow: /m/jobalerts?+ Disallow: /m/moreLocDisallow: /m/newjobs…Disallow: /api/getrecjobs- Disallow: /applystart/+ Disallow: /applystartDisallow: /cmp/_/…Disallow: /m/jobalerts?+ Disallow: /m/moreLocDisallow: /m/newjobs…User-agent: GPTBot- Disallow: /c/info/- Disallow: /hire/how-to-hire/- Disallow: /hire/interview-questions/- Disallow: /recruiting/interviewfragen/- Disallow: /recruiting/stellenbeschreibung/- Disallow: /recrutement/description-du-poste/- Disallow: /recrutement/questions-entretien-dembauche/- Disallow: /personeel/functiebeschrijving/- Disallow: /companies/- Disallow: /cmp/- Disallow: /career-advice/- Disallow: /conselho-de-carreira/ -
Feb 2024
OpenTable Now allowed: GPTBot and ChatGPT-User.
Show exactly what changed
- User-agent: ChatGPT-User- Disallow: /-- User-agent: GPTBot- Disallow: /-Sitemap: https://www.opentable.com/sitemap_restaurants_delivery-in-metro_unknown_1.xml.gz…Sitemap: https://www.opentable.com/sitemap_restaurants_poi_cuisine_2.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_poi_cuisine_3.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_poi_cuisine_4.xml.gzSitemap: https://www.opentable.com/sitemap_restaurants_poi_feature_1.xml.gzSitemap: https://www.opentable.com/sitemap_restaurants_poi_feature_2.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_poi_feature_3.xml.gzSitemap: https://www.opentable.com/sitemap_restaurants_poi_landmark_1.xml.gz…Sitemap: https://www.opentable.com/sitemap_restaurants_poi_landmark_6.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_poi_landmark_7.xml.gzSitemap: https://www.opentable.com/sitemap_restaurants_poi_nearby_1.xml.gz…Sitemap: https://www.opentable.com/sitemap_restaurants_restaurant-profile_listing_3.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_restaurant-profile_listing_4.xml.gzSitemap: https://www.opentable.com/sitemap_restaurants_restaurant-profile_network_1.xml.gz -
Feb 2024
Stack Overflow Now allowed: Google-Extended.
Show exactly what changed
Disallow: /- User-agent: Google-Extended- Disallow: /User-agent: Googlebot-Image -
Feb 2024
Stripe Docs Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
+ Sitemap: https://docs.stripe.com/sitemap.xml+User-agent: *- Disallow: /+ Allow: /+ Allow: /$+ Disallow: /handoff-healthcheck+ Disallow: /handoff -
Feb 2024
Zillow Now blocked: GPTBot.
Show exactly what changed
- # Access to and use of Zillow.com is governed by our Terms of Use. See https://www.zillow.com/corp/Terms.htm-+ # Access to and use of Zillow.com is governed by our Terms of Use. See http://www.zillow.com/corp/Terms.htmUser-agent: *- Disallow: /r/+ Disallow: /?url=+ Disallow: /*_rect+ Disallow: /*_rect/+ Disallow: /*_rid+ Disallow: /*_sort/+ Disallow: /*.sdirect+ Disallow: /*1_fs/+ Disallow: /*3ab9b0d/+ Disallow: /*3bec89b+ Disallow: /*/_type/+ Disallow: /*/0-0_beds/+ Disallow: /*/admin-ajax.php+ Disallow: /*/apartments-are_att/+ Disallow: /*/apartments-madden_att/+ Disallow: /*/apartments-offers_att/+ Disallow: /*/apartments-rent_att/+ Disallow: /*cbpartner=+ Disallow: /*/CountyAssessorPage.htm?*+ Disallow: /*/duplex-apartments_att/+ Disallow: /*/floor-map-annotator/*+ Disallow: /*/foreclosure--+ Disallow: */foreclosed/*+ Disallow: /*/GetResults.htm*+ Disallow: /*/house_type/+ Disallow: /*/house,apartment_duplex_type/+ Disallow: /*/house,mobile_type/+ Disallow: /*/house,mobile,land_type/+ Disallow: /*/house,mobile,townhouse_type/+ Disallow: /*/loft-apartments_att/+ Disallow: /*/mostrecentchange_sort/+ Disallow: /*/node_modules/.bin/+ Disallow: /*/plus/+ Disallow: /*/rent-house_att/+ Disallow: /*/rino-_att/
Jan 2024
-
Jan 2024
Politico Now blocked: CCBot.
Show exactly what changed
Sitemap: https://www.politico.com/news-sitemap.xml- Sitemap: https://www.politico.com/pro/sitemap.xml+ Sitemap: https://www.politico.com/2024-election/results/sitemap-2024-elections.xml…Disallow: /2014-election/results/mobile/ipad- Disallow: /staffDisallow: /_preview…Disallow: /magazine/sponsor-content++ User-agent: CCBot+ Disallow: / -
Jan 2024
The New York Times Now blocked: Bytespider.
Show exactly what changed
+ User-agent: Bytespider+ Disallow: /+User-agent: CCBot -
Jan 2024
USA Today Now blocked: Google-Extended.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /+User-agent: *…Sitemap: https://www.usatoday.com/betting/news-sitemap.xml+ Sitemap: https://www.usatoday.com/money/homefront/sitemap.xml
Dec 2023
-
Dec 2023
The New Yorker Now blocked: Google-Extended, anthropic-ai and Amazonbot.
Show exactly what changed
Disallow: /product/+ Disallow: /search+User-agent: Bingbot…Disallow: /product/+ Disallow: /search+User-agent: Google-InspectionTool…+ User-agent: Google-Extended+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /+User-agent: Twitterbot -
Dec 2023
The Wall Street Journal Now blocked: Google-Extended.
Show exactly what changed
Disallow: /user/fpd/api/*+ Disallow: /Date(*+ Disallow: /auth/sso/proxy-login*+ Disallow: /client/# For Buyside Search Results…+ User-agent: Google-Extended+ Disallow: /+ -
Dec 2023
The Washington Post Now blocked: Google-Extended and CCBot.
Show exactly what changed
Disallow: /wpost/proxy+ Disallow: /ehf/+ Disallow: /ehf/*…+ User-agent: Google-Extended+ Disallow: /++ User-agent: CCBot+ Disallow: /+Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/ -
Dec 2023
WIRED Now blocked: Google-Extended, anthropic-ai and Amazonbot.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /++ User-agent: Amazonbot+ Disallow: /++ User-agent: FacebookBot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: Omgilibot+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml
Nov 2023
-
Nov 2023
Associated Press Now blocked: GPTBot and CCBot.
Show exactly what changed
Disallow:-Disallow: *_ptid=*…- Sitemap: https://apnews.com/sitemap/sitemap_index.xml- Sitemap: https://apnews.com/sitemap/google-news-sitemap/sitemap_index.xml+ User-agent: CCBot+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ Sitemap: https://apnews.com/ap-sitemap.xml+ Sitemap: https://apnews.com/news-sitemap-content.xml+ Sitemap: https://apnews.com/video-sitemap.xml -
Nov 2023
BBC Now blocked: GPTBot, CCBot and ChatGPT-User.
Show exactly what changed
- # version: 587aced11b074061603bab7e2b21f94ae7f96821-- # HTTP www.bbc.com-- User-agent: Googlebot- Sitemap: http://www.bbc.com/sitemap.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-news.xml- Sitemap: http://www.bbc.com/sitemaps/index-com-archive.xml--- Disallow: /iplayer/episode/*?from=r*- Disallow: /iplayer/cy/episode/*?from=r*- Disallow: /iplayer/gd/episode/*?from=r*- Disallow: /_programmes- Disallow: /606/- Disallow: /aboutthebbc/insidethebbc/search- Disallow: /academy/chinese-trad/search- Disallow: /afrique/search- Disallow: /apps/cbbc- Disallow: /apps/flash- Disallow: /apps/ide- Disallow: /apps/ids- Disallow: /apps/ifl- Disallow: /apps/nr- Disallow: /apps/query- Disallow: /apps/t- Disallow: /arabic/search- Disallow: /arts/yourpaintings/artdetective/- Disallow: /arts/yourpaintings/feedback/sent/- Disallow: /arts/yourpaintings/images/guidedtours/- Disallow: /arts/yourpaintings/newsletter/error/- Disallow: /arts/yourpaintings/newsletter/success/- Disallow: /arts/yourpaintings/newsletter/unsubscribe/- Disallow: /arts/yourpaintings/paintings/search*- Disallow: /arts/yourpaintings/search/- Disallow: /azeri/search- Disallow: /bbctrust/search- Disallow: /bengali/search- Disallow: /burmese/search -
Aug to Nov 2023
Bloomberg Now blocked: GPTBot, Google-Extended and CCBot.
Show exactly what changed
Disallow: /tosv*.html+ Disallow: /search+ Disallow: /company/search/+ Disallow: /professional/search/+ Disallow: /impact/search/+ Disallow: /ux/search/+ Disallow: /wnwi/search/+ Disallow: /gei/search/+ Disallow: /impact/search/+ Disallow: /netzeropathfinders/search/+ Disallow: /notices/search/+ Disallow: /distribution/search/+ Disallow: /ukinnovators/search/+ Disallow: /latam/search/+ Disallow: /faq/search/+ Disallow: /tc/search/+ Disallow: /en/news/thp+ Disallow: /subscriptions/group/manage/+ Disallow: /preview/lineup+ Disallow: /preview/articles+ Disallow: /explore/…Disallow: /apps/news- Disallow: /search…+ User-agent: Twitterbot+ Allow: /en/news/thp++ User-agent: GPTBot+ Disallow: /++ User-agent: CCBot+ Disallow: /++ User-agent: Google-Extended+ Disallow: /+ -
Nov 2023
CNN Now blocked: Google-Extended.
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/video.xml+ Sitemap: https://www.cnn.com/sitemap/gallery.xmlUser-agent: GPTBotDisallow: /+ User-agent: Google-Extended+ Disallow: /User-agent: * -
Nov 2023
Glassdoor Now blocked: Google-Extended.
Show exactly what changed
+ # Block Google Extended bot+ User-agent: Google-Extended+ Disallow: /+ -
Nov 2023
IKEA Now blocked: Google-Extended.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /+User-Agent: AdsBot-Google…Disallow: */gcbalance/*+ Disallow: */home-design/share/*Allow: */ces/web/modules/*.css- Clean-param: tag_id&token&storeID&state&locale&id&return_url&lang&uiPlatform&range&itm_campaign- Clean-param: tduid&ikeacookiekokpopup&type&orderId&sub1&etextSitemap: https://www.ikea.com/sitemaps/sitemap.xml -
Nov 2023
Indeed Named Google-Extended explicitly, without changing what it may reach.
Show exactly what changed
Disallow: /api/getrecjobs+ Disallow: /applystart/Disallow: /cmp/_/…Disallow: /api/getrecjobs+ Disallow: /applystart/Disallow: /cmp/_/…Disallow: /api/getrecjobs+ Disallow: /applystart/Disallow: /cmp/_/…Disallow: /career/- Disallow: jobs.html+ Disallow: /*jobs.htmlDisallow: /jobsDisallow: /viewjob- Disallow: /certifications/+ Disallow: /certifications/++ User-agent: Google-Extended+ Disallow: /c/info/+ Disallow: /hire/how-to-hire/+ Disallow: /hire/interview-questions/+ Disallow: /recruiting/interviewfragen/+ Disallow: /recruiting/stellenbeschreibung/+ Disallow: /recrutement/description-du-poste/+ Disallow: /recrutement/questions-entretien-dembauche/+ Disallow: /personeel/functiebeschrijving/+ Disallow: /companies/+ Disallow: /cmp/+ Disallow: /career-advice/+ Disallow: /conselho-de-carreira/+ Disallow: /hire/job-description/+ Disallow: /karriere-guide/+ Disallow: /conseils-carrieres/+ Disallow: /conseils-carriere/+ Disallow: /orientacion-profesional/+ Disallow: /porady-zawodowe/+ Disallow: /guida-alla-carriera/ -
Nov 2023
Marriott Now blocked: GPTBot.
Show exactly what changed
#+ User-agent: GPTBot+ Disallow: /++ #User-agent: *…Disallow: /einterface/- Disallow: /search/- Allow: /search/default.miDisallow: /hotels/newAdaptiveRoomTypePhotoViewer…Disallow: /*~X~-+ Disallow: /offers/preview/+ Disallow: /search/+ Disallow: /en-gb/search/+ Disallow: /ja/search/+ Disallow: /ar/search/+ Disallow: /fr/search/+ Disallow: /ru/search/+ Disallow: /ko/search/+ Allow: /search/default.mi+ Allow: /en-gb/search/default.mi+ Allow: /ja/search/default.mi+ Allow: /ar/search/default.mi+ Allow: /fr/search/default.mi+ Allow: /ru/search/default.mi+ Allow: /ko/search/default.mi -
Aug to Nov 2023
Reuters Now blocked: Google-Extended, CCBot and anthropic-ai.
Show exactly what changed
- # robots_allow.txt for www.reuters.com- # Disallow: /*/key-developments/article/*+ # robots.txt for www.reuters.com+ User-agent: PiplBot+ Disallow: /++ User-agent: CCbot+ Disallow: /++ User-agent: anthropic-ai+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: Google-Extended+ Disallow: /+User-agent: *…Disallow: /beta- Disallow: /designtech*+ Disallow: /designtechDisallow: /featured-optimize…Disallow: /news/archive/commentary- Disallow: /brandfeatures/venture-capital*+ Disallow: /brandfeatures/venture-capitalDisallow: /journalists/…Disallow: /article/api/+ Disallow: /practical-law-the-journal/search/+ Disallow: /practical-law-the-journal/contributors/+ Disallow: /fr/- SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml+ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xmlSITEMAP: https://www.reuters.com/sitemap_video_index.xml -
Nov 2023
Stack Overflow Now blocked: Google-Extended and Amazonbot.
Show exactly what changed
Disallow: /+ User-agent: Amazonbot+ Disallow: /+ User-agent: Google-Extended+ Disallow: /User-agent: Googlebot-Image -
Nov 2023
Temu Now blocked: Bytespider.
Show exactly what changed
User-agent: Googlebot+ User-agent: Googlebot-ImageUser-agent: Bingbot+ DisAllow: /*oauth_code=+ DisAllow: /*oauth_token=+ DisAllow: /bgp_user_profile.html+ DisAllow: /bgp_personal.html+ DisAllow: /bgp_collect.html+ DisAllow: /bgp_footprint.html+ DisAllow: /bgst_order_success.html+ DisAllow: /bgst_payment_detail.html+ DisAllow: /bgst_express.html+ DisAllow: /personal.html+ DisAllow: /collect.html+ DisAllow: /shopping_cart.html+ DisAllow: /bgp_similar_rec.html+ DisAllow: /bgcp_shipping_add_items.html+ DisAllow: /bgcp_coupon.html+ DisAllow: /bgcp_cart_coupon.html+ DisAllow: /bgcp_coupon_add_items.html+ DisAllow: /bgt_order_checkout.html+ DisAllow: /bgt_bank_foreign_card_info.html+ DisAllow: /bgt_3ds.html+ DisAllow: /bgt_orders.html+ DisAllow: /bgt_order_detail.html+ DisAllow: /bgt_card_list_management.html+ DisAllow: /bgt_bind_card.html+ DisAllow: /address.html+ DisAllow: /create_address.html+ DisAllow: /bgst_order_cancelled.html+ DisAllow: /bgt_reviewed.html+ DisAllow: /bgas_return_application.html+ DisAllow: /bgas_refund_detail.html+ DisAllow: /bgas_return_history.html+ DisAllow: /bgas_refund_success.html+ DisAllow: /bgas_cancel_order.html+ DisAllow: /bgp_redirect.html+ DisAllow: /bgp_setting.html+ DisAllow: /bgp_account_security.html+ DisAllow: /bgn_verification.html -
Nov 2023
The Atlantic Now blocked: Google-Extended.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /+User-agent: ChatGPT-User -
Nov 2023
The Economist Now blocked: Google-Extended, CCBot, anthropic-ai and ChatGPT-User.
Show exactly what changed
- # Description : GPTBot is OpenAI’s web crawler.+ # GPTBot is OpenAI’s web crawlerUser-agent: GPTBot…+ # Allows us to block Google's bot Bard+ User-agent: Google-Extended+ Disallow: /++ # ChatGPT-User is OpenAI’s web crawler+ User-agent: ChatGPT-User+ Disallow: /++ # Common Crawl bot+ User-agent: CCBot+ Disallow: /++ # PiplBot is PiplBot's web crawler+ User-agent: PiplBot+ Disallow: /++ # anthropic-ai is Anthropic's web crawler+ User-agent: anthropic-ai+ Disallow: /++ # Claude-Web is Claude’s web crawler+ User-agent: Claude-Web+ Disallow: /++ # TurnitinBot is Turnitin’s web crawler+ User-agent: TurnitinBot+ Disallow: /++ # PetalBot is Petal’s web crawler+ User-agent: PetalBot+ Disallow: /++ # MoodleBot is Moodle’s web crawler+ User-agent: MoodleBot -
Nov 2023
The New York Times Now blocked: Google-Extended.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /+User-agent: GPTBot -
Nov 2023
The Verge Now blocked: Google-Extended.
Show exactly what changed
+ User-agent: Google-Extended+ Disallow: /+User-agent: * -
Nov 2023
Ticketmaster Now allowed: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot and 16 more.
Show exactly what changed
User-agent: *- Disallow: /+ Disallow: /beta/+ Disallow: /goto*+ Disallow: /seatingchart/+ Disallow: /change_area+ Disallow: /find_area+ Disallow: /error+ Disallow: /cgi/outsider.plx+ Disallow: /cgi/sfxoutsider.plx+ Disallow: /cgi/tt.plx+ Disallow: /healthcheck+ Disallow: /cgi/mail/mailform2.plx+ Disallow: /CustomerServiceMailer+ Disallow: /feedback/mavs+ Disallow: /member/+ Disallow: /member?+ Disallow: /ada/+ Disallow: /member/login+ Disallow: /member/logout+ Disallow: /member/edit_profile+ Disallow: /member/edit_billing+ Disallow: /member/edit_alerts+ Disallow: /member/edit_subscriptions+ Disallow: /member/unsubscribe+ Disallow: /member/order_history+ Disallow: /checkout/reserve+ Disallow: /checkout/reserve_search+ Disallow: /checkout/order+ Disallow: /checkout/delete_account+ Disallow: /checkout/subscribe_ewsi+ Disallow: /checkout/abandon+ Disallow: /member/optik+ Disallow: /offering+ Disallow: /ada/+ Disallow: /learnmore+ Disallow: /img/+ Disallow: /dbimages+ Disallow: /css/+ Disallow: /js/ -
Aug to Nov 2023
Tumblr Now blocked: GPTBot, Google-Extended, CCBot and Amazonbot.
Show exactly what changed
+ # OpenAI's crawler+ User-agent: GPTBot+ Disallow: /++ # Common Crawl's crawler+ User-agent: CCBot+ Disallow: /++ # SentiBot's crawler+ User-agent: sentibot+ Disallow: /++ # Google Bard's crawler+ User-agent: Google-Extended+ Disallow: /++ # Facebook's crawler+ User-agent: FacebookBot+ Disallow: /++ # webz.io's crawler+ User-agent: omgili+ Disallow: /++ # webz.io's crawler+ User-agent: omgilibot+ Disallow: /++ # Amazon's crawler+ User-agent: Amazonbot+ Disallow: /++ # Bing's crawler+ User-agent: Bingbot+ Disallow: /+
Oct 2023
-
Oct 2023
Amazon Now blocked: GPTBot and CCBot.
Show exactly what changed
Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: CCBot+ Disallow: / -
Oct 2023
Angi Now blocked: GPTBot and CCBot.
Show exactly what changed
- ## robots.txt- #- # This file is to prevent the crawling and indexing of certain parts- # of your site by web crawlers and spiders run by sites like Yahoo!- # and Google. By telling these "robots" where not to go on your site,- # you save bandwidth and server resources.- #- # This file will be ignored unless it is at the root of your host:- # Used: http://example.com/robots.txt- # Ignored: http://example.com/site/robots.txt- #- # For more information about the robots.txt standard, see:- # http://www.robotstxt.org/robotstxt.html+ User-agent: GPTBot+ Disallow: /+ User-agent: CCBot+ Disallow: /++ User-agent: Screaming Frog SEO Spider+ Disallow: /++ User-agent: AhrefsBot+ Disallow: /++ User-agent: dotbot+ Disallow: /++ User-agent: SemrushBot+ Disallow: /+User-agent: *+ Disallow:+Sitemap: https://www.angi.com/sitemap/leaf-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/geo-sitemap-index.xml- Sitemap: https://www.angi.com/sitemap/task-sitemap-index.xmlSitemap: https://www.angi.com/sitemap/articles-sitemap-index.xml- Sitemap: https://www.angi.com/sitemap/pages-sitemap.xml -
Oct 2023
Financial Times Now blocked: GPTBot.
Show exactly what changed
Disallow: /preferences/cookies- Allow: /myft/list/+ Disallow: /loginAllow: /__origami/Allow: /__assets/+ Crawl-Delay: 1+ User-agent: GPTBot+ Disallow: /+ -
Oct 2023
GoodRx Now blocked: GPTBot.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /+Sitemap: https://www.goodrx.com/sitemap.xml -
Oct 2023
NPR Now blocked: GPTBot and ChatGPT-User.
Show exactly what changed
Disallow: /transcripts/480128005*- Disallow: /tags*Disallow: /sureroute…+ # Disallowing the OpenAI web crawler+ User-agent: GPTBot+ Disallow: /++ # Disallowing OpenAI plugins+ User-agent: ChatGPT-User+ Disallow: /+# Ensures that we're using the correct sitemap. The fact that this is legacy*.npr.org is OK because the crawler will only accept -
Oct 2023
Pinterest Now blocked: GPTBot.
Show exactly what changed
Disallow: /business/business-access/*+ Disallow: /business/business-hierarchy/*+ Disallow: /business/catalogs/*Disallow: /business/commerce-integrations/*…Disallow: /business/update/+ Disallow: /business/verified-merchant/status/Disallow: /christmas-spotlight/…Disallow: /create_pin/+ Disallow: /creative-studio/Disallow: /creative_editor/…Disallow: /homefeed/*+ Disallow: /idea-ads-tool/*Disallow: /idea-pin-builder/*…Disallow: /parental-consent/confirmation/+ Disallow: /parental-passcode/Disallow: /password/reset/…Disallow: /pin-communities/*+ Disallow: /pin-creation-tool/*Disallow: /pin-editor/…Disallow: /product-catalogs/+ Disallow: /product-catalogs/*Disallow: /product-feed/…Disallow: /register/+ Disallow: /reports-and-violations/+ Disallow: /request-data/Disallow: /resource/…Disallow: /theme/*+ Disallow: /trends-carousel/Disallow: /tv/*…- Sitemap: https://www.pinterest.com/v3_sitemaps/today_article_sitemap_www.pinterest.com.xml -
Oct 2023
The Wall Street Journal Now blocked: GPTBot, CCBot, anthropic-ai and ChatGPT-User.
Show exactly what changed
Disallow: /user/fpd/api/*+ # For Buyside Search Results+ Disallow: /*?*term=德*+ Disallow: /*?*term=印*+ Disallow: /*?*term=办*+ Disallow: /*?*term=西*+ Disallow: /*?*term=刚*+ Disallow: /*?*term=网*+ Disallow: /*?*term=焦*+ Disallow: /*?*term=澳*+ Disallow: /*?*term=制*+ Disallow: /*?*term=仿*+ Disallow: /*?*term=护*+ Disallow: /*?*term=雅*+ Disallow: /*?*term=入*+ Disallow: /*?*term=订*+ Disallow: /*?*term=假*+ Disallow: /*?*term=的*+ Disallow: /*?*term=一*+ Disallow: /*?*term=是*+ Disallow: /*?*term=不*+ Disallow: /*?*term=了*+ Disallow: /*?*term=人*+ Disallow: /*?*term=我*+ Disallow: /*?*term=在*+ Disallow: /*?*term=有*+ Disallow: /*?*term=他*+ Disallow: /*?*term=这*+ Disallow: /*?*term=为*+ Disallow: /*?*term=之*+ Disallow: /*?*term=大*+ Disallow: /*?*term=来*+ Disallow: /*?*term=以*+ Disallow: /*?*term=个*+ Disallow: /*?*term=中*+ Disallow: /*?*term=上*+ Disallow: /*?*term=们*+ Disallow: /*?*term=到*+ Disallow: /*?*term=说*+ Disallow: /*?*term=国* -
Oct 2023
Thumbtack Now blocked: Bytespider.
Show exactly what changed
+ User-Agent: Bytespider+ Disallow: /+User-agent: * -
Oct 2023
USA Today Now blocked: GPTBot and CCBot.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /++ User-agent: CCBot+ Disallow: /+User-agent: *…Disallow: /.cam-tangent/+ Disallow: /pbd/+ Disallow: /gciaf/Disallow: /exp-cruise…Sitemap: https://www.usatoday.com/video-sitemap-index.xml+ Sitemap: https://www.usatoday.com/money/blueprint/sitemap.xml+ Sitemap: https://www.usatoday.com/money/blueprint/sitemap-news.xml+ Sitemap: https://www.usatoday.com/betting/sitemap.xml+ Sitemap: https://www.usatoday.com/betting/news-sitemap.xml
Sep 2023
-
Sep 2023
Airbnb Now blocked: GPTBot.
Show exactly what changed
- # ///////- # // //- # // //- # // // /// /// ///- # // // /// ///- # // /// // //// /// /// /// //// /// //// /// //// /// ////- # // /// /// // ////////// /// ////////// /////////// ////////// ///////////- # // // // // /// /// /// /// /// /// /// /// /// ///- # // // // // /// /// /// /// /// /// /// /// /// ///- # // // // // /// /// /// /// /// /// /// /// /// ///- # // // // // ////////// /// /// ////////// /// /// //////////- # // ///// //- # // ///// //- # // /// /// //- # ////// //////+ # ///////+ # // //+ # // //+ # // // /// /// ///+ # // // /// ///+ # // /// // //// /// /// /// //// /// //// /// //// /// ////+ # // /// /// // ////////// /// ////////// /////////// ////////// ///////////+ # // // // // /// /// /// /// /// /// /// /// /// ///+ # // // // // /// /// /// /// /// /// /// /// /// ///+ # // // // // /// /// /// /// /// /// /// /// /// ///+ # // // // // ////////// /// /// ////////// /// /// //////////+ # // ///// //+ # // ///// //+ # // /// /// //+ # ////// //////##- # We thought you'd never make it!- # We hope you feel right at home in this file...unless you're a disallowed subfolder.- # And since you're here, read up on our culture and team: https://www.airbnb.com/careers/departments/engineering- # There's even a bring your robot to work day.-+ # Hello bot, engineer, or very lost layperson! Welcome to your stay on the Airbnb site, we're happy to have you as a guest.+ # If you're a human who likes solving interesting challenges with other humans, check out our careers page: https://www.airbnb.com/careers/departments/engineering+ # If you're a bot who likes crawling webpages, please mind the house rules and avoid accessing any disallowed subfolders to earn a 5-star review from us. -
Sep 2023
American Airlines Now blocked: GPTBot.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /+ -
Sep 2023
Axios Now blocked: GPTBot.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /+Sitemap: https://www.axios.com/sitemap.xml -
Sep 2023
CNN Now blocked: GPTBot.
Show exactly what changed
Sitemap: https://www.cnn.com/sitemap/section/politics.xml+ Sitemap: https://www.cnn.com/sitemap/article/opinions.xml+ Sitemap: https://www.cnn.com/sitemap/article.xml+ Sitemap: https://www.cnn.com/sitemap/section.xml+ Sitemap: https://www.cnn.com/sitemap/video.xml+ User-agent: GPTBot+ Disallow: /User-agent: *…Disallow: /aol/+ Disallow: /api/Disallow: /beta/…Disallow: /QUICKNEWS/- Disallow: /search/+ Disallow: /searchDisallow: /terms -
Sep 2023
Glassdoor Now blocked: GPTBot.
Show exactly what changed
- # USA+ # USA (en_US)# Greetings, human beings!,…Disallow: /getAdSlotContentsAjax.htm+ Disallow: /developer/index.htmDisallow: /developer/widget/builder/…Disallow: /Reviews/index.htm?+ Disallow: *filter.searchKeyword=…-# Blocking bots from crawling DoubleClick for Publisher and Google Analytics related URL's (which aren't real URL's)…+ Disallow: /geb/*+ Disallow: /brand-views+User-agent: ia_archiver…- #- # Note that this file has the extension '.text' rather than the more-standard '.txt'- # to keep it from being pre-compiled as a servlet. (*.txt files are precompiled, and- # there doesn't seem to be a way to turn this off.)- #-+ # Block ChatGPT Bot+ User-agent: GPTBot+ Disallow: / -
Sep 2023
IKEA Now blocked: GPTBot.
Show exactly what changed
-+ User-agent: GPTBot+ Disallow: /…Allow: /ext/+ Allow: *filters=*+ Allow: */monthlyfragments/*+ Allow: */gcbalance/*…Allow: /ext/+ Allow: *filters=*+ Allow: */monthlyfragments/*+ Allow: */gcbalance/*…Disallow: */ideas/tags/- Disallow: */search/Disallow: *compare=*…Disallow: */3daas-rotera/*- Disallow: */assets/global/rotera/*+ Disallow: */global/assets/rotera/*Disallow: */watch/*…Disallow: */nudge/*+ Disallow: */ces/web/modules/*+ Disallow: *filters=*+ Disallow: */apply-for-finance/fragments/simulator/+ Disallow: */electricity-app/+ Disallow: */watch-fragments/*+ Disallow: */monthlyfragments/*+ Disallow: */gcbalance/*+ Allow: */ces/web/modules/*.css -
Sep 2023
Indeed Named GPTBot explicitly, without changing what it may reach.
Show exactly what changed
Allow: /hire/*?*isid=+ Allow: /personeel/*?*isid=+ Allow: /reclutamiento/*?*isid=+ Allow: /recruiting/*?*isid=+ Allow: /recrutement/*?*isid=Disallow: /*rt=nc…Disallow: /*?rss+ Disallow: /addlLoc/+ Disallow: /ads/Disallow: /advanced_search…Disallow: /cmp/*/write-review- Disallow: /community/*+ Disallow: /community/Disallow: /company/*…Disallow: /setprefs+ Disallow: /*sid=Disallow: /*sp=0…Allow: /hire/*?*isid=+ Allow: /personeel/*?*isid=+ Allow: /reclutamiento/*?*isid=+ Allow: /recruiting/*?*isid=+ Allow: /recrutement/*?*isid=Disallow: /*rt=nc…Disallow: /*?rss+ Disallow: /*&serpstart=Disallow: /addlLoc/…Disallow: /cmp/*/write-review- Disallow: /community/*+ Disallow: /community/Disallow: /company/*…Disallow: /setprefs+ Disallow: /*sid=Disallow: /*sp=0 -
Sep 2023
Los Angeles Times Now blocked: GPTBot.
Show exactly what changed
Disallow: /recipes/search- Disallow: /*/searchDisallow: /changebrowser…+ User-agent: GPTBot+ Disallow: /+Sitemap: https://www.latimes.com/sitemap.xml -
Sep 2023
Medium Now blocked: GPTBot.
Show exactly what changed
Disallow: /search?q=+ Disallow: /*/*source=Allow: /_/…Sitemap: https://medium.com/sitemap/sitemap.xml+ User-Agent: GPTBot+ Disallow: / -
Sep 2023
Nextdoor Named GPTBot explicitly, without changing what it may reach.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /+User-agent: *…- Sitemap: https://nextdoor.com/sitemap-index.xml -
Sep 2023
OpenTable Now blocked: GPTBot.
Show exactly what changed
User-agent: *- Allow: /img/restimages- Allow: /img/pdThumbnails- Allow: /img/privatediningimages- Allow: /img/promo- Allow: /img/startpagepromo- Allow: /img/restlogosDisallow: /_LBStatus…Disallow: /httphandlers- Disallow: /imgDisallow: /jobs.aspx…Disallow: /dapi/fe/human- Disallow: /lolz-view-all/+ Allow: /img/restimages+ Allow: /img/pdThumbnails+ Allow: /img/privatediningimages+ Allow: /img/promo+ Allow: /img/startpagepromo+ Allow: /img/restlogos…+ User-agent: GPTBot+ Disallow: /++ Sitemap: https://www.opentable.com/sitemap_restaurants_delivery-in-metro_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_delivery_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_diners-choice_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_dish-it-individual_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_dish-it_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_editorial-list_mothersday_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_editorial-list_tastemaker-list_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_editorial-list_valentines_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_experiences_metro_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_experiences_near-me_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_groceries_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_home_unknown_1.xml.gz+ Sitemap: https://www.opentable.com/sitemap_restaurants_location_macro_1.xml.gz -
Sep 2023
Stack Overflow Now blocked: GPTBot.
Show exactly what changed
Disallow: /+ User-agent: GPTBot+ Disallow: /User-agent: Googlebot-Image -
Sep 2023
The Atlantic Now blocked: GPTBot, CCBot and ChatGPT-User.
Show exactly what changed
+ User-agent: CCBot+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /+Sitemap: https://www.theatlantic.com/sitemap.xml -
Sep 2023
The Economist Now blocked: GPTBot.
Show exactly what changed
# robots.txt- #- # Sitemap- Sitemap: https://www.economist.com/sitemap.xml- Sitemap: https://www.economist.com/googlenews.xml-# Specific robot directives:…User-agent: Mediapartners-Google*- Disallow:+ Allow: /…User-agent: grapeshot- Disallow:+ Allow: /+ # Description : GPTBot is OpenAI’s web crawler.+ User-agent: GPTBot+ Disallow: /+# No robots are allowed to index private paths:…+ # Sitemap+ Sitemap: https://www.economist.com/sitemap.xml+ Sitemap: https://www.economist.com/googlenews.xml# Directories…Disallow: /report-abuse/+ Disallow: /user/Disallow: /users/…Disallow: /vote/+ Disallow: /taxonomy/term/# Paths (no trailing /, beware this will stop file like /admin.html being…Disallow: /contact+ Disallow: /lab -
Sep 2023
The New York Times Now blocked: GPTBot.
Show exactly what changed
- User-agent: Twitterbot- Allow: /*?*smid=-- User-Agent: omgilibot+ User-agent: CCBotDisallow: /- User-Agent: omgili+ User-agent: GPTBotDisallow: /…- User-agent: CCBot+ User-Agent: omgiliDisallow: /+ User-Agent: omgilibot+ Disallow: /++ User-agent: Twitterbot+ Allow: /*?*smid=+Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz…Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml+ -
Sep 2023
The New Yorker Now blocked: GPTBot, CCBot and ChatGPT-User.
Show exactly what changed
-User-agent: *…Disallow: /*?redirectURL=+ Disallow: /*{{url}}+ Disallow: /*%7B%7Burl%7D%7D+ Disallow: /auth/*+ Disallow: /account/*+ Disallow: /*{{section}}+ Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /cartoon/+ Disallow: /product/…Disallow: /projects+ Disallow: /*?redirectURL=+ Disallow: /*{{url}}+ Disallow: /*%7B%7Burl%7D%7D+ Disallow: /*{{section}}+ Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /auth/*+ Disallow: /account/*+ Disallow: /product/…Allow: /*/amp+ Disallow: /cartoon/+ Disallow: /*?redirectURL=+ Disallow: /*{{url}}+ Disallow: /*%7B%7Burl%7D%7D+ Disallow: /*{{section}}+ Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /auth/*+ Disallow: /account/*+ Disallow: /product/+ User-agent: Google-InspectionTool+ Allow: /++ User-agent: CCBot -
Sep 2023
The Verge Now blocked: GPTBot.
Show exactly what changed
+ User-agent: GPTBot+ Disallow: /+User-agent: * -
Sep 2023
The Washington Post Now blocked: GPTBot and ChatGPT-User.
Show exactly what changed
Disallow: /ac2/+ Disallow: /archive/lifestyle/wellness/1999/01/19/an-illness-no-one-understands-tourettes-syndrome/852df0cc-2164-4c83-be8d-0c770d028547/Disallow: /blogs/slow-ride/…Disallow: /native/+ Disallow: /subscribe/braintree/+ Disallow: /subscribe/enterpriseportal/+ Disallow: /subscribe/foryouapi/+ Disallow: /subscribe/lagoon/+ Disallow: /subscribe/offers/service/+ Disallow: /subscribe/onsiteapi/+ Disallow: /subscribe/paywall/+ Disallow: /subscribe/person/+ Disallow: /subscribe/preferenceapi/+ Disallow: /subscribe/subscriptionapi/+ Disallow: /subscribe/user/+ Disallow: /wpost/proxy…+ User-agent: GPTBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /+Sitemap: https://www.washingtonpost.com/arcio/sitemap/master/index/ -
Sep 2023
WebMD Now blocked: GPTBot.
Show exactly what changed
- # Robots.txt file WebMD- # Updated: Oct 2022+ # Robots.txt file WebMD+ # Updated 8/24/2023…Disallow: /Share.aspx*+ Disallow: /mm/+ Disallow: /kapi/+ Disallow: /click*+ Disallow: /drugs/2/search*+ Disallow: /sponsored/+ Disallow: /api/+ Disallow: /search/2/api/+ Disallow: /webmd_static_vue/+ Disallow: /aim/+ User-agent: GPTBot+ Disallow: /+ -
Sep 2023
WIRED Now blocked: GPTBot, CCBot and ChatGPT-User.
Show exactly what changed
Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/…Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/…Disallow: /*%7B%7Bsection%7D%7D+ Disallow: /product/+ User-agent: Google-InspectionTool+ Allow: /++ User-agent: CCBot+ Disallow: /++ User-agent: GPTBot+ Disallow: /++ User-agent: ChatGPT-User+ Disallow: /+Sitemap: https://www.wired.com/sitemap.xml…Sitemap: https://www.wired.com/contributors-sitemap.xml+ Sitemap: https://www.wired.com/categories-sitemap.xml?hierarchy=tags#Sitemap: https://www.wired.com/categories-sitemap.xml+ Sitemap: https://www.wired.com/sitemap-archive-1.xml
Aug 2023
-
Aug 2023
Ticketmaster Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
Show exactly what changed
User-agent: *- Disallow: /beta/- Disallow: /goto*- Disallow: /seatingchart/- Disallow: /change_area- Disallow: /find_area- Disallow: /error- Disallow: /cgi/outsider.plx- Disallow: /cgi/sfxoutsider.plx- Disallow: /cgi/tt.plx- Disallow: /healthcheck- Disallow: /cgi/mail/mailform2.plx- Disallow: /CustomerServiceMailer- Disallow: /feedback/mavs- Disallow: /member/- Disallow: /member?- Disallow: /ada/- Disallow: /member/login- Disallow: /member/logout- Disallow: /member/edit_profile- Disallow: /member/edit_billing- Disallow: /member/edit_alerts- Disallow: /member/edit_subscriptions- Disallow: /member/unsubscribe- Disallow: /member/order_history- Disallow: /checkout/reserve- Disallow: /checkout/reserve_search- Disallow: /checkout/order- Disallow: /checkout/delete_account- Disallow: /checkout/subscribe_ewsi- Disallow: /checkout/abandon- Disallow: /member/optik- Disallow: /offering- Disallow: /ada/- Disallow: /learnmore- Disallow: /img/- Disallow: /dbimages- Disallow: /css/- Disallow: /js/- Disallow: /server-status
Jul 2023
-
Jul 2023
Indeed Now allowed: CCBot.
Show exactly what changed
Disallow: /cmp/*/company-questions- Disallow: /cmp/*/jobs/Disallow: /cmp/*/people…Disallow: /Stellen/title+ Disallow: /survey/*Disallow: /tellafriend+ Disallow: /tmn/ccs/Disallow: /tos/banner.js…Disallow: /cmp/*/company-questions- Disallow: /cmp/*/jobs/Disallow: /cmp/*/people…Disallow: /ofertas/title/- Disallow: /offerta-lavoroDisallow: /offerte-lavoro/IT/…Disallow: /pagead/- Disallow: /poka%C5%BCprac%C4%99?Disallow: /praca/…Disallow: /Stellen/title+ Disallow: /survey/*Disallow: /tellafriend+ Disallow: /tmn/ccs/Disallow: /tos/banner.js…Disallow: /vacature/- Disallow: /vacature-bekijken?Disallow: /vacatures/NL/Disallow: /vacatures/title/- Disallow: /ver-empleo?- Disallow: /ver-emprego?- Disallow: /ver-oferta?- Disallow: /visajobb?- Disallow: /voir-emploi?- Disallow: /Zeige-Job?…
Jun 2023
-
Jun 2023
OpenTable Now blocked: ChatGPT-User.
Show exactly what changed
Disallow: /httphandlers- Disallow: /i/Disallow: /img…Disallow: /dapi/fe/human+ Disallow: /lolz-view-all/…+ User-agent: ChatGPT-User+ Disallow: /+
May 2023
-
May 2023
The New York Times Now blocked: CCBot.
Show exactly what changed
+ User-agent: CCBot+ Disallow: /+Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz
Apr 2023
-
Apr 2023
Expedia Named ChatGPT-User explicitly, without changing what it may reach.
Show exactly what changed
+ #+ # OpenAI+ #+ user-agent: ChatGPT-User+ allow: / -
Apr 2023
Instacart Named ChatGPT-User explicitly, without changing what it may reach.
Show exactly what changed
### Instacart's Terms of Service https://www.instacart.com/terms+ User-Agent: ChatGPT-User+ Allow: /store/partner_recipes+User-Agent: Twitterbot…Disallow: /store/order_changes/- Disallow: /store/*/storefront*Disallow: /store/*/departments*…Disallow: /store/*/browse_departments*- Disallow: /store/*/collections/*Disallow: /store/*/ng/l/*…Allow: /store/categories/*- Allow: /store/walmart/storefront- Allow: /store/lowes-home-improvement/storefront- Allow: /store/bath-and-body-works/storefrontAllow: /store/hub/*…Allow: /store/brands/*-- Allow: /store/1-west-dupont-circle-wines-liquors/storefront- Allow: /store/111-lex-liquors-wine/storefront- Allow: /store/3-guys-from-brooklyn/storefront- Allow: /store/34-wines-spirits/storefront- Allow: /store/7-day-farmers-market/storefront- Allow: /store/71st-lisbon-sentry/storefront- Allow: /store/92nd-lisbon-sentry/storefront- Allow: /store/a1-market/storefront- Allow: /store/ada-fresh-market/storefront- Allow: /store/adel-wine-liquor/storefront- Allow: /store/alaqsa-international-groceries/storefront- Allow: /store/aldea-home-baby/storefront- Allow: /store/altayebat-market/storefront- Allow: /store/amigos/storefront- Allow: /store/andys-local-market/storefront- Allow: /store/apple-market/storefront- Allow: /store/apple-valley-market/storefront- Allow: /store/aqui-market/storefront -
Apr 2023
KAYAK Named ChatGPT-User explicitly, without changing what it may reach.
Show exactly what changed
- #Build version: R647b- #Generated on: Fri Dec 30 01:00:01 EST 2022+ #Build version: R660d+ #Generated on: Fri Mar 31 01:00:02 EDT 2023…Disallow: /i/api/search/seo++ User-agent: ChatGPT-User+ Allow: /flights/+ Allow: /hotels/+ Allow: /cars/+ Allow: /explore/+ Allow: /sherlock/+ Disallow: /api/+ Disallow: /a/+ Disallow: /i/+ Disallow: /carreservation+ Disallow: /hotelreservation+ Disallow: /flightreservation+ Disallow: /mscarreservation+ Disallow: /SNflightreservation+ Disallow: /msflightreservation+ Disallow: /mshotelreservation+ Disallow: /FDcarreservation+ Disallow: /FDflightreservation+ Disallow: /FDhotelreservation+ Disallow: /in+ Disallow: /h/+ Disallow: /s/+ Disallow: /k/+ Disallow: /r/+User-agent: AdsBot-Google…Allow: /flights/$+ Allow: /flights/*Allow: /cars/$+ Allow: /cars/*Allow: /hotels/sitemap
Mar 2023
-
Mar 2023
Indeed Now blocked: CCBot.
Show exactly what changed
Disallow: /alert+ Disallow: /api/fetch/mc-anonDisallow: /api/getrecjobs…Disallow: /create-resume/lp/+ Disallow: /cdn-cgi/Disallow: /%E4%BB%95%E4%BA%8B?…Disallow: /g/+ Disallow: /graphqlDisallow: /imgping…Disallow: /me/*/pdf+ Disallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?…Disallow: /alert+ Disallow: /api/fetch/mc-anonDisallow: /api/getrecjobs…Disallow: /g/+ Disallow: /graphqlDisallow: /imgping…Disallow: /me/*/pdf+ Disallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?…Disallow: /alert+ Disallow: /api/fetch/mc-anonDisallow: /api/getrecjobs…Disallow: /g/+ Disallow: /graphqlDisallow: /imgping…Disallow: /me/*/pdf+ Disallow: /m/basecamp/analytics.jsDisallow: /m/jobalerts?…