The Walled Web Newsletter

All sites / Retail and marketplaces

Instacart

instacart.com
Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 22 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. OAI-SearchBot and ChatGPT-User can reach most of it. The same rules ask our bot to stay out, so we read only robots.txt.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Blocked

9 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

7 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Some blocked

6 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Blocked
GPTBotOpenAI Blocked Not named; the catch-all rule applies Says yes
ClaudeBotAnthropic Blocked Not named; the catch-all rule applies Says yes
Google-ExtendedGoogle · opt-out name, not a bot Blocked Not named; the catch-all rule applies Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Not named; the catch-all rule applies Says yes
CCBotCommon Crawl Blocked Not named; the catch-all rule applies Says yes
Meta-ExternalAgentMeta Blocked Named, line 520 Suggests yes
BytespiderByteDance Blocked Not named; the catch-all rule applies No statement found
MistralAI-TrainingMistral AI Blocked Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Blocked Not named; the catch-all rule applies Does not say
anthropic-aiAnthropic (legacy) · old name Blocked Not named; the catch-all rule applies Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI Allowed Named, line 313 Says yes
Claude-SearchBotAnthropic Blocked Not named; the catch-all rule applies Says yes
PerplexityBotPerplexity Blocked Named, line 346 Suggests yes
AmazonbotAmazon Blocked Not named; the catch-all rule applies Says yes
Amzn-SearchBotAmazon Blocked Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Blocked Not named; the catch-all rule applies Says yes
MistralAI-IndexMistral AI Blocked Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Blocked Not named; the catch-all rule applies Suggests yes
AI assistants · Some blocked
ChatGPT-UserOpenAI Allowed Named, line 385 Says not always
Claude-UserAnthropic Blocked Not named; the catch-all rule applies Says yes
Perplexity-UserPerplexity Blocked Named, line 349 Says it mostly ignores it
Google-AgentGoogle Blocked Not named; the catch-all rule applies Says it mostly ignores it
Meta-ExternalFetcherMeta Blocked Not named; the catch-all rule applies Says not always
Amzn-UserAmazon Blocked Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Blocked Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNamed
BingbotRegular search engine, for comparisonAllowedNamed

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

ApplebotHas rulesline 352Documented by its operator
FacebookBotBlockedline 526Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 534 lines · 107 lines about AI bots Source
### Any bot scraping or crawling this site must abide by### Terms of Service https://www.instacart.com/terms### Last Updated 02-20-2026 ### Rules for Google search ###User-Agent: GooglebotAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: Googlebot-ImageAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: Googlebot-MobileAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: Storebot-GoogleAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/meals/*Disallow: /store/*/l/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: Adsbot-Google-MobileAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: Adsbot-GoogleAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= ### Rules for Bing Search and powered by Bing search ###User-Agent: BingbotAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= ### Rules for Yahoo! Search ###User-Agent: SlurpAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= ### Rules for DuckDuckGo Search ###User-Agent: DuckDuckGoAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= … 311 lines not about AI bots### Rules for Additional Search Engines ###User-Agent: OAI-SearchBotAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: PerplexityBotDisallow: / User-Agent: Perplexity-UserDisallow: / User-agent: ApplebotAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: ChatGPT-UserAllow: /Allow: /store/brands/l/pads-tampons-liners-wipesDisallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/meals/*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /store/*/meals/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= ### Rules for Social Sharing and Link Previews ###User-Agent: LinkedInBotAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: TwitterbotAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= User-Agent: facebookexternalhitDisallow: /assets User-Agent: FacebotDisallow: /assets User-Agent: Slackbot-LinkExpandingAllow: /Disallow: /accounts/unlockDisallow: /api/Disallow: /ex/Disallow: /orders/Disallow: /store/orders/Disallow: /redeem_coupon_tokenDisallow: /store/order_changes/Disallow: /store/*/browse_departments*Disallow: /store/*/ng/l/*Disallow: /store/referrals*Disallow: /store/item*Disallow: /store/pickup/status/*Disallow: /store/*/buy_it_again*Disallow: /store/*/l/*Disallow: /widgets/*Disallow: /client-side-redirect/*Disallow: /track/*Disallow: /affiliate-program*Disallow: /family/invite/*Disallow: /family/accept/*Disallow: /rate/orders/*Disallow: /store_manager_adminDisallow: /rest/v8/Disallow: /login?next=Disallow: /*?*interaction_id=Disallow: /*?*item_detail_product_id=Disallow: /store/*/collections/*?*sisid=Disallow: /store/*/*?*xrs_id= … 100 lines not about AI bots### Rules for Meta AI Training Crawlers ###User-Agent: meta-externalagentDisallow: / User-Agent: Meta-ExternalAgentDisallow: / User-Agent: FacebookBotDisallow: / ### All other bots ###User-Agent: *Disallow: / ### If you're not a bot, we're hiring: https://instacart.careers/current-openings/ … 6 lines not about AI bots

Notes about AI written in the file: "## Any bot scraping or crawling this site must abide by" · "## Rules for Meta AI Training Crawlers ###"

Everything else we checked

Front doorWe did not visit: the rules ask our bot to stay out.
llms.txtDisallowed for AgentAccessIndexBot by robots.txt
ai.txtDisallowed for AgentAccessIndexBot by robots.txt
TDMRepDisallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report)
Page tagsHomepage not read.
Anti-bot softwareServed through Amazon CloudFront.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIHomepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of ServiceNo Terms of Service page found.
Not foundNo Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

42 versions on record since 1 Jan 2023

OverallJan 2023: Overall allowedFeb 2023: Overall allowedMar 2023: Overall allowedApr 2023: Overall allowedMay 2023: Overall allowedJun 2023: Overall allowedJul 2023: Overall allowedAug 2023: Overall allowedSep 2023: Overall allowedOct 2023: Overall allowedNov 2023: Overall allowedDec 2023: Overall allowedJan 2024: Overall allowedFeb 2024: Overall allowedMar 2024: Overall allowedApr 2024: Overall allowedMay 2024: Overall allowedJun 2024: Overall allowedJul 2024: Overall allowedAug 2024: Overall allowedSep 2024: Overall allowedOct 2024: Overall allowedNov 2024: Overall allowedDec 2024: Overall allowedJan 2025: Overall allowedFeb 2025: Overall allowedMar 2025: Overall partialApr 2025: Overall partialMay 2025: Overall partialJun 2025: Overall partialJul 2025: Overall partialAug 2025: Overall partialSep 2025: Overall partialOct 2025: Overall partialNov 2025: Overall partialDec 2025: Overall partialJan 2026: Overall partialFeb 2026: Overall partialMar 2026: Overall partialApr 2026: Overall partialMay 2026: Overall partialJun 2026: Overall partialJul 2026: Overall partialAug 2026: Overall partialSep 2026: Overall partialOct 2026: Overall partialTrainingJan 2023: Training AllowedFeb 2023: Training AllowedMar 2023: Training AllowedApr 2023: Training AllowedMay 2023: Training AllowedJun 2023: Training AllowedJul 2023: Training AllowedAug 2023: Training AllowedSep 2023: Training AllowedOct 2023: Training AllowedNov 2023: Training AllowedDec 2023: Training AllowedJan 2024: Training AllowedFeb 2024: Training AllowedMar 2024: Training AllowedApr 2024: Training AllowedMay 2024: Training AllowedJun 2024: Training AllowedJul 2024: Training AllowedAug 2024: Training AllowedSep 2024: Training AllowedOct 2024: Training AllowedNov 2024: Training AllowedDec 2024: Training AllowedJan 2025: Training AllowedFeb 2025: Training AllowedMar 2025: Training BlockedApr 2025: Training BlockedMay 2025: Training BlockedJun 2025: Training BlockedJul 2025: Training BlockedAug 2025: Training BlockedSep 2025: Training BlockedOct 2025: Training BlockedNov 2025: Training BlockedDec 2025: Training BlockedJan 2026: Training BlockedFeb 2026: Training BlockedMar 2026: Training BlockedApr 2026: Training BlockedMay 2026: Training BlockedJun 2026: Training BlockedJul 2026: Training BlockedAug 2026: Training BlockedSep 2026: Training BlockedOct 2026: Training BlockedSearchJan 2023: Search AllowedFeb 2023: Search AllowedMar 2023: Search AllowedApr 2023: Search AllowedMay 2023: Search AllowedJun 2023: Search AllowedJul 2023: Search AllowedAug 2023: Search AllowedSep 2023: Search AllowedOct 2023: Search AllowedNov 2023: Search AllowedDec 2023: Search AllowedJan 2024: Search AllowedFeb 2024: Search AllowedMar 2024: Search AllowedApr 2024: Search AllowedMay 2024: Search AllowedJun 2024: Search AllowedJul 2024: Search AllowedAug 2024: Search AllowedSep 2024: Search AllowedOct 2024: Search AllowedNov 2024: Search AllowedDec 2024: Search AllowedJan 2025: Search AllowedFeb 2025: Search AllowedMar 2025: Search Some blockedApr 2025: Search Some blockedMay 2025: Search Some blockedJun 2025: Search Some blockedJul 2025: Search Some blockedAug 2025: Search Some blockedSep 2025: Search Some blockedOct 2025: Search Some blockedNov 2025: Search Some blockedDec 2025: Search Some blockedJan 2026: Search Some blockedFeb 2026: Search Some blockedMar 2026: Search Some blockedApr 2026: Search Some blockedMay 2026: Search Some blockedJun 2026: Search Some blockedJul 2026: Search Some blockedAug 2026: Search Some blockedSep 2026: Search Some blockedOct 2026: Search Some blockedAssistantsJan 2023: Assistants AllowedFeb 2023: Assistants AllowedMar 2023: Assistants AllowedApr 2023: Assistants AllowedMay 2023: Assistants AllowedJun 2023: Assistants AllowedJul 2023: Assistants AllowedAug 2023: Assistants AllowedSep 2023: Assistants AllowedOct 2023: Assistants AllowedNov 2023: Assistants AllowedDec 2023: Assistants AllowedJan 2024: Assistants AllowedFeb 2024: Assistants AllowedMar 2024: Assistants AllowedApr 2024: Assistants AllowedMay 2024: Assistants AllowedJun 2024: Assistants AllowedJul 2024: Assistants AllowedAug 2024: Assistants AllowedSep 2024: Assistants AllowedOct 2024: Assistants AllowedNov 2024: Assistants AllowedDec 2024: Assistants AllowedJan 2025: Assistants AllowedFeb 2025: Assistants AllowedMar 2025: Assistants Some blockedApr 2025: Assistants Some blockedMay 2025: Assistants Some blockedJun 2025: Assistants Some blockedJul 2025: Assistants Some blockedAug 2025: Assistants Some blockedSep 2025: Assistants Some blockedOct 2025: Assistants Some blockedNov 2025: Assistants Some blockedDec 2025: Assistants Some blockedJan 2026: Assistants Some blockedFeb 2026: Assistants Some blockedMar 2026: Assistants Some blockedApr 2026: Assistants Some blockedMay 2026: Assistants Some blockedJun 2026: Assistants Some blockedJul 2026: Assistants Some blockedAug 2026: Assistants Some blockedSep 2026: Assistants Some blockedOct 2026: Assistants Some blockedBotsblockedMar 2025: 21 of 24 AI bots blockedApr 2025: 21 of 24 AI bots blockedMay 2025: 21 of 24 AI bots blockedJun 2025: 21 of 24 AI bots blockedJul 2025: 21 of 24 AI bots blockedAug 2025: 21 of 24 AI bots blockedSep 2025: 21 of 24 AI bots blockedOct 2025: 21 of 24 AI bots blockedNov 2025: 21 of 24 AI bots blockedDec 2025: 21 of 24 AI bots blockedJan 2026: 21 of 24 AI bots blockedFeb 2026: 21 of 24 AI bots blockedMar 2026: 22 of 24 AI bots blockedApr 2026: 22 of 24 AI bots blockedMay 2026: 22 of 24 AI bots blockedJun 2026: 22 of 24 AI bots blockedJul 2026: 22 of 24 AI bots blockedAug 2026: 22 of 24 AI bots blockedSep 2026: 22 of 24 AI bots blockedOct 2026: 22 of 24 AI bots blocked222023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • May 2026
    Signals Named Meta-ExternalAgent explicitly, without changing what it may reach.
    Found in the Internet Archive's copies: changed between 1 Apr 2026 and 1 May 2026. Evidence: before · after
    Show exactly what changed
    + ### Rules for Meta AI Training Crawlers ###
    + User-Agent: meta-externalagent
    + Disallow: /
    +
    + User-Agent: Meta-ExternalAgent
    + Disallow: /
    +
    + User-Agent: FacebookBot
    + Disallow: /
    +
    ### All other bots ###
  • Mar 2026
    More blocking Now blocked: PerplexityBot.
    Found in the Internet Archive's copies: changed between 1 Feb 2026 and 1 Mar 2026. Evidence: before · after
    Show exactly what changed
    ### Instacart's Terms of Service https://www.instacart.com/terms
    - ### Last Updated 2-28-2025
    + ### Last Updated 02-20-2026
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Allow: /
    + Allow: /store/brands/l/pads-tampons-liners-wipes
    Disallow: /accounts/unlock
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Disallow: /ex/
    + Disallow: /orders/
    Disallow: /store/orders/
    …
    Disallow: /store/*/collections/*?*sisid=
    - Disallow: /store/*/*?*xrs_id= 
    + Disallow: /store/*/*?*xrs_id=
    … 61 more lines
  • Mar 2025
    More blocking Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 14 more.
    Found in the Internet Archive's copies: changed between 1 Feb 2025 and 11 Mar 2025. Evidence: before · after
    Show exactly what changed
    ### Instacart's Terms of Service https://www.instacart.com/terms
    - User-Agent: ChatGPT-User
    - Allow: /store/partner_recipes
    + ### Last Updated 2-28-2025
    - User-Agent: Twitterbot
    - Allow: /store/list/*
    + ### Rules for Google search ###
    + User-Agent: Googlebot
    + Allow: /
    + Disallow: /accounts/unlock
    + Disallow: /api/
    + Disallow: /ex/
    + Disallow: /store/orders/
    + Disallow: /redeem_coupon_token
    + Disallow: /store/order_changes/
    + Disallow: /store/*/browse_departments*
    + Disallow: /store/*/ng/l/*
    + Disallow: /store/referrals*
    + Disallow: /store/item*
    + Disallow: /store/pickup/status/*
    + Disallow: /store/*/buy_it_again*
    + Disallow: /store/*/l/*
    + Disallow: /widgets/*
    + Disallow: /client-side-redirect/*
    + Disallow: /track/*
    + Disallow: /affiliate-program*
    + Disallow: /family/invite/*
    + Disallow: /family/accept/*
    + Disallow: /rate/orders/*
    + Disallow: /store_manager_admin
    + Disallow: /rest/v8/
    + Disallow: /login?next=
    + Disallow: /*?*interaction_id=
    + Disallow: /*?*item_detail_product_id=
    + Disallow: /store/*/collections/*?*sisid=
    + Disallow: /store/*/*?*xrs_id=
    - User-Agent: Yandex
    - Disallow: /
    + User-Agent: Googlebot-Image
    + Allow: /
    + Disallow: /accounts/unlock
    + Disallow: /api/
    + Disallow: /ex/
    + Disallow: /store/orders/
    + Disallow: /redeem_coupon_token
    + Disallow: /store/order_changes/
    + Disallow: /store/*/browse_departments*
    + Disallow: /store/*/ng/l/*
    + Disallow: /store/referrals*
    + Disallow: /store/item*
    + Disallow: /store/pickup/status/*
    + Disallow: /store/*/buy_it_again*
    + Disallow: /store/*/l/*
    + Disallow: /widgets/*
    + Disallow: /client-side-redirect/*
    + Disallow: /track/*
    + Disallow: /affiliate-program*
    + Disallow: /family/invite/*
    … 433 more lines
  • Apr 2023
    Signals Named ChatGPT-User explicitly, without changing what it may reach.
    Found in the Internet Archive's copies: changed between 1 Mar 2023 and 1 Apr 2023. Evidence: before · after
    Show exactly what changed
    ### Instacart's Terms of Service https://www.instacart.com/terms
    + User-Agent: ChatGPT-User
    + Allow: /store/partner_recipes
    +
    User-Agent: Twitterbot
    …
    Disallow: /store/order_changes/
    - Disallow: /store/*/storefront*
    Disallow: /store/*/departments*
    …
    Disallow: /store/*/browse_departments*
    - Disallow: /store/*/collections/*
    Disallow: /store/*/ng/l/*
    …
    Allow: /store/categories/*
    - Allow: /store/walmart/storefront
    - Allow: /store/lowes-home-improvement/storefront
    - Allow: /store/bath-and-body-works/storefront
    Allow: /store/hub/*
    …
    Allow: /store/brands/*
    -
    - Allow: /store/1-west-dupont-circle-wines-liquors/storefront
    - Allow: /store/111-lex-liquors-wine/storefront
    - Allow: /store/3-guys-from-brooklyn/storefront
    - Allow: /store/34-wines-spirits/storefront
    - Allow: /store/7-day-farmers-market/storefront
    - Allow: /store/71st-lisbon-sentry/storefront
    - Allow: /store/92nd-lisbon-sentry/storefront
    - Allow: /store/a1-market/storefront
    - Allow: /store/ada-fresh-market/storefront
    - Allow: /store/adel-wine-liquor/storefront
    - Allow: /store/alaqsa-international-groceries/storefront
    - Allow: /store/aldea-home-baby/storefront
    - Allow: /store/altayebat-market/storefront
    - Allow: /store/amigos/storefront
    - Allow: /store/andys-local-market/storefront
    - Allow: /store/apple-market/storefront
    - Allow: /store/apple-valley-market/storefront
    - Allow: /store/aqui-market/storefront
    - Allow: /store/andys-liquor/storefront
    - Allow: /store/associated-supermarkets/storefront
    - Allow: /store/atko-market-bakery/storefront
    - Allow: /store/augusta-beverage/storefront
    - Allow: /store/avery-ranch-wine-spirits/storefront
    - Allow: /store/baja-ranch-supermarket/storefront
    - Allow: /store/balady-market/storefront
    - Allow: /store/bashas-dine/storefront
    - Allow: /store/beer-shack/storefront
    - Allow: /store/bensonhurst-market/storefront
    - Allow: /store/bergenline-marketplace/storefront
    - Allow: /store/beverage-world-package-store/storefront
    - Allow: /store/big-deal-food-market/storefront
    - Allow: /store/bluebonnet-liquor/storefront
    - Allow: /store/bones-baths-pet-care/storefront
    - Allow: /store/bottlebargains/storefront
    - Allow: /store/bottlebuys/storefront
    - Allow: /store/bottlerocket-wine-spirits/storefront
    - Allow: /store/bottlevalues/storefront
    - Allow: /store/boulevard-liquor/storefront
    … 694 more lines

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.