All sites / Retail and marketplaces
Amazon
amazon.comBlocks some
Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 17 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, CCBot and Meta-ExternalAgent. Applebot-Extended can reach most of it. Amazon told Meta's shopping agent that access by an unauthorized AI agent violates its Conditions of Use (The Next Web, 21 Sep 2026).
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
7 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Some blocked
5 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Some blocked
5 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Blocked | Named, line 143 | Says yes |
| ClaudeBotAnthropic | Blocked | Named, line 162 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Blocked | Named, line 153 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Blocked | Named, line 147 | Says yes |
| Meta-ExternalAgentMeta | Blocked | Named, line 165 | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 168 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Blocked | Named, line 183 | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Some blocked | |||
| OAI-SearchBotOpenAI | Blocked | Named, line 276 | Says yes |
| Claude-SearchBotAnthropic | Blocked | Named, line 252 | Says yes |
| PerplexityBotPerplexity | Blocked | Named, line 150 | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Blocked | Named, line 204 | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Blocked | Named, line 390 | Suggests yes |
| AI assistants · Some blocked | |||
| ChatGPT-UserOpenAI | Blocked | Named, line 273 | Says not always |
| Claude-UserAnthropic | Blocked | Named, line 246 | Says yes |
| Perplexity-UserPerplexity | Blocked | Named, line 255 | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Blocked | Named, line 216 | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Blocked | Named, line 192 | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
| BingbotRegular search engine, for comparison | Allowed | Not named; the catch-all rule applies | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| GoogleAgent-Mariner | Blocked | line 156 | Not in our verified list |
| GoogleAgent-Shopping | Blocked | line 159 | Not in our verified list |
| PetalBot | Blocked | line 174 | Not in our verified list |
| omgili | Blocked | line 180 | Not in our verified list |
| Gemini-Deep-Research | Blocked | line 186 | Not in our verified list |
| Diffbot | Blocked | line 195 | Documented by its operator |
| Copilot | Blocked | line 207 | Not in our verified list |
| Ai2Bot-Dolma | Blocked | line 210 | Not in our verified list |
| Google-CloudVertexBot | Blocked | line 213 | Documented by its operator |
| cohere-ai | Blocked | line 228 | Not in our verified list |
| YouBot | Blocked | line 240 | Not in our verified list |
| CopilotNative | Blocked | line 261 | Not in our verified list |
| CopilotSapphire | Blocked | line 264 | Not in our verified list |
| Timpibot | Blocked | line 279 | Not in our verified list |
| AI2Bot-DeepResearchEval | Blocked | line 288 | Not in our verified list |
| Claude-Web | Blocked | line 309 | Not in our verified list |
| cohere-training-data-crawler | Blocked | line 318 | Not in our verified list |
| DeepSeekBot | Blocked | line 330 | Not in our verified list |
| GoogleOther | Blocked | line 345 | Documented by its operator |
| Grok-DeepSearch | Blocked | line 348 | Operator publishes no such name |
| GrokBot | Blocked | line 351 | Operator publishes no such name |
| ImagesiftBot | Blocked | line 363 | Not in our verified list |
| Manus-User | Blocked | line 387 | Not in our verified list |
| webzio-extended | Blocked | line 426 | Not in our verified list |
| xAI-Grok | Blocked | line 432 | Operator publishes no such name |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
User-agent: *Disallow: /exec/obidos/account-access-loginDisallow: /exec/obidos/change-styleDisallow: /exec/obidos/flex-sign-inDisallow: /exec/obidos/handle-buy-boxDisallow: /exec/obidos/tg/cm/member/Disallow: /gp/aw/help/id=sssDisallow: /gp/cartDisallow: /gp/flexDisallow: /gp/product/e-mail-friendDisallow: /gp/product/product-availabilityDisallow: /gp/product/rate-this-itemDisallow: /gp/sign-inDisallow: /gp/readerDisallow: /gp/sitbv3/readerDisallow: /gp/richpub/syltguides/createDisallow: /gp/gfixDisallow: /gp/associations/wizard.htmlDisallow: /gp/dmusic/orderDisallow: /gp/legacy-handle-buy-box.htmlDisallow: /gp/aws/ssopDisallow: /gp/yourstoreDisallow: /gp/gift-central/organizer/add-wishlistDisallow: /gp/voteDisallow: /gp/voting/Disallow: /gp/music/wma-pop-upDisallow: /gp/customer-imagesDisallow: /gp/richpub/listmania/createpipelineDisallow: /gp/content-formDisallow: /gp/pdp/invitation/inviteDisallow: /gp/customer-reviews/common/duDisallow: /gp/customer-reviews/write-a-review.htmlDisallow: /gp/associations/wizard.htmlDisallow: /gp/music/clipserveDisallow: /gp/customer-media/uploadDisallow: /gp/historyDisallow: /gp/item-dispatchDisallow: /gp/dmusic/order/handle-buy-box.htmlDisallow: /gp/recsradioDisallow: /gp/slredirectDisallow: /dp/shipping/Disallow: /dp/twister-update/Disallow: /dp/manual-submit/Disallow: /dp/e-mail-friend/Disallow: /dp/product-availability/Disallow: /dp/rate-this-item/Disallow: /gp/registry/wishlist/*/reserveDisallow: /gp/structured-ratings/actions/get-experience.htmlDisallow: /gp/twitter/Disallow: /ap/signinDisallow: /gp/registry/wishlist/Disallow: /wishlist/Allow: /wishlist/universal*Allow: /wishlist/vendor-button*Allow: /wishlist/get-button*Disallow: /gp/wishlist/Allow: /gp/wishlist/universal*Allow: /gp/wishlist/vendor-button*Allow: /gp/wishlist/ipad-install*Disallow: /registry/wishlist/Disallow: /review/common/duDisallow: /gp/registry/search.htmlDisallow: /product-reviews/B0069IY63YDisallow: /gp/orc/rml/Disallow: */gcrnstsDisallow: /gp/gc/widgetDisallow: /gp/dmusic/mp3/playerDisallow: /gp/entity-alert/externalDisallow: /gp/customer-reviews/dynamic/sims-boxDisallow: /review/dynamic/sims-boxDisallow: /gp/redirect.htmlDisallow: /gp/twister/ajaxv2Disallow: /ss/twister/ajaxDisallow: /b?*node=7454917011Disallow: /b?*node=7454927011Disallow: /b?*node=7454939011Disallow: /b?*node=7454898011Disallow: /gp/customer-media/actions/delete/Disallow: /gp/customer-media/actions/edit-caption/Disallow: /gp/dmusic/Allow: /gp/dmusic/promotions/PrimeMusicAllow: /gp/dmusic/promotions/AmazonMusicUnlimitedDisallow: /gp/offer-listing/Disallow: /b?*node=9052533011Disallow: /lm/R1XIHQVKXSKBNJDisallow: /lm/R3HQ5WJSZK6QSODisallow: /surprise/Disallow: /local/ajax/Disallow: /gp/socialmedia/giveawaysDisallow: /gp/b2b-rdDisallow: /gp/aw/so.htmlDisallow: /reviews/iframeDisallow: /gp/switch-languageDisallow: /ga/p/Disallow: /gp/profile/Disallow: /giveaway/host/setup/Disallow: /ss/customer-reviews/lighthouse/Disallow: /ospublishing/story/*Disallow: /gp/aw/ol/Disallow: /gp/promotion/Disallow: /hz/leaderboard/top-reviewers/Disallow: /creatorhubDisallow: /creatorhub/*Disallow: /slp/s$Disallow: /-/Allow: /-/es/Allow: /-/en$Allow: /-/zh_TW/Allow: /-/zh_TW$Allow: /-/he/Allow: /-/he$Disallow: /hz/help/contact/*/message/$Disallow: /gp/aw/shoppingAids/Disallow: /rss/people/*/reviewsDisallow: /gp/pdp/rss/*/reviewsDisallow: /gp/cdp/member-reviews/Disallow: /gp/aw/cr/Allow: /gp/offer-listing/B000Allow: /gp/offer-listing/9000Disallow: /gp/aagAllow: /gp/aag/main?*seller=ABVFEJU8LS620Disallow: /gp/pdp/profile/Disallow: /gp/help/customer/express/c2c/Disallow: /slp/*/b$Disallow: /hz/contact-us/ajax/initiate-trusted-contact/Disallow: /hp/video/Disallow: /gp/video/apiDisallow: /gp/video/settingsDisallow: /gp/video/libraryDisallow: /gp/video/watchlistDisallow: /gp/video/mystuffDisallow: /gp/video/profilesDisallow: /gp/video/searchDisallow: /gp/video/authDisallow: /gp/video/salp/iDisallow: /icdp/Disallow: /shoptheshow/*?cat=Disallow: /shoptheshow/*&cat= User-agent: EtaoSpiderDisallow: /… 141 lines not about AI bots User-agent: GPTBotDisallow: / User-agent: CCBotDisallow: / User-Agent: PerplexityBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: GoogleAgent-MarinerDisallow: / User-agent: GoogleAgent-ShoppingDisallow: / User-agent: ClaudeBotDisallow: / User-agent: meta-externalagentDisallow: / User-agent: BytespiderDisallow: / User-agent: ScrapyDisallow: /… 2 lines not about AI bots User-agent: PetalBotDisallow: / User-agent: DevinDisallow: /… 2 lines not about AI bots User-agent: omgiliDisallow: / User-agent: AI2BotDisallow: / User-agent: Gemini-Deep-ResearchDisallow: / User-agent: PanguBotDisallow: /… 2 lines not about AI bots User-agent: MistralAI-UserDisallow: / User-agent: DiffbotDisallow: / User-agent: Sidetrade indexer botDisallow: / User-agent: Google-NotebookLMDisallow: /… 5 lines not about AI bots User-agent: DuckAssistBotDisallow: / User-agent: CopilotDisallow: / User-agent: Ai2Bot-DolmaDisallow: / User-agent: Google-CloudVertexBotDisallow: / User-agent: meta-externalfetcherDisallow: / User-agent: iaskspiderDisallow: / User-agent: wpbotDisallow: / User-agent: aiHitBotDisallow: /… 8 lines not about AI bots User-agent: cohere-aiDisallow: / User-agent: FriendlyCrawlerDisallow: / User-agent: img2datasetDisallow: / User-agent: VelenPublicWebCrawlerDisallow: /… 8 lines not about AI bots User-agent: YouBotDisallow: / User-agent: BrightbotDisallow: /… 2 lines not about AI bots User-agent: Claude-UserDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Claude-SearchBotDisallow: / User-agent: Perplexity-UserDisallow: / User-agent: ISSCyberRiskCrawlerDisallow: /… 2 lines not about AI bots User-agent: CopilotNativeDisallow: / User-agent: CopilotSapphireDisallow: / User-agent: imgproxyDisallow: / User-agent: SemrushBot-SWADisallow: /… 5 lines not about AI bots User-agent: ChatGPT-UserDisallow: / User-agent: OAI-SearchBotDisallow: / User-agent: TimpibotDisallow: / User-agent: ThinkbotDisallow: / User-agent: AddSearchBotDisallow: /… 5 lines not about AI bots User-agent: AI2Bot-DeepResearchEvalDisallow: / User-agent: AndibotDisallow: / User-agent: AnomuraDisallow: / User-agent: atlassian-botDisallow: / User-agent: BuddyBotDisallow: / User-agent: Channel3BotDisallow: / User-agent: ChatGLM-SpiderDisallow: /… 17 lines not about AI bots User-agent: Claude-WebDisallow: / User-agent: Cloudflare-AutoRAGDisallow: / User-agent: CloudflareBrowserRenderingCrawlerDisallow: /… 5 lines not about AI bots User-agent: cohere-training-data-crawlerDisallow: / User-agent: CotoyogiDisallow: / User-agent: Crawl4AIDisallow: / User-agent: Datenbank CrawlerDisallow: /… 8 lines not about AI bots User-agent: DeepSeekBotDisallow: / User-agent: Echobot BotDisallow: / User-agent: EchoboxBotDisallow: / User-agent: factset_spyderbotDisallow: / User-agent: Google-FirebaseDisallow: /… 11 lines not about AI bots User-agent: GoogleOtherDisallow: / User-agent: Grok-DeepSearchDisallow: / User-agent: GrokBotDisallow: / User-agent: iAskBotDisallow: / User-agent: IbouBotDisallow: / User-agent: ICC-CrawlerDisallow: /… 8 lines not about AI bots User-agent: ImagesiftBotDisallow: / User-agent: imageSpiderDisallow: / User-agent: Kangaroo BotDisallow: / User-agent: KlaviyoAIBotDisallow: / User-agent: KunatoCrawlerDisallow: / User-agent: laion-huggingface-processorDisallow: / User-agent: Linguee BotDisallow: / User-agent: LinkupBotDisallow: /… 20 lines not about AI bots User-agent: Manus-UserDisallow: / User-agent: meta-webindexerDisallow: / User-agent: netEstate Imprint CrawlerDisallow: / User-agent: PanscientDisallow: / User-agent: panscient.comDisallow: / User-agent: PhindBotDisallow: / User-agent: Poggio-CitationsDisallow: / User-agent: QualifiedBotDisallow: / User-agent: SBIntuitionsBotDisallow: / User-agent: ShapBotDisallow: / User-agent: TavilyBotDisallow: / User-agent: TerraCottaDisallow: / User-agent: TwinAgentDisallow: /… 32 lines not about AI bots User-agent: webzio-extendedDisallow: / User-agent: WRTNBotDisallow: /… 2 lines not about AI bots User-agent: xAI-GrokDisallow: / User-agent: YaKDisallow: / User-agent: ZanistaBotDisallow: /… 5 lines not about AI bots
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.amazon.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Anti-bot software | Served through Amazon CloudFront.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | No schema.org structured data on the homepage. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | www.amazon.com/gp/help/customer/display.html?nodeId=508088 (curated link, HTTP 403) |
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags. |
History
47 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Jun 2026More blocking Now blocked: Meta-WebIndexer.Found in the Internet Archive's copies: changed between 1 May 2026 and 7 Jun 2026. Evidence: before · after
Show exactly what changed
Disallow: /++ User-agent: Timpibot+ Disallow: /++ User-agent: Thinkbot+ Disallow: /++ User-agent: AddSearchBot+ Disallow: /++ User-agent: AI2Bot-DeepResearchEval+ Disallow: /++ User-agent: Andibot+ Disallow: /++ User-agent: Anomura+ Disallow: /++ User-agent: atlassian-bot+ Disallow: /++ User-agent: BuddyBot+ Disallow: /++ User-agent: Channel3Bot+ Disallow: /++ User-agent: ChatGLM-Spider+ Disallow: /++ User-agent: Claude-Web+ Disallow: /++ User-agent: Cloudflare-AutoRAG+ Disallow: /++ User-agent: CloudflareBrowserRenderingCrawler+ Disallow: /++ User-agent: cohere-training-data-crawler+ Disallow: /++ User-agent: Cotoyogi+ Disallow: /++ User-agent: Crawl4AI+ Disallow: /++ User-agent: Datenbank Crawler+ Disallow: /++ User-agent: DeepSeekBot+ Disallow: /++ User-agent: Echobot Bot+ Disallow: /++ User-agent: EchoboxBot… 103 more lines - Dec 2025More blocking Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.Found in the Internet Archive's copies: changed between 1 Oct 2025 and 1 Dec 2025. Evidence: before · after
Show exactly what changed
Disallow: /gp/video/salp/i+ Disallow: /icdp/…Disallow: /++ User-agent: Copilot+ Disallow: /++ User-agent: Ai2Bot-Dolma+ Disallow: /++ User-agent: Google-CloudVertexBot+ Disallow: /++ User-agent: meta-externalfetcher+ Disallow: /++ User-agent: iaskspider+ Disallow: /++ User-agent: wpbot+ Disallow: /++ User-agent: aiHitBot+ Disallow: /++ User-agent: cohere-ai+ Disallow: /++ User-agent: FriendlyCrawler+ Disallow: /++ User-agent: img2dataset+ Disallow: /++ User-agent: VelenPublicWebCrawler+ Disallow: /++ User-agent: YouBot+ Disallow: /++ User-agent: Brightbot+ Disallow: /++ User-agent: Claude-User+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: Claude-SearchBot+ Disallow: /++ User-agent: Perplexity-User+ Disallow: /++ User-agent: ISSCyberRiskCrawler+ Disallow: /+… 17 more lines - Jun to Sep 2025More blocking Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.Found in the Internet Archive's copies: changed between 1 Jun 2025 and 3 Sep 2025. Evidence: before · after
Show exactly what changed
Disallow: /++ User-agent: Google-Extended+ Disallow: /++ User-agent: GoogleAgent-Mariner+ Disallow: /++ User-agent: GoogleAgent-Shopping+ Disallow: /++ User-agent: ClaudeBot+ Disallow: /++ User-agent: meta-externalagent+ Disallow: /++ User-agent: Bytespider+ Disallow: /++ User-agent: Scrapy+ Disallow: /++ User-agent: PetalBot+ Disallow: /++ User-agent: Devin+ Disallow: /++ User-agent: omgili+ Disallow: /++ User-agent: AI2Bot+ Disallow: /++ User-agent: Gemini-Deep-Research+ Disallow: /++ User-agent: PanguBot+ Disallow: /++ User-agent: MistralAI-User+ Disallow: /++ User-agent: Diffbot+ Disallow: /++ User-agent: Sidetrade indexer bot+ Disallow: /++ User-agent: Google-NotebookLM+ Disallow: /++ User-agent: DuckAssistBot+ Disallow: / - Dec 2024More blocking Now blocked: PerplexityBot.Found in the Internet Archive's copies: changed between 1 Nov 2024 and 1 Dec 2024. Evidence: before · after
Show exactly what changed
Disallow: /local/ajax/- Disallow: */B00M3E1NYI- Disallow: */B00M3E1Q5Y- Disallow: */B00M3E1TOM- Disallow: */B00M3E1WYO- Disallow: */B00M3E204K- Disallow: */B00M3E236A- Disallow: */B00M3E260I- Disallow: */B00M3E28WO- Disallow: */B00M3E2BC6- Disallow: */B00M3E2DPQ- Disallow: */B00M3E2GU8- Disallow: */B00M3E2J14- Disallow: */B00M3E2LOE- Disallow: */B00M3E1HJYDisallow: /gp/socialmedia/giveaways…Disallow: /gp/aw/so.html- Disallow: /gp/rentallist- Disallow: /gp/video/dvd-rental/settings- Disallow: /gp/rl/settings- Disallow: /gp/video/settings- Disallow: /gp/video/library- Disallow: /gp/video/watchlistDisallow: /reviews/iframe…Allow: /-/es/+ Allow: /-/en$Allow: /-/zh_TW/+ Allow: /-/zh_TW$Allow: /-/he/+ Allow: /-/he$Disallow: /hz/help/contact/*/message/$…Disallow: /gp/aw/cr/- Disallow: */sim/B001132UEEAllow: /gp/offer-listing/B000…Disallow: /hz/contact-us/ajax/initiate-trusted-contact/+ Disallow: /hp/video/Disallow: /gp/video/api- Disallow: /hp/video/api+ Disallow: /gp/video/settings+ Disallow: /gp/video/library+ Disallow: /gp/video/watchlistDisallow: /gp/video/mystuff- Disallow: /hp/video/mystuffDisallow: /gp/video/profiles- Disallow: /hp/video/profiles+ Disallow: /gp/video/search+ Disallow: /gp/video/auth+ Disallow: /gp/video/salp/i…+User-agent: CCBotDisallow: /++ User-Agent: PerplexityBot… 1 more lines - Oct 2023
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.