{"domain":"amazon.com","name":"Amazon","category":"retail","subcategory":"marketplace","country":"US","tags":["named-claim"],"notes":"Named in the @trikcode post (Grok and other shopping agents)","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 17 of the 24 AI bots we track from the whole site, including GPTBot, ClaudeBot, Google-Extended, CCBot and Meta-ExternalAgent. Applebot-Extended can reach most of it. Amazon told Meta's shopping agent that access by an unauthorized AI agent violates its Conditions of Use (The Next Web, 21 Sep 2026).","computed_at":"2026-10-20T03:43:13.360Z","last_checked":"2026-10-19T13:23:46.494Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.amazon.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.amazon.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.amazon.com/robots.txt","hops":[{"from":"https://amazon.com/robots.txt","to":"https://www.amazon.com/robots.txt","status":301}],"blob":"74fa27e28cd9d80bdd155ffecf3a033ea2e195a80c83d6fa46acb20665c3cb4d","text":"User-agent: *\nDisallow: /exec/obidos/account-access-login\nDisallow: /exec/obidos/change-style\nDisallow: /exec/obidos/flex-sign-in\nDisallow: /exec/obidos/handle-buy-box\nDisallow: /exec/obidos/tg/cm/member/\nDisallow: /gp/aw/help/id=sss\nDisallow: /gp/cart\nDisallow: /gp/flex\nDisallow: /gp/product/e-mail-friend\nDisallow: /gp/product/product-availability\nDisallow: /gp/product/rate-this-item\nDisallow: /gp/sign-in\nDisallow: /gp/reader\nDisallow: /gp/sitbv3/reader\nDisallow: /gp/richpub/syltguides/create\nDisallow: /gp/gfix\nDisallow: /gp/associations/wizard.html\nDisallow: /gp/dmusic/order\nDisallow: /gp/legacy-handle-buy-box.html\nDisallow: /gp/aws/ssop\nDisallow: /gp/yourstore\nDisallow: /gp/gift-central/organizer/add-wishlist\nDisallow: /gp/vote\nDisallow: /gp/voting/\nDisallow: /gp/music/wma-pop-up\nDisallow: /gp/customer-images\nDisallow: /gp/richpub/listmania/createpipeline\nDisallow: /gp/content-form\nDisallow: /gp/pdp/invitation/invite\nDisallow: /gp/customer-reviews/common/du\nDisallow: /gp/customer-reviews/write-a-review.html\nDisallow: /gp/associations/wizard.html\nDisallow: /gp/music/clipserve\nDisallow: /gp/customer-media/upload\nDisallow: /gp/history\nDisallow: /gp/item-dispatch\nDisallow: /gp/dmusic/order/handle-buy-box.html\nDisallow: /gp/recsradio\nDisallow: /gp/slredirect\nDisallow: /dp/shipping/\nDisallow: /dp/twister-update/\nDisallow: /dp/manual-submit/\nDisallow: /dp/e-mail-friend/\nDisallow: /dp/product-availability/\nDisallow: /dp/rate-this-item/\nDisallow: /gp/registry/wishlist/*/reserve\nDisallow: /gp/structured-ratings/actions/get-experience.html\nDisallow: /gp/twitter/\nDisallow: /ap/signin\nDisallow: /gp/registry/wishlist/\nDisallow: /wishlist/\nAllow: /wishlist/universal*\nAllow: /wishlist/vendor-button*\nAllow: /wishlist/get-button*\nDisallow: /gp/wishlist/\nAllow: /gp/wishlist/universal*\nAllow: /gp/wishlist/vendor-button*\nAllow: /gp/wishlist/ipad-install*\nDisallow: /registry/wishlist/\nDisallow: /review/common/du\nDisallow: /gp/registry/search.html\nDisallow: /product-reviews/B0069IY63Y\nDisallow: /gp/orc/rml/\nDisallow: */gcrnsts\nDisallow: /gp/gc/widget\nDisallow: /gp/dmusic/mp3/player\nDisallow: /gp/entity-alert/external\nDisallow: /gp/customer-reviews/dynamic/sims-box\nDisallow: /review/dynamic/sims-box\nDisallow: /gp/redirect.html\nDisallow: /gp/twister/ajaxv2\nDisallow: /ss/twister/ajax\nDisallow: /b?*node=7454917011\nDisallow: /b?*node=7454927011\nDisallow: /b?*node=7454939011\nDisallow: /b?*node=7454898011\nDisallow: /gp/customer-media/actions/delete/\nDisallow: /gp/customer-media/actions/edit-caption/\nDisallow: /gp/dmusic/\nAllow: /gp/dmusic/promotions/PrimeMusic\nAllow: /gp/dmusic/promotions/AmazonMusicUnlimited\nDisallow: /gp/offer-listing/\nDisallow: /b?*node=9052533011\nDisallow: /lm/R1XIHQVKXSKBNJ\nDisallow: /lm/R3HQ5WJSZK6QSO\nDisallow: /surprise/\nDisallow: /local/ajax/\nDisallow: /gp/socialmedia/giveaways\nDisallow: /gp/b2b-rd\nDisallow: /gp/aw/so.html\nDisallow: /reviews/iframe\nDisallow: /gp/switch-language\nDisallow: /ga/p/\nDisallow: /gp/profile/\nDisallow: /giveaway/host/setup/\nDisallow: /ss/customer-reviews/lighthouse/\nDisallow: /ospublishing/story/*\nDisallow: /gp/aw/ol/\nDisallow: /gp/promotion/\nDisallow: /hz/leaderboard/top-reviewers/\nDisallow: /creatorhub\nDisallow: /creatorhub/*\nDisallow: /slp/s$\nDisallow: /-/\nAllow: /-/es/\nAllow: /-/en$\nAllow: /-/zh_TW/\nAllow: /-/zh_TW$\nAllow: /-/he/\nAllow: /-/he$\nDisallow: /hz/help/contact/*/message/$\nDisallow: /gp/aw/shoppingAids/\nDisallow: /rss/people/*/reviews\nDisallow: /gp/pdp/rss/*/reviews\nDisallow: /gp/cdp/member-reviews/\nDisallow: /gp/aw/cr/\nAllow: /gp/offer-listing/B000\nAllow: /gp/offer-listing/9000\nDisallow: /gp/aag\nAllow: /gp/aag/main?*seller=ABVFEJU8LS620\nDisallow: /gp/pdp/profile/\nDisallow: /gp/help/customer/express/c2c/\nDisallow: /slp/*/b$\nDisallow: /hz/contact-us/ajax/initiate-trusted-contact/\nDisallow: /hp/video/\nDisallow: /gp/video/api\nDisallow: /gp/video/settings\nDisallow: /gp/video/library\nDisallow: /gp/video/watchlist\nDisallow: /gp/video/mystuff\nDisallow: /gp/video/profiles\nDisallow: /gp/video/search\nDisallow: /gp/video/auth\nDisallow: /gp/video/salp/i\nDisallow: /icdp/\nDisallow: /shoptheshow/*?cat=\nDisallow: /shoptheshow/*&cat=\n\nUser-agent: EtaoSpider\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\n\nUser-agent: CCBot\nDisallow: /\n\nUser-Agent: PerplexityBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: GoogleAgent-Mariner\nDisallow: /\n\nUser-agent: GoogleAgent-Shopping\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: Devin\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: AI2Bot\nDisallow: /\n\nUser-agent: Gemini-Deep-Research\nDisallow: /\n\nUser-agent: PanguBot\nDisallow: /\n\nUser-agent: MistralAI-User\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: Sidetrade indexer bot\nDisallow: /\n\nUser-agent: Google-NotebookLM\nDisallow: /\n\nUser-agent: DuckAssistBot\nDisallow: /\n\nUser-agent: Copilot\nDisallow: /\n\nUser-agent: Ai2Bot-Dolma\nDisallow: /\n\nUser-agent: Google-CloudVertexBot\nDisallow: /\n\nUser-agent: meta-externalfetcher\nDisallow: /\n\nUser-agent: iaskspider\nDisallow: /\n\nUser-agent: wpbot\nDisallow: /\n\nUser-agent: aiHitBot\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: FriendlyCrawler\nDisallow: /\n\nUser-agent: img2dataset\nDisallow: /\n\nUser-agent: VelenPublicWebCrawler\nDisallow: /\n\nUser-agent: YouBot\nDisallow: /\n\nUser-agent: Brightbot\nDisallow: /\n\nUser-agent: Claude-User\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Claude-SearchBot\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: ISSCyberRiskCrawler\nDisallow: /\n\nUser-agent: CopilotNative\nDisallow: /\n\nUser-agent: CopilotSapphire\nDisallow: /\n\nUser-agent: imgproxy\nDisallow: /\n\nUser-agent: SemrushBot-SWA\nDisallow: /\n\nUser-agent: ChatGPT-User\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\n\nUser-agent: Timpibot\nDisallow: /\n\nUser-agent: Thinkbot\nDisallow: /\n\nUser-agent: AddSearchBot\nDisallow: /\n\nUser-agent: AI2Bot-DeepResearchEval\nDisallow: /\n\nUser-agent: Andibot\nDisallow: /\n\nUser-agent: Anomura\nDisallow: /\n\nUser-agent: atlassian-bot\nDisallow: /\n\nUser-agent: BuddyBot\nDisallow: /\n\nUser-agent: Channel3Bot\nDisallow: /\n\nUser-agent: ChatGLM-Spider\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: Cloudflare-AutoRAG\nDisallow: /\n\nUser-agent: CloudflareBrowserRenderingCrawler\nDisallow: /\n\nUser-agent: cohere-training-data-crawler\nDisallow: /\n\nUser-agent: Cotoyogi\nDisallow: /\n\nUser-agent: Crawl4AI\nDisallow: /\n\nUser-agent: Datenbank Crawler\nDisallow: /\n\nUser-agent: DeepSeekBot\nDisallow: /\n\nUser-agent: Echobot Bot\nDisallow: /\n\nUser-agent: EchoboxBot\nDisallow: /\n\nUser-agent: factset_spyderbot\nDisallow: /\n\nUser-agent: Google-Firebase\nDisallow: /\n\nUser-agent: GoogleOther\nDisallow: /\n\nUser-agent: Grok-DeepSearch\nDisallow: /\n\nUser-agent: GrokBot\nDisallow: /\n\nUser-agent: iAskBot\nDisallow: /\n\nUser-agent: IbouBot\nDisallow: /\n\nUser-agent: ICC-Crawler\nDisallow: /\n\nUser-agent: ImagesiftBot\nDisallow: /\n\nUser-agent: imageSpider\nDisallow: /\n\nUser-agent: Kangaroo Bot\nDisallow: /\n\nUser-agent: KlaviyoAIBot\nDisallow: /\n\nUser-agent: KunatoCrawler\nDisallow: /\n\nUser-agent: laion-huggingface-processor\nDisallow: /\n\nUser-agent: Linguee Bot\nDisallow: /\n\nUser-agent: LinkupBot\nDisallow: /\n\nUser-agent: Manus-User\nDisallow: /\n\nUser-agent: meta-webindexer\nDisallow: /\n\nUser-agent: netEstate Imprint Crawler\nDisallow: /\n\nUser-agent: Panscient\nDisallow: /\n\nUser-agent: panscient.com\nDisallow: /\n\nUser-agent: PhindBot\nDisallow: /\n\nUser-agent: Poggio-Citations\nDisallow: /\n\nUser-agent: QualifiedBot\nDisallow: /\n\nUser-agent: SBIntuitionsBot\nDisallow: /\n\nUser-agent: ShapBot\nDisallow: /\n\nUser-agent: TavilyBot\nDisallow: /\n\nUser-agent: TerraCotta\nDisallow: /\n\nUser-agent: TwinAgent\nDisallow: /\n\nUser-agent: webzio-extended\nDisallow: /\n\nUser-agent: WRTNBot\nDisallow: /\n\nUser-agent: xAI-Grok\nDisallow: /\n\nUser-agent: YaK\nDisallow: /\n\nUser-agent: ZanistaBot\nDisallow: /","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","EtaoSpider","GPTBot","CCBot","PerplexityBot","Google-Extended","GoogleAgent-Mariner","GoogleAgent-Shopping","ClaudeBot","meta-externalagent","Bytespider","Scrapy","PetalBot","Devin","omgili","AI2Bot","Gemini-Deep-Research","PanguBot","MistralAI-User","Diffbot","Sidetrade indexer bot","Google-NotebookLM","DuckAssistBot","Copilot","Ai2Bot-Dolma","Google-CloudVertexBot","meta-externalfetcher","iaskspider","wpbot","aiHitBot","cohere-ai","FriendlyCrawler","img2dataset","VelenPublicWebCrawler","YouBot","Brightbot","Claude-User","Claude-SearchBot","Perplexity-User","ISSCyberRiskCrawler","CopilotNative","CopilotSapphire","imgproxy","SemrushBot-SWA","ChatGPT-User","OAI-SearchBot","Timpibot","Thinkbot","AddSearchBot","AI2Bot-DeepResearchEval","Andibot","Anomura","atlassian-bot","BuddyBot","Channel3Bot","ChatGLM-Spider","Claude-Web","Cloudflare-AutoRAG","CloudflareBrowserRenderingCrawler","cohere-training-data-crawler","Cotoyogi","Crawl4AI","Datenbank Crawler","DeepSeekBot","Echobot Bot","EchoboxBot","factset_spyderbot","Google-Firebase","GoogleOther","Grok-DeepSearch","GrokBot","iAskBot","IbouBot","ICC-Crawler","ImagesiftBot","imageSpider","Kangaroo Bot","KlaviyoAIBot","KunatoCrawler","laion-huggingface-processor","Linguee Bot","LinkupBot","Manus-User","meta-webindexer","netEstate Imprint Crawler","Panscient","panscient.com","PhindBot","Poggio-Citations","QualifiedBot","SBIntuitionsBot","ShapBot","TavilyBot","TerraCotta","TwinAgent","webzio-extended","WRTNBot","xAI-Grok","YaK","ZanistaBot"],"lines":439},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":144},"any":{"allowed":false,"line":144},"search":{"allowed":false,"line":144},"cart":{"allowed":false,"line":144},"checkout":{"allowed":false,"line":144},"account":{"allowed":false,"line":144}},"disallows":1,"lines":[143,144],"groupRanges":[[143,144]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":163},"any":{"allowed":false,"line":163},"search":{"allowed":false,"line":163},"cart":{"allowed":false,"line":163},"checkout":{"allowed":false,"line":163},"account":{"allowed":false,"line":163}},"disallows":2,"lines":[162,163,249,250],"groupRanges":[[162,163],[249,250]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":154},"any":{"allowed":false,"line":154},"search":{"allowed":false,"line":154},"cart":{"allowed":false,"line":154},"checkout":{"allowed":false,"line":154},"account":{"allowed":false,"line":154}},"disallows":1,"lines":[153,154],"groupRanges":[[153,154]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":148},"any":{"allowed":false,"line":148},"search":{"allowed":false,"line":148},"cart":{"allowed":false,"line":148},"checkout":{"allowed":false,"line":148},"account":{"allowed":false,"line":148}},"disallows":1,"lines":[147,148],"groupRanges":[[147,148]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":166},"any":{"allowed":false,"line":166},"search":{"allowed":false,"line":166},"cart":{"allowed":false,"line":166},"checkout":{"allowed":false,"line":166},"account":{"allowed":false,"line":166}},"disallows":1,"lines":[165,166],"groupRanges":[[165,166]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":169},"any":{"allowed":false,"line":169},"search":{"allowed":false,"line":169},"cart":{"allowed":false,"line":169},"checkout":{"allowed":false,"line":169},"account":{"allowed":false,"line":169}},"disallows":1,"lines":[168,169],"groupRanges":[[168,169]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"AI2Bot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":184},"any":{"allowed":false,"line":184},"search":{"allowed":false,"line":184},"cart":{"allowed":false,"line":184},"checkout":{"allowed":false,"line":184},"account":{"allowed":false,"line":184}},"disallows":1,"lines":[183,184],"groupRanges":[[183,184]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":277},"any":{"allowed":false,"line":277},"search":{"allowed":false,"line":277},"cart":{"allowed":false,"line":277},"checkout":{"allowed":false,"line":277},"account":{"allowed":false,"line":277}},"disallows":1,"lines":[276,277],"groupRanges":[[276,277]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":253},"any":{"allowed":false,"line":253},"search":{"allowed":false,"line":253},"cart":{"allowed":false,"line":253},"checkout":{"allowed":false,"line":253},"account":{"allowed":false,"line":253}},"disallows":1,"lines":[252,253],"groupRanges":[[252,253]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":151},"any":{"allowed":false,"line":151},"search":{"allowed":false,"line":151},"cart":{"allowed":false,"line":151},"checkout":{"allowed":false,"line":151},"account":{"allowed":false,"line":151}},"disallows":1,"lines":[150,151],"groupRanges":[[150,151]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":205},"any":{"allowed":false,"line":205},"search":{"allowed":false,"line":205},"cart":{"allowed":false,"line":205},"checkout":{"allowed":false,"line":205},"account":{"allowed":false,"line":205}},"disallows":1,"lines":[204,205],"groupRanges":[[204,205]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":391},"any":{"allowed":false,"line":391},"search":{"allowed":false,"line":391},"cart":{"allowed":false,"line":391},"checkout":{"allowed":false,"line":391},"account":{"allowed":false,"line":391}},"disallows":1,"lines":[390,391],"groupRanges":[[390,391]]},{"token":"ChatGPT-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":274},"any":{"allowed":false,"line":274},"search":{"allowed":false,"line":274},"cart":{"allowed":false,"line":274},"checkout":{"allowed":false,"line":274},"account":{"allowed":false,"line":274}},"disallows":1,"lines":[273,274],"groupRanges":[[273,274]]},{"token":"Claude-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":247},"any":{"allowed":false,"line":247},"search":{"allowed":false,"line":247},"cart":{"allowed":false,"line":247},"checkout":{"allowed":false,"line":247},"account":{"allowed":false,"line":247}},"disallows":1,"lines":[246,247],"groupRanges":[[246,247]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":256},"any":{"allowed":false,"line":256},"search":{"allowed":false,"line":256},"cart":{"allowed":false,"line":256},"checkout":{"allowed":false,"line":256},"account":{"allowed":false,"line":256}},"disallows":1,"lines":[255,256],"groupRanges":[[255,256]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":217},"any":{"allowed":false,"line":217},"search":{"allowed":false,"line":217},"cart":{"allowed":false,"line":217},"checkout":{"allowed":false,"line":217},"account":{"allowed":false,"line":217}},"disallows":1,"lines":[216,217],"groupRanges":[[216,217]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"MistralAI-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":193},"any":{"allowed":false,"line":193},"search":{"allowed":false,"line":193},"cart":{"allowed":false,"line":193},"checkout":{"allowed":false,"line":193},"account":{"allowed":false,"line":193}},"disallows":1,"lines":[192,193],"groupRanges":[[192,193]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":120,"lines":[1,138],"groupRanges":[[1,138]]}],"lanes":{"training":{"level":"mixed","none":7,"full":2,"restricted":0,"n":9},"search":{"level":"mixed","none":5,"full":3,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":5,"full":2,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":["Amazon CloudFront"],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.amazon.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.amazon.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.amazon.com/.well-known/tdmrep.json"},"meta":{"title":"Amazon.com. Spend less. Smile more.","robotsMeta":[],"noai":[],"noindex":false,"jsonLdTypes":[],"platform":null,"textRatio":0.008,"llmsLink":false},"x_robots_tag":null,"tos":{"url":"https://www.amazon.com/gp/help/customer/display.html?nodeId=508088","source":"curated","http":403,"hash":null,"chars":0,"thin":false,"clauses":[],"door":{"door":"challenged","http":403,"detail":"Generic bot challenge page"}},"paid":null,"program":null,"statements":[{"domain":"amazon.com","company":"Amazon","effect":"blocks_agents","summary":"told Meta's shopping agent that access by an unauthorized AI agent violates its Conditions of Use","source":{"publisher":"The Next Web","date":"2026-09-21","url":"https://thenextweb.com/news/amazon-blocks-muse-perplexity-amended-complaint"}}],"claim":{"domain":"amazon.com","company":"Amazon","claim":"Amazon is blocking Grok and other shopping agents.","grade":"partly","grade_label":"Partly supported","statement":"For other shopping agents, there is a public record: in September 2026 Amazon showed Meta's Muse agent a notice that \"continued access by an unauthorized AI agent violates Amazon's Conditions of Use\", and it is in litigation with Perplexity over the Comet browser agent. For Grok specifically, the only evidence found is an X post; Amazon did not respond to a request for comment. Amazon's robots.txt does disallow Grok-style tokens (GrokBot, xAI-Grok, Grok-DeepSearch), but we found no xAI documentation of any such crawler.","sources":[{"publisher":"The Next Web","date":"2026-09-21","url":"https://thenextweb.com/news/amazon-blocks-muse-perplexity-amended-complaint"},{"publisher":"Cooley","date":"2026-08-06","url":"https://www.cooley.com/news/insight/2026/2026-08-06-ninth-circuit-rules-on-ai-agent-access-to-third-party-websites-under-cfaa"},{"publisher":"Moneywise via Yahoo Finance","date":"2026-10-15","url":"https://finance.yahoo.com/small-business/articles/elon-musk-wants-start-amazon-220000094.html"},{"publisher":"Amazon robots.txt","date":"2026-10-19","url":"https://www.amazon.com/robots.txt"}]},"reports":[],"history":{"points":47,"first":"2023-01-01","archive_points":46,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":3},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":10},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":11},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":16},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":17}]},"events":[{"domain":"amazon.com","name":"Amazon","category":"retail","date":"2026-06","precision":"month","source":"archive","provisional":false,"between":["2026-05-01","2026-06-07"],"summary":"Now blocked: Meta-WebIndexer.","dir":"up","up":["Meta-WebIndexer"],"down":[],"major":false,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20260501162735/https://www.amazon.com/robots.txt","https://web.archive.org/web/20260607202731/https://www.amazon.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Timpibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Thinkbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AddSearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AI2Bot-DeepResearchEval"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Andibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Anomura"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: atlassian-bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: BuddyBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Channel3Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ChatGLM-Spider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Cloudflare-AutoRAG"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CloudflareBrowserRenderingCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-training-data-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Cotoyogi"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Crawl4AI"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Datenbank Crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DeepSeekBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Echobot Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: EchoboxBot"},{"t":"gap","s":"… 103 more lines"}]},{"domain":"amazon.com","name":"Amazon","category":"retail","date":"2025-12","precision":"month","source":"archive","provisional":false,"between":["2025-10-01","2025-12-01"],"summary":"Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.","dir":"up","up":["OAI-SearchBot","Claude-SearchBot","ChatGPT-User","Claude-User","Perplexity-User","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20251001000703/https://www.amazon.com/robots.txt","https://web.archive.org/web/20251201130651/https://www.amazon.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /gp/video/salp/i"},{"t":"add","s":"Disallow: /icdp/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Copilot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Ai2Bot-Dolma"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-CloudVertexBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: meta-externalfetcher"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: iaskspider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: wpbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: aiHitBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FriendlyCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: img2dataset"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: VelenPublicWebCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: YouBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Brightbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-SearchBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ISSCyberRiskCrawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"gap","s":"… 17 more lines"}]},{"domain":"amazon.com","name":"Amazon","category":"retail","date":"2025-09","precision":"month","source":"archive","provisional":false,"between":["2025-06-01","2025-09-03"],"summary":"Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.","dir":"up","up":["ClaudeBot","Google-Extended","Meta-ExternalAgent","Bytespider","AI2Bot","DuckAssistBot","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250601000120/https://www.amazon.com/robots.txt","https://web.archive.org/web/20250903165358/https://www.amazon.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GoogleAgent-Mariner"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GoogleAgent-Shopping"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Scrapy"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PetalBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Devin"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: AI2Bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Gemini-Deep-Research"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PanguBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: MistralAI-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Sidetrade indexer bot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-NotebookLM"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"add","s":"Disallow: /"}]},{"domain":"amazon.com","name":"Amazon","category":"retail","date":"2024-12","precision":"month","source":"archive","provisional":false,"between":["2024-11-01","2024-12-01"],"summary":"Now blocked: PerplexityBot.","dir":"up","up":["PerplexityBot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241101004322/https://www.amazon.com/robots.txt","https://web.archive.org/web/20241201001940/https://www.amazon.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /local/ajax/"},{"t":"del","s":"Disallow: */B00M3E1NYI"},{"t":"del","s":"Disallow: */B00M3E1Q5Y"},{"t":"del","s":"Disallow: */B00M3E1TOM"},{"t":"del","s":"Disallow: */B00M3E1WYO"},{"t":"del","s":"Disallow: */B00M3E204K"},{"t":"del","s":"Disallow: */B00M3E236A"},{"t":"del","s":"Disallow: */B00M3E260I"},{"t":"del","s":"Disallow: */B00M3E28WO"},{"t":"del","s":"Disallow: */B00M3E2BC6"},{"t":"del","s":"Disallow: */B00M3E2DPQ"},{"t":"del","s":"Disallow: */B00M3E2GU8"},{"t":"del","s":"Disallow: */B00M3E2J14"},{"t":"del","s":"Disallow: */B00M3E2LOE"},{"t":"del","s":"Disallow: */B00M3E1HJY"},{"t":"ctx","s":"Disallow: /gp/socialmedia/giveaways"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /gp/aw/so.html"},{"t":"del","s":"Disallow: /gp/rentallist"},{"t":"del","s":"Disallow: /gp/video/dvd-rental/settings"},{"t":"del","s":"Disallow: /gp/rl/settings"},{"t":"del","s":"Disallow: /gp/video/settings"},{"t":"del","s":"Disallow: /gp/video/library"},{"t":"del","s":"Disallow: /gp/video/watchlist"},{"t":"ctx","s":"Disallow: /reviews/iframe"},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /-/es/"},{"t":"add","s":"Allow: /-/en$"},{"t":"ctx","s":"Allow: /-/zh_TW/"},{"t":"add","s":"Allow: /-/zh_TW$"},{"t":"ctx","s":"Allow: /-/he/"},{"t":"add","s":"Allow: /-/he$"},{"t":"ctx","s":"Disallow: /hz/help/contact/*/message/$"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /gp/aw/cr/"},{"t":"del","s":"Disallow: */sim/B001132UEE"},{"t":"ctx","s":"Allow: /gp/offer-listing/B000"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /hz/contact-us/ajax/initiate-trusted-contact/"},{"t":"add","s":"Disallow: /hp/video/"},{"t":"ctx","s":"Disallow: /gp/video/api"},{"t":"del","s":"Disallow: /hp/video/api"},{"t":"add","s":"Disallow: /gp/video/settings"},{"t":"add","s":"Disallow: /gp/video/library"},{"t":"add","s":"Disallow: /gp/video/watchlist"},{"t":"ctx","s":"Disallow: /gp/video/mystuff"},{"t":"del","s":"Disallow: /hp/video/mystuff"},{"t":"ctx","s":"Disallow: /gp/video/profiles"},{"t":"del","s":"Disallow: /hp/video/profiles"},{"t":"add","s":"Disallow: /gp/video/search"},{"t":"add","s":"Disallow: /gp/video/auth"},{"t":"add","s":"Disallow: /gp/video/salp/i"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":""},{"t":"ctx","s":"User-agent: CCBot"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-Agent: PerplexityBot"},{"t":"gap","s":"… 1 more lines"}]},{"domain":"amazon.com","name":"Amazon","category":"retail","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-10-01"],"summary":"Now blocked: GPTBot and CCBot.","dir":"up","up":["GPTBot","CCBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801000123/https://www.amazon.com/robots.txt","https://web.archive.org/web/20231001000733/https://www.amazon.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"}]}]}