All sites / Retail and marketplaces
Temu
temu.comAllows AI
Its robots.txt (the public rules file for bots) mentions GPTBot, ClaudeBot, Google-Extended, Bytespider and 5 more, but still lets all 12 major AI bots into the site.
Last checked 19 Oct 2026. Rules file read today. Everything we checked · History
Bots that train AI Some blocked
1 of 9 blocked
Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.
Bots for AI search Allowed
0 of 8 blocked
Blocking these usually means the site stops showing up as a source in AI answers.
AI assistants Allowed
0 of 7 blocked
5 of these 7 companies say their assistants may ignore robots.txt.
Bot by bot
| AI bot | Allowed? | Where the rule is | Company says it follows robots.txt? |
|---|---|---|---|
| Bots that train AI · Some blocked | |||
| GPTBotOpenAI | Allowed | Named, line 1 | Says yes |
| ClaudeBotAnthropic | Allowed | Named, line 1 | Says yes |
| Google-ExtendedGoogle · opt-out name, not a bot | Allowed | Named, line 1 | Says yes |
| Applebot-ExtendedApple · opt-out name, not a bot | Allowed | Not named; the catch-all rule applies | Says yes |
| CCBotCommon Crawl | Allowed | Not named; the catch-all rule applies | Says yes |
| Meta-ExternalAgentMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| BytespiderByteDance | Blocked | Named, line 259 | No statement found |
| MistralAI-TrainingMistral AI | Allowed | Not named; the catch-all rule applies | Says yes |
| AI2BotAllen Institute for AI | Allowed | Not named; the catch-all rule applies | Does not say |
| anthropic-aiAnthropic (legacy) · old name | Allowed | Not named; the catch-all rule applies | Unknown |
| Bots for AI search · Allowed | |||
| OAI-SearchBotOpenAI | Allowed | Named, line 1 | Says yes |
| Claude-SearchBotAnthropic | Allowed | Named, line 1 | Says yes |
| PerplexityBotPerplexity | Allowed | Named, line 1 | Suggests yes |
| AmazonbotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| Amzn-SearchBotAmazon | Allowed | Not named; the catch-all rule applies | Says yes |
| DuckAssistBotDuckDuckGo | Allowed | Not named; the catch-all rule applies | Says yes |
| MistralAI-IndexMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| Meta-WebIndexerMeta | Allowed | Not named; the catch-all rule applies | Suggests yes |
| AI assistants · Allowed | |||
| ChatGPT-UserOpenAI | Allowed | Named, line 1 | Says not always |
| Claude-UserAnthropic | Allowed | Named, line 1 | Says yes |
| Perplexity-UserPerplexity | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Google-AgentGoogle | Allowed | Not named; the catch-all rule applies | Says it mostly ignores it |
| Meta-ExternalFetcherMeta | Allowed | Not named; the catch-all rule applies | Says not always |
| Amzn-UserAmazon | Allowed | Not named; the catch-all rule applies | Says not always |
| MistralAI-UserMistral AI | Allowed | Not named; the catch-all rule applies | Suggests yes |
| For comparison | |||
| GooglebotRegular search engine, for comparison | Allowed | Named | |
| BingbotRegular search engine, for comparison | Allowed | Named | |
Other AI names in this file
Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.
| Applebot | Has rules | line 118 | Documented by its operator |
| FacebookBot | Has rules | line 121 | Not in our verified list |
We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.
The rules file (robots.txt)
User-agent: GooglebotUser-agent: Googlebot-ImageUser-agent: BingbotUser-agent: Google-ExtendedUser-agent: GPTBotUser-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: PerplexityBotUser-agent: ClaudeBotUser-agent: Claude-UserUser-agent: Claude-SearchBotDisAllow: /*oauth_code=DisAllow: /*oauth_token=DisAllow: /bgp_user_profile.htmlDisAllow: /bgp_personal.htmlDisAllow: /bgp_collect.htmlDisAllow: /bgp_footprint.htmlDisAllow: /bgst_order_success.htmlDisAllow: /bgst_payment_detail.htmlDisAllow: /bgst_express.htmlDisAllow: /personal.htmlDisAllow: /collect.htmlDisAllow: /shopping_cart.htmlDisAllow: /bgp_similar_rec.htmlDisAllow: /bgcp_shipping_add_items.htmlDisAllow: /bgcp_coupon.htmlDisAllow: /bgcp_cart_coupon.htmlDisAllow: /bgcp_coupon_add_items.htmlDisAllow: /bgt_order_checkout.htmlDisAllow: /bgt_bank_foreign_card_info.htmlDisAllow: /bgt_3ds.htmlDisAllow: /bgt_orders.htmlDisAllow: /bgt_order_detail.htmlDisAllow: /bgt_card_list_management.htmlDisAllow: /bgt_bind_card.htmlDisAllow: /address.htmlDisAllow: /create_address.htmlDisAllow: /bgst_order_cancelled.htmlDisAllow: /bgt_reviewed.htmlDisAllow: /bgas_return_application.htmlDisAllow: /bgas_refund_detail.htmlDisAllow: /bgas_return_history.htmlDisAllow: /bgas_refund_success.htmlDisAllow: /bgas_cancel_order.htmlDisAllow: /bgp_redirect.htmlDisAllow: /bgp_setting.htmlDisAllow: /bgp_account_security.htmlDisAllow: /bgn_verification.htmlDisAllow: /bgn_login_verification.htmlDisAllow: /inno_promotion.htmlDisAllow: /student-discount.htmlDisAllow: /moss_product_news.htmlDisAllow: /moss_freelancer_register.htmlDisAllow: /moss_assistant_bonus.htmlDisAllow: /*/*oauth_code=DisAllow: /*/*oauth_token=DisAllow: /*/bgp_user_profile.htmlDisAllow: /*/bgp_personal.htmlDisAllow: /*/bgp_collect.htmlDisAllow: /*/bgp_footprint.htmlDisAllow: /*/bgst_order_success.htmlDisAllow: /*/bgst_payment_detail.htmlDisAllow: /*/bgst_express.htmlDisAllow: /*/personal.htmlDisAllow: /*/collect.htmlDisAllow: /*/shopping_cart.htmlDisAllow: /*/bgp_similar_rec.htmlDisAllow: /*/bgcp_shipping_add_items.htmlDisAllow: /*/bgcp_coupon.htmlDisAllow: /*/bgcp_cart_coupon.htmlDisAllow: /*/bgcp_coupon_add_items.htmlDisAllow: /*/bgt_order_checkout.htmlDisAllow: /*/bgt_bank_foreign_card_info.htmlDisAllow: /*/bgt_3ds.htmlDisAllow: /*/bgt_orders.htmlDisAllow: /*/bgt_order_detail.htmlDisAllow: /*/bgt_card_list_management.htmlDisAllow: /*/bgt_bind_card.htmlDisAllow: /*/address.htmlDisAllow: /*/create_address.htmlDisAllow: /*/bgst_order_cancelled.htmlDisAllow: /*/bgt_reviewed.htmlDisAllow: /*/bgas_return_application.htmlDisAllow: /*/bgas_refund_detail.htmlDisAllow: /*/bgas_return_history.htmlDisAllow: /*/bgas_refund_success.htmlDisAllow: /*/bgas_cancel_order.htmlDisAllow: /*/bgp_redirect.htmlDisAllow: /*/bgp_setting.htmlDisAllow: /*/bgp_account_security.htmlDisAllow: /*/bgn_verification.htmlDisAllow: /*/bgn_login_verification.htmlDisAllow: /*/inno_promotion.htmlDisAllow: /*/student-discount.htmlDisAllow: /*/moss_product_news.htmlDisAllow: /*/moss_freelancer_register.htmlDisAllow: /*/moss_assistant_bonus.htmlDisAllow: /m/DisAllow: /k/Disallow: /w/Disallow: /s/Disallow: /*?*opt_level=*title=*Disallow: /*-m-*?goods_id=*DisAllow: /blog/*DisAllow: /*/blog/*DisAllow: /blog-gallery.html*DisAllow: /*/blog-gallery.html*DisAllow: */c/1418078/1583288/18350*Disallow: /*MEDICAL_CONTEXT*Disallow: /*US_FEMALE_NAME*Disallow: /*US_CENSUS_NAME*Disallow: /*tmp0611*Disallow: /be/be-fr*DisAllow: /*@context*https://schema.org/* User-agent: AdsBot-GoogleUser-agent: AdIdxBotUser-agent: ApplebotUser-agent: PinterestbotUser-agent: TwitterbotUser-agent: FacebookBotDisAllow: /*oauth_code=DisAllow: /*oauth_token=DisAllow: /bgp_user_profile.htmlDisAllow: /bgp_personal.htmlDisAllow: /bgp_collect.htmlDisAllow: /bgp_footprint.htmlDisAllow: /bgst_order_success.htmlDisAllow: /bgst_payment_detail.htmlDisAllow: /bgst_express.htmlDisAllow: /personal.htmlDisAllow: /collect.htmlDisAllow: /shopping_cart.htmlDisAllow: /bgp_similar_rec.htmlDisAllow: /bgcp_shipping_add_items.htmlDisAllow: /bgcp_coupon.htmlDisAllow: /bgcp_cart_coupon.htmlDisAllow: /bgcp_coupon_add_items.htmlDisAllow: /bgt_order_checkout.htmlDisAllow: /bgt_bank_foreign_card_info.htmlDisAllow: /bgt_3ds.htmlDisAllow: /bgt_orders.htmlDisAllow: /bgt_order_detail.htmlDisAllow: /bgt_card_list_management.htmlDisAllow: /bgt_bind_card.htmlDisAllow: /address.htmlDisAllow: /create_address.htmlDisAllow: /bgst_order_cancelled.htmlDisAllow: /bgt_reviewed.htmlDisAllow: /bgas_return_application.htmlDisAllow: /bgas_refund_detail.htmlDisAllow: /bgas_return_history.htmlDisAllow: /bgas_refund_success.htmlDisAllow: /bgas_cancel_order.htmlDisAllow: /bgp_redirect.htmlDisAllow: /bgp_setting.htmlDisAllow: /bgp_account_security.htmlDisAllow: /bgn_verification.htmlDisAllow: /bgn_login_verification.htmlDisAllow: /inno_promotion.htmlDisAllow: /student-discount.htmlDisAllow: /moss_product_news.htmlDisAllow: /moss_freelancer_register.htmlDisAllow: /moss_assistant_bonus.htmlDisAllow: /m/DisAllow: /k/Disallow: /w/Disallow: /s/Disallow: /*?*top_goods=*Disallow: /*?*opt_level=*title=*Disallow: /*?*top_mall_list=*Disallow: /*-m-*?goods_id=*Disallow: /channel/*-g-*Disallow: /*lightning-deals.html?*top_goods_low_price=*Disallow: /*-g-*html?*thumb_url=*Disallow: /*-g-*html?*top_gallery_url=*Disallow: /*-g-*html?*_oak_page_source=*DisAllow: /blog/*DisAllow: /*/blog/*DisAllow: /blog-gallery.html*DisAllow: /*/blog-gallery.html*DisAllow: */c/1418078/1583288/18350*Disallow: /*MEDICAL_CONTEXT*Disallow: /*US_FEMALE_NAME*Disallow: /*US_CENSUS_NAME*Disallow: /*tmp0611*Disallow: /be/be-fr*DisAllow: /*@context*https://schema.org/* User-agent: facebookexternalhitDisAllow: /*oauth_code=DisAllow: /*oauth_token=DisAllow: /bgp_user_profile.htmlDisAllow: /bgp_personal.htmlDisAllow: /bgp_collect.htmlDisAllow: /bgp_footprint.htmlDisAllow: /bgst_order_success.htmlDisAllow: /bgst_payment_detail.htmlDisAllow: /bgst_express.htmlDisAllow: /personal.htmlDisAllow: /collect.htmlDisAllow: /shopping_cart.htmlDisAllow: /bgp_similar_rec.htmlDisAllow: /bgcp_shipping_add_items.htmlDisAllow: /bgcp_coupon.htmlDisAllow: /bgcp_cart_coupon.htmlDisAllow: /bgcp_coupon_add_items.htmlDisAllow: /bgt_order_checkout.htmlDisAllow: /bgt_bank_foreign_card_info.htmlDisAllow: /bgt_3ds.htmlDisAllow: /bgt_orders.htmlDisAllow: /bgt_order_detail.htmlDisAllow: /bgt_card_list_management.htmlDisAllow: /bgt_bind_card.htmlDisAllow: /address.htmlDisAllow: /create_address.htmlDisAllow: /bgst_order_cancelled.htmlDisAllow: /bgt_reviewed.htmlDisAllow: /bgas_return_application.htmlDisAllow: /bgas_refund_detail.htmlDisAllow: /bgas_return_history.htmlDisAllow: /bgas_refund_success.htmlDisAllow: /bgas_cancel_order.htmlDisAllow: /bgp_redirect.htmlDisAllow: /bgp_setting.htmlDisAllow: /bgp_account_security.htmlDisAllow: /bgn_verification.htmlDisAllow: /bgn_login_verification.htmlDisAllow: /inno_promotion.htmlDisAllow: /student-discount.htmlDisAllow: /moss_product_news.htmlDisAllow: /moss_freelancer_register.htmlDisAllow: /moss_assistant_bonus.htmlDisallow: /w/Disallow: /*?*top_goods=*Disallow: /*?*opt_level=*title=*Disallow: /*?*top_mall_list=*Disallow: /*-m-*?goods_id=*Disallow: /channel/*-g-*Disallow: /*lightning-deals.html?*top_goods_low_price=*Disallow: /*-g-*html?*thumb_url=*Disallow: /*-g-*html?*top_gallery_url=*Disallow: /*-g-*html?*_oak_page_source=*DisAllow: /blog/*DisAllow: /*/blog/*DisAllow: /blog-gallery.html*DisAllow: /*/blog-gallery.html*DisAllow: */c/1418078/1583288/18350*Disallow: /*MEDICAL_CONTEXT*Disallow: /*US_FEMALE_NAME*Disallow: /*US_CENSUS_NAME*Disallow: /*tmp0611*Disallow: /be/be-fr*DisAllow: /*@context*https://schema.org/* User-agent: *Allow: /.well-known/… 68 lines not about AI bots User-agent: SlurpUser-agent: Y!J-BRU/VSIDXUser-agent: Y!J-BRWUser-agent: Y!J-ASRUser-agent: Y!J-MMP/dscvUser-agent: Y!J-WSCUser-agent: Y!J-DLCUser-agent: YJ-BRU/VSIDXUser-agent: YJ-BRWUser-agent: YJ-ASRUser-agent: YJ-MMP/dscvUser-agent: YJ-WSCUser-agent: YJ-DLCUser-agent: BytespiderDisAllow: /* Sitemap: https://www.temu.com/sitemap-index.xml… 1 line not about AI bots
Everything else we checked
| Front door | Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.temu.com/ as WalledWebBot, from a residential connection in the United States |
|---|---|
| Anti-bot software | No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up. |
| Readable by AI | Structured data types: Organization, PostalAddress, WebSite, SearchAction, EntryPoint, FAQPage, Question, Answer. Most of the page is script; an agent that does not run JavaScript sees little text.Homepage only. Useful for site owners; not part of the result. |
| Terms of Service | www.temu.com/terms-of-use.html (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context.
|
| Not found | No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags. |
History
36 versions on record since 1 Jan 2023
One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.
- Sep 2025Signals Named GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 2 more explicitly, without changing what they may reach.Found in the Internet Archive's copies: changed between 1 Aug 2025 and 1 Sep 2025. Evidence: before · after
Show exactly what changed
User-agent: Bingbot+ User-agent: Google-Extended+ User-agent: GPTBot+ User-agent: OAI-SearchBot+ User-agent: ChatGPT-User+ User-agent: PerplexityBot+ User-agent: ClaudeBot+ User-agent: Claude-User+ User-agent: Claude-SearchBotDisAllow: /*oauth_code=…Disallow: /*-m-*?goods_id=*- DisAllow: /pinduoduo-select*- DisAllow: /*/pinduoduo-select*DisAllow: /blog/*…User-agent: Applebot- User-agent: facebookexternalhitUser-agent: Pinterestbot…Disallow: /*-g-*html?*_oak_page_source=*- DisAllow: /pinduoduo-select*- DisAllow: /*/pinduoduo-select*DisAllow: /blog/*…+ User-agent: facebookexternalhit+ DisAllow: /*oauth_code=+ DisAllow: /*oauth_token=+ DisAllow: /bgp_user_profile.html+ DisAllow: /bgp_personal.html+ DisAllow: /bgp_collect.html+ DisAllow: /bgp_footprint.html+ DisAllow: /bgst_order_success.html+ DisAllow: /bgst_payment_detail.html+ DisAllow: /bgst_express.html+ DisAllow: /personal.html+ DisAllow: /collect.html+ DisAllow: /shopping_cart.html+ DisAllow: /bgp_similar_rec.html+ DisAllow: /bgcp_shipping_add_items.html+ DisAllow: /bgcp_coupon.html+ DisAllow: /bgcp_cart_coupon.html+ DisAllow: /bgcp_coupon_add_items.html+ DisAllow: /bgt_order_checkout.html+ DisAllow: /bgt_bank_foreign_card_info.html+ DisAllow: /bgt_3ds.html+ DisAllow: /bgt_orders.html+ DisAllow: /bgt_order_detail.html+ DisAllow: /bgt_card_list_management.html+ DisAllow: /bgt_bind_card.html+ DisAllow: /address.html+ DisAllow: /create_address.html+ DisAllow: /bgst_order_cancelled.html+ DisAllow: /bgt_reviewed.html+ DisAllow: /bgas_return_application.html+ DisAllow: /bgas_refund_detail.html+ DisAllow: /bgas_return_history.html+ DisAllow: /bgas_refund_success.html+ DisAllow: /bgas_cancel_order.html… 33 more lines - Nov 2023More blocking Now blocked: Bytespider.Found in the Internet Archive's copies: changed between 1 Oct 2023 and 1 Nov 2023. Evidence: before · after
Show exactly what changed
User-agent: Googlebot+ User-agent: Googlebot-ImageUser-agent: Bingbot+ DisAllow: /*oauth_code=+ DisAllow: /*oauth_token=+ DisAllow: /bgp_user_profile.html+ DisAllow: /bgp_personal.html+ DisAllow: /bgp_collect.html+ DisAllow: /bgp_footprint.html+ DisAllow: /bgst_order_success.html+ DisAllow: /bgst_payment_detail.html+ DisAllow: /bgst_express.html+ DisAllow: /personal.html+ DisAllow: /collect.html+ DisAllow: /shopping_cart.html+ DisAllow: /bgp_similar_rec.html+ DisAllow: /bgcp_shipping_add_items.html+ DisAllow: /bgcp_coupon.html+ DisAllow: /bgcp_cart_coupon.html+ DisAllow: /bgcp_coupon_add_items.html+ DisAllow: /bgt_order_checkout.html+ DisAllow: /bgt_bank_foreign_card_info.html+ DisAllow: /bgt_3ds.html+ DisAllow: /bgt_orders.html+ DisAllow: /bgt_order_detail.html+ DisAllow: /bgt_card_list_management.html+ DisAllow: /bgt_bind_card.html+ DisAllow: /address.html+ DisAllow: /create_address.html+ DisAllow: /bgst_order_cancelled.html+ DisAllow: /bgt_reviewed.html+ DisAllow: /bgas_return_application.html+ DisAllow: /bgas_refund_detail.html+ DisAllow: /bgas_return_history.html+ DisAllow: /bgas_refund_success.html+ DisAllow: /bgas_cancel_order.html+ DisAllow: /bgp_redirect.html+ DisAllow: /bgp_setting.html+ DisAllow: /bgp_account_security.html+ DisAllow: /bgn_verification.html+ DisAllow: /bgn_login_verification.html+ DisAllow: /inno_promotion.html+ DisAllow: /student-discount.html+ DisAllow: /moss_product_news.html+ DisAllow: /moss_freelancer_register.html+ DisAllow: /moss_assistant_bonus.html+ DisAllow: /*/*oauth_code=+ DisAllow: /*/*oauth_token=+ DisAllow: /*/bgp_user_profile.html+ DisAllow: /*/bgp_personal.html+ DisAllow: /*/bgp_collect.html+ DisAllow: /*/bgp_footprint.html+ DisAllow: /*/bgst_order_success.html+ DisAllow: /*/bgst_payment_detail.html+ DisAllow: /*/bgst_express.html+ DisAllow: /*/personal.html+ DisAllow: /*/collect.html+ DisAllow: /*/shopping_cart.html+ DisAllow: /*/bgp_similar_rec.html+ DisAllow: /*/bgcp_shipping_add_items.html… 152 more lines
Reports from readers
No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.