The Walled Web Newsletter

AI bots we track / AI training

Who blocks Bytespider?

Bytespider is ByteDance's bot that collects pages to train AI. Blocking it keeps a site out of future AI training. It does not stop AI from reading the site to answer questions. ByteDance has no operator documentation we could find: No operator documentation found. Widely seen in server logs; independent reports dispute that it follows robots.txt.

40of 159
sites block it from the whole site
28of 40
of those name it in robots.txt
0of 159
block it from part of the site
2of 159
name it and let it in

Out of the 159 tracked sites whose robots.txt we could read. Test any other name.

Share of sites blocking Bytespider, Jan 2023 to now

The same 143 sites in every month. Before this bot existed, sites that block every bot they don't name already kept it out.

Bytespider 25%
0%10%20%30%2023202420252026Bytespider 25%0%10%20%30%'23'24'25'26
Table view
MonthBytespiderSites with a file
Jan 20234%143
Feb 20235%143
Mar 20234%143
Apr 20234%143
May 20235%143
Jun 20234%143
Jul 20234%143
Aug 20235%143
Sep 20235%143
Oct 20236%143
Nov 20236%143
Dec 20236%143
Jan 20247%143
Feb 20246%143
Mar 20248%143
Apr 20249%143
May 202411%143
Jun 202414%143
Jul 202415%143
Aug 202417%143
Sep 202416%143
Oct 202417%143
Nov 202417%143
Dec 202418%143
Jan 202517%143
Feb 202517%143
Mar 202518%143
Apr 202518%143
May 202520%143
Jun 202520%143
Jul 202520%143
Aug 202520%143
Sep 202521%143
Oct 202522%143
Nov 202522%143
Dec 202522%143
Jan 202622%143
Feb 202622%143
Mar 202622%143
Apr 202623%143
May 202623%143
Jun 202623%143
Jul 202623%143
Aug 202624%143
Sep 202624%143
Oct 202625%143

Blocked by name (28)

AmazoneBayTemuAliExpressTripadvisorGlassdoorThe New York TimesThe Washington PostThe GuardianBBCCNNBloombergFinancial TimesThe AtlanticThe New YorkerNPRUSA TodayAxiosPoliticoThe VergeWIREDThe EconomistLos Angeles TimesMediumLinkedInTikTokQuoraGitHub

Blocked by the general rules for all crawlers (12)

InstacartYelpNextdoorThe Wall Street JournalReutersRedditXFacebookInstagramPinterestThreadsIMDb

Blocked from part of the site (0)

None.

Named and allowed (2)

Best BuyIndeed

Recent changes

  • Oct 2026
    AliExpress Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.
    More blocking
  • Oct 2026
    GitHub Now blocked: Bytespider.
    More blocking
  • May to Sep 2026
    IMDb Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.
    More blocking
  • Mar to Jul 2026
    LinkedIn Now blocked: Bytespider. Now allowed: Claude-SearchBot.
    Mixed
  • Feb 2026
    Reddit Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    More blocking
  • Oct 2025
    eBay Now blocked: Bytespider.
    More blocking
  • Jun to Sep 2025
    Amazon Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.
    More blocking
  • Sep 2025
    Craigslist Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.
    Less blocking
  • Sep 2025
    Politico Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.
    More blocking
  • Jul to Sep 2025
    Reddit Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.
    Less blocking
  • Aug 2025
    Bloomberg Now blocked: Meta-ExternalAgent, Bytespider and Amazonbot.
    More blocking
  • Aug 2025
    Los Angeles Times Now blocked: ClaudeBot and Bytespider.
    More blocking