Tumblr

tumblr.com

Blocks some

Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 6 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 3 more can reach most of it.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Some blocked

5 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Some blocked

1 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Some blocked
GPTBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
ClaudeBotAnthropic	Blocked	Named, line 148	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Blocked	Named, line 128	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Blocked	Named, line 160	Says yes
CCBotCommon Crawl	Blocked	Named, line 120	Says yes
Meta-ExternalAgentMeta	Blocked	Named, line 168	Suggests yes
BytespiderByteDance	Allowed	Not named; the catch-all rule applies	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Allowed	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Blocked	Named, line 152	Unknown
Bots for AI search · Some blocked
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
PerplexityBotPerplexity	Allowed	Not named; the catch-all rule applies	Suggests yes
AmazonbotAmazon	Blocked	Named, line 144	Says yes
Amzn-SearchBotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Allowed	Not named; the catch-all rule applies	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI	Allowed	Not named; the catch-all rule applies	Says not always
Claude-UserAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Perplexity-UserPerplexity	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Allowed	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Named
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

FacebookBot	Blocked	line 132	Not in our verified list
omgili	Blocked	line 136	Not in our verified list
omgilibot	Blocked	line 140	Not in our verified list
ImagesiftBot	Blocked	line 156	Not in our verified list
YouBot	Blocked	line 172	Not in our verified list

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 174 lines · 24 lines about AI bots Source

Sitemap: https://www.tumblr.com/sitemap.xml # Google Search Engine RobotUser-agent: GooglebotDisallow: /radarDisallow: /widgets/Disallow: /audio_fileDisallow: /video_fileDisallow: /dashboardDisallow: /xDisallow: /svc/accountDisallow: /dashboard/notesDisallow: /customizeDisallow: /impixuDisallow: /likedDisallow: /search/*?before=*Disallow: /search/*?language=*Disallow: /tagged/*+*Disallow: /tagged/*?before=*Disallow: /tagged/*?language=*Disallow: /openDisallow: /link_toDisallow: /sticky-ad-iframe.htmlDisallow: /privacy/consent # Yahoo! Search Engine RobotUser-agent: SlurpDisallow: /radarDisallow: /widgets/Disallow: /audio_fileDisallow: /video_fileDisallow: /dashboardDisallow: /xDisallow: /svc/accountDisallow: /dashboard/notesDisallow: /customizeDisallow: /impixuDisallow: /likedDisallow: /search/*?before=*Disallow: /search/*?language=*Disallow: /tagged/*+*Disallow: /tagged/*?before=*Disallow: /tagged/*?language=*Disallow: /openDisallow: /link_toDisallow: /sticky-ad-iframe.htmlDisallow: /privacy/consent # Yandex Search Engine RobotUser-agent: YandexDisallow: /radarDisallow: /widgets/Disallow: /audio_fileDisallow: /video_fileDisallow: /dashboardDisallow: /xDisallow: /svc/accountDisallow: /dashboard/notesDisallow: /customizeDisallow: /impixuDisallow: /likedDisallow: /search/*?before=*Disallow: /search/*?language=*Disallow: /tagged/*+*Disallow: /tagged/*?before=*Disallow: /tagged/*?language=*Disallow: /openDisallow: /link_toDisallow: /sticky-ad-iframe.htmlDisallow: /privacy/consent # Microsoft Search Engine RobotUser-agent: msnbotDisallow: /radarDisallow: /widgets/Disallow: /audio_fileDisallow: /video_fileDisallow: /dashboardDisallow: /xDisallow: /svc/accountDisallow: /dashboard/notesDisallow: /customizeDisallow: /impixuDisallow: /likedDisallow: /search/*?before=*Disallow: /search/*?language=*Disallow: /tagged/*+*Disallow: /tagged/*?before=*Disallow: /tagged/*?language=*Disallow: /openDisallow: /link_toDisallow: /sticky-ad-iframe.htmlDisallow: /privacy/consent # Every bot that might possibly read and respect this file.User-agent: *Disallow: /radarDisallow: /widgets/Disallow: /audio_fileDisallow: /video_fileDisallow: /dashboardDisallow: /xDisallow: /svc/accountDisallow: /dashboard/notesDisallow: /customizeDisallow: /impixuDisallow: /likedDisallow: /search/*?before=*Disallow: /search/*?language=*Disallow: /tagged/*+*Disallow: /tagged/*?before=*Disallow: /tagged/*?language=*Disallow: /openDisallow: /link_toDisallow: /sticky-ad-iframe.htmlDisallow: /privacy/consentCrawl-delay: 1 … 118 lines not about AI bots# Common Crawl's crawlerUser-agent: CCBotDisallow: / # SentiBot's crawlerUser-agent: sentibotDisallow: / … 4 lines not about AI bots# Google Bard's crawlerUser-agent: Google-ExtendedDisallow: / # Facebook's crawlerUser-agent: FacebookBotDisallow: / # webz.io's crawlerUser-agent: omgiliDisallow: / # webz.io's crawlerUser-agent: omgilibotDisallow: / # Amazon's crawlerUser-agent: AmazonbotDisallow: / # ClaudeBot's crawlerUser-agent: ClaudeBotDisallow: / # anthropic-ai's crawlerUser-agent: anthropic-aiDisallow: / # ImageSift's AI crawlerUser-agent: ImagesiftBotDisallow: / # Apple's AI crawlerUser-agent: Applebot-ExtendedDisallow: / # TurnitinBot crawlerUser-agent: TurnitinBotDisallow: / … 4 lines not about AI bots# Meta AI crawlerUser-agent: meta-externalagentDisallow: / # Block YouBot from entire siteUser-agent: YouBotDisallow: /

Notes about AI written in the file: "anthropic-ai's crawler" · "ImageSift's AI crawler" · "Apple's AI crawler" · "Meta AI crawler"

Everything else we checked

Front door	Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://www.tumblr.com/ as WalledWebBot, from a residential connection in the United States
Page tags	`<meta name="bingbot" content="nocache">`
Anti-bot software	No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	No schema.org structured data on the homepage. Most of the page is script; an agent that does not run JavaScript sees little text. Built on WordPress.Homepage only. Useful for site owners; not part of the result.
Terms of Service	www.tumblr.com/policy/terms-of-service (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context. You further release, to the fullest extent permitted by law, Tumblr and its employees, agents, consultants, directors, shareholders, any other person or entity that directly or indirectly controls, is under common control with, or is directly or indirectly controlled by, Tumblr (the "Tumblr Affiliat…matched "agents" · reads like a restriction …tement that the Notifying Party has a good faith belief that the material is not authorized by the copyright owner, its agent or law.matched "agent" · reads like a restriction …nterfaces that are provided by Tumblr (and only pursuant to those terms and conditions) or unless permitted by Tumblr's robots.txt file or other robot exclusion mechanisms;matched "robots" (d) scrape the Services, and particularly scrape Content (as defined below) from the Services;matched "scrape" …poration and wholly-owned subsidiary of Automattic Inc., a Delaware corporation (" Automattic "), collectively with its agents, representatives, consultants, employees, officers, and directors, " Tumblr ," " we ," or " us ").matched "agents"
Not found	No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

44 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Sep 2024
More blocking Now blocked: Meta-ExternalAgent.
Found in the Internet Archive's copies: changed between 1 Aug 2024 and 1 Sep 2024. Evidence: before · after

Show exactly what changed

+ # TurnitinBot crawler
+ User-agent: TurnitinBot
+ Disallow: /
+
+ # Meta AI crawler
+ User-agent: meta-externalagent
+ Disallow: /
+
Jul 2024
More blocking Now blocked: Applebot-Extended.
Found in the Internet Archive's copies: changed between 1 Jun 2024 and 1 Jul 2024. Evidence: before · after

Show exactly what changed

- # Bing's crawler
- User-agent: Bingbot
- Disallow: /
-
# ClaudeBot's crawler
…

+ # ImageSift's AI crawler
+ User-agent: ImagesiftBot
+ Disallow: /
+
+ # Apple's AI crawler
+ User-agent: Applebot-Extended
+ Disallow: /
+
Mar 2024
Mixed Now blocked: ClaudeBot and anthropic-ai. Now allowed: GPTBot.
Found in the Internet Archive's copies: changed between 1 Feb 2024 and 1 Mar 2024. Evidence: before · after

Show exactly what changed

- # OpenAI's crawler
- User-agent: GPTBot
- Disallow: /
-
# Common Crawl's crawler
…

+ # ClaudeBot's crawler
+ User-agent: ClaudeBot
+ Disallow: /
+
+ # anthropic-ai's crawler
+ User-agent: anthropic-ai
+ Disallow: /
+
Aug to Nov 2023
More blocking Now blocked: GPTBot, Google-Extended, CCBot and Amazonbot.
Found in the Internet Archive's copies: changed between 1 Aug 2023 and 1 Nov 2023. Evidence: before · after

Show exactly what changed

+ # OpenAI's crawler
+ User-agent: GPTBot
+ Disallow: /
+
+ # Common Crawl's crawler
+ User-agent: CCBot
+ Disallow: /
+
+ # SentiBot's crawler
+ User-agent: sentibot
+ Disallow: /
+
+ # Google Bard's crawler
+ User-agent: Google-Extended
+ Disallow: /
+
+ # Facebook's crawler
+ User-agent: FacebookBot
+ Disallow: /
+
+ # webz.io's crawler
+ User-agent: omgili
+ Disallow: /
+
+ # webz.io's crawler
+ User-agent: omgilibot
+ Disallow: /
+
+ # Amazon's crawler
+ User-agent: Amazonbot
+ Disallow: /
+
+ # Bing's crawler
+ User-agent: Bingbot
+ Disallow: /
+

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.