{"domain":"tumblr.com","name":"Tumblr","category":"social","subcategory":"network","country":"US","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 6 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 3 more can reach most of it.","computed_at":"2026-10-20T03:43:15.524Z","last_checked":"2026-10-19T13:37:22.105Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.tumblr.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.tumblr.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.tumblr.com/robots.txt","hops":[{"from":"https://tumblr.com/robots.txt","to":"https://www.tumblr.com/robots.txt","status":301}],"blob":"2892c51fb7e06e7629ccb662ecf47dec3b0bfa6a679b4e112f25f7565ac1fcb0","text":"Sitemap: https://www.tumblr.com/sitemap.xml\n\n# Google Search Engine Robot\nUser-agent: Googlebot\nDisallow: /radar\nDisallow: /widgets/\nDisallow: /audio_file\nDisallow: /video_file\nDisallow: /dashboard\nDisallow: /x\nDisallow: /svc/account\nDisallow: /dashboard/notes\nDisallow: /customize\nDisallow: /impixu\nDisallow: /liked\nDisallow: /search/*?before=*\nDisallow: /search/*?language=*\nDisallow: /tagged/*+*\nDisallow: /tagged/*?before=*\nDisallow: /tagged/*?language=*\nDisallow: /open\nDisallow: /link_to\nDisallow: /sticky-ad-iframe.html\nDisallow: /privacy/consent\n\n# Yahoo! Search Engine Robot\nUser-agent: Slurp\nDisallow: /radar\nDisallow: /widgets/\nDisallow: /audio_file\nDisallow: /video_file\nDisallow: /dashboard\nDisallow: /x\nDisallow: /svc/account\nDisallow: /dashboard/notes\nDisallow: /customize\nDisallow: /impixu\nDisallow: /liked\nDisallow: /search/*?before=*\nDisallow: /search/*?language=*\nDisallow: /tagged/*+*\nDisallow: /tagged/*?before=*\nDisallow: /tagged/*?language=*\nDisallow: /open\nDisallow: /link_to\nDisallow: /sticky-ad-iframe.html\nDisallow: /privacy/consent\n\n# Yandex Search Engine Robot\nUser-agent: Yandex\nDisallow: /radar\nDisallow: /widgets/\nDisallow: /audio_file\nDisallow: /video_file\nDisallow: /dashboard\nDisallow: /x\nDisallow: /svc/account\nDisallow: /dashboard/notes\nDisallow: /customize\nDisallow: /impixu\nDisallow: /liked\nDisallow: /search/*?before=*\nDisallow: /search/*?language=*\nDisallow: /tagged/*+*\nDisallow: /tagged/*?before=*\nDisallow: /tagged/*?language=*\nDisallow: /open\nDisallow: /link_to\nDisallow: /sticky-ad-iframe.html\nDisallow: /privacy/consent\n\n# Microsoft Search Engine Robot\nUser-agent: msnbot\nDisallow: /radar\nDisallow: /widgets/\nDisallow: /audio_file\nDisallow: /video_file\nDisallow: /dashboard\nDisallow: /x\nDisallow: /svc/account\nDisallow: /dashboard/notes\nDisallow: /customize\nDisallow: /impixu\nDisallow: /liked\nDisallow: /search/*?before=*\nDisallow: /search/*?language=*\nDisallow: /tagged/*+*\nDisallow: /tagged/*?before=*\nDisallow: /tagged/*?language=*\nDisallow: /open\nDisallow: /link_to\nDisallow: /sticky-ad-iframe.html\nDisallow: /privacy/consent\n\n# Every bot that might possibly read and respect this file.\nUser-agent: *\nDisallow: /radar\nDisallow: /widgets/\nDisallow: /audio_file\nDisallow: /video_file\nDisallow: /dashboard\nDisallow: /x\nDisallow: /svc/account\nDisallow: /dashboard/notes\nDisallow: /customize\nDisallow: /impixu\nDisallow: /liked\nDisallow: /search/*?before=*\nDisallow: /search/*?language=*\nDisallow: /tagged/*+*\nDisallow: /tagged/*?before=*\nDisallow: /tagged/*?language=*\nDisallow: /open\nDisallow: /link_to\nDisallow: /sticky-ad-iframe.html\nDisallow: /privacy/consent\nCrawl-delay: 1\n\n# Common Crawl's crawler\nUser-agent: CCBot\nDisallow: /\n\n# SentiBot's crawler\nUser-agent: sentibot\nDisallow: /\n\n# Google Bard's crawler\nUser-agent: Google-Extended\nDisallow: /\n\n# Facebook's crawler\nUser-agent: FacebookBot\nDisallow: /\n\n# webz.io's crawler\nUser-agent: omgili\nDisallow: /\n\n# webz.io's crawler\nUser-agent: omgilibot\nDisallow: /\n\n# Amazon's crawler\nUser-agent: Amazonbot\nDisallow: /\n\n# ClaudeBot's crawler\nUser-agent: ClaudeBot\nDisallow: /\n\n# anthropic-ai's crawler\nUser-agent: anthropic-ai\nDisallow: /\n\n# ImageSift's AI crawler\nUser-agent: ImagesiftBot\nDisallow: /\n\n# Apple's AI crawler\nUser-agent: Applebot-Extended\nDisallow: /\n\n# TurnitinBot crawler\nUser-agent: TurnitinBot\nDisallow: /\n\n# Meta AI crawler\nUser-agent: meta-externalagent\nDisallow: /\n\n# Block YouBot from entire site\nUser-agent: YouBot\nDisallow: /\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":151,"text":"anthropic-ai's crawler"},{"line":155,"text":"ImageSift's AI crawler"},{"line":159,"text":"Apple's AI crawler"},{"line":167,"text":"Meta AI crawler"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["Googlebot","Slurp","Yandex","msnbot","*","CCBot","sentibot","Google-Extended","FacebookBot","omgili","omgilibot","Amazonbot","ClaudeBot","anthropic-ai","ImagesiftBot","Applebot-Extended","TurnitinBot","meta-externalagent","YouBot"],"lines":174},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":149},"any":{"allowed":false,"line":149},"search":{"allowed":false,"line":149},"cart":{"allowed":false,"line":149},"checkout":{"allowed":false,"line":149},"account":{"allowed":false,"line":149}},"disallows":1,"lines":[148,149],"groupRanges":[[148,149]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":129},"any":{"allowed":false,"line":129},"search":{"allowed":false,"line":129},"cart":{"allowed":false,"line":129},"checkout":{"allowed":false,"line":129},"account":{"allowed":false,"line":129}},"disallows":1,"lines":[128,129],"groupRanges":[[128,129]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":161},"any":{"allowed":false,"line":161},"search":{"allowed":false,"line":161},"cart":{"allowed":false,"line":161},"checkout":{"allowed":false,"line":161},"account":{"allowed":false,"line":161}},"disallows":1,"lines":[160,161],"groupRanges":[[160,161]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":121},"any":{"allowed":false,"line":121},"search":{"allowed":false,"line":121},"cart":{"allowed":false,"line":121},"checkout":{"allowed":false,"line":121},"account":{"allowed":false,"line":121}},"disallows":1,"lines":[120,121],"groupRanges":[[120,121]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":169},"any":{"allowed":false,"line":169},"search":{"allowed":false,"line":169},"cart":{"allowed":false,"line":169},"checkout":{"allowed":false,"line":169},"account":{"allowed":false,"line":169}},"disallows":1,"lines":[168,169],"groupRanges":[[168,169]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":153},"any":{"allowed":false,"line":153},"search":{"allowed":false,"line":153},"cart":{"allowed":false,"line":153},"checkout":{"allowed":false,"line":153},"account":{"allowed":false,"line":153}},"disallows":1,"lines":[152,153],"groupRanges":[[152,153]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":145},"any":{"allowed":false,"line":145},"search":{"allowed":false,"line":145},"cart":{"allowed":false,"line":145},"checkout":{"allowed":false,"line":145},"account":{"allowed":false,"line":145}},"disallows":1,"lines":[144,145],"groupRanges":[[144,145]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[4,24],"groupRanges":[[4,24]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":20,"lines":[96,117],"groupRanges":[[96,117]]}],"lanes":{"training":{"level":"mixed","none":5,"full":4,"restricted":0,"n":9},"search":{"level":"mixed","none":1,"full":7,"restricted":0,"n":8},"user_directed":{"level":"full","none":0,"full":7,"restricted":0,"n":7}},"door":{"door":"open","http":200},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"absent","http":404,"url":"https://www.tumblr.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.tumblr.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.tumblr.com/.well-known/tdmrep.json"},"meta":{"title":"Tumblr","robotsMeta":[{"name":"bingbot","content":"nocache"}],"noai":[],"noindex":false,"jsonLdTypes":[],"platform":"WordPress","textRatio":0.001,"llmsLink":false},"x_robots_tag":null,"tos":{"url":"https://www.tumblr.com/policy/terms-of-service","source":"discovered","http":200,"hash":"a964f4be73c5eaf65348a5b8e641a2d43a34d414944ad86d1f763b568e148b2f","chars":38319,"thin":false,"clauses":[{"keyword":"agents","prohibition":true,"excerpt":"You further release, to the fullest extent permitted by law, Tumblr and its employees, agents, consultants, directors, shareholders, any other person or entity that directly or indirectly controls, is under common control with, or is directly or indirectly controlled by, Tumblr (the \"Tumblr Affiliat…"},{"keyword":"agent","prohibition":true,"excerpt":"…tement that the Notifying Party has a good faith belief that the material is not authorized by the copyright owner, its agent or law."},{"keyword":"robots","prohibition":false,"excerpt":"…nterfaces that are provided by Tumblr (and only pursuant to those terms and conditions) or unless permitted by Tumblr's robots.txt file or other robot exclusion mechanisms;"},{"keyword":"scrape","prohibition":false,"excerpt":"(d) scrape the Services, and particularly scrape Content (as defined below) from the Services;"},{"keyword":"agents","prohibition":false,"excerpt":"…poration and wholly-owned subsidiary of Automattic Inc., a Delaware corporation (\" Automattic \"), collectively with its agents, representatives, consultants, employees, officers, and directors, \" Tumblr ,\" \" we ,\" or \" us \")."}],"door":{"door":"open","http":200}},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":44,"first":"2023-01-01","archive_points":43,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":2},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":4},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":5},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6}]},"events":[{"domain":"tumblr.com","name":"Tumblr","category":"social","date":"2024-09","precision":"month","source":"archive","provisional":false,"between":["2024-08-01","2024-09-01"],"summary":"Now blocked: Meta-ExternalAgent.","dir":"up","up":["Meta-ExternalAgent"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240801000836/https://www.tumblr.com/robots.txt","https://web.archive.org/web/20240901000313/https://www.tumblr.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# TurnitinBot crawler"},{"t":"add","s":"User-agent: TurnitinBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Meta AI crawler"},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"tumblr.com","name":"Tumblr","category":"social","date":"2024-07","precision":"month","source":"archive","provisional":false,"between":["2024-06-01","2024-07-01"],"summary":"Now blocked: Applebot-Extended.","dir":"up","up":["Applebot-Extended"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240601000154/https://www.tumblr.com/robots.txt","https://web.archive.org/web/20240701004429/https://www.tumblr.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# Bing's crawler"},{"t":"del","s":"User-agent: Bingbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"# ClaudeBot's crawler"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# ImageSift's AI crawler"},{"t":"add","s":"User-agent: ImagesiftBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Apple's AI crawler"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"tumblr.com","name":"Tumblr","category":"social","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-02-01","2024-03-01"],"summary":"Now blocked: ClaudeBot and anthropic-ai. Now allowed: GPTBot.","dir":"mixed","up":["ClaudeBot","anthropic-ai"],"down":["GPTBot"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240201001731/https://www.tumblr.com/robots.txt","https://web.archive.org/web/20240301005930/https://www.tumblr.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"del","s":"# OpenAI's crawler"},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"ctx","s":"# Common Crawl's crawler"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"# ClaudeBot's crawler"},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# anthropic-ai's crawler"},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"tumblr.com","name":"Tumblr","category":"social","date":"2023-11","precision":"month","source":"archive","provisional":false,"between":["2023-08-01","2023-11-01"],"summary":"Now blocked: GPTBot, Google-Extended, CCBot and Amazonbot.","dir":"up","up":["GPTBot","Google-Extended","CCBot","Amazonbot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230801002934/https://www.tumblr.com/robots.txt","https://web.archive.org/web/20231101002158/https://www.tumblr.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"# OpenAI's crawler"},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Common Crawl's crawler"},{"t":"add","s":"User-agent: CCBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# SentiBot's crawler"},{"t":"add","s":"User-agent: sentibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Google Bard's crawler"},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Facebook's crawler"},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# webz.io's crawler"},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# webz.io's crawler"},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Amazon's crawler"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"# Bing's crawler"},{"t":"add","s":"User-agent: Bingbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]}]}