{"domain":"imdb.com","name":"IMDb","category":"reference","subcategory":"entertainment","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 12 major AI bots we track to stay out of the whole site, and 20 of the 24 bots in total. It does not name any of them. A catch-all rule (<code>User-agent: *</code>) keeps out every bot it does not list by name, AI or not. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.638Z","last_checked":"2026-10-19T13:39:38.069Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://www.imdb.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.imdb.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.imdb.com/robots.txt","hops":[{"from":"https://imdb.com/robots.txt","to":"https://www.imdb.com/robots.txt","status":302}],"blob":"8c80a1f3076cb1ea938bcd8ed7d29f855ac81076e79c9c57fd3a31b84b969b3c","text":"# The content on this site is owned by IMDb.com, Inc., an Amazon.com company\n# and is made available for your personal, non-commercial use subject\n# to our Conditions of Use here: https://www.imdb.com/conditions/\n# Use of any device, tool, or process designed to data mine or scrape the content\n# using automated means is prohibited without prior written permission from IMDb.\n# For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/\n\nUser-agent: Googlebot\nUser-agent: Googlebot-Image\nUser-agent: Googlebot-Video\nUser-agent: Google-PageRenderer\nUser-agent: Google-InspectionTool\nUser-agent: Google-Safety\nUser-agent: Google-Site-Verification\nUser-agent: AdsBot-Google\nUser-agent: AdsBot-Google-Mobile\nUser-agent: Chrome-Lighthouse\nUser-agent: Storebot-Google\nUser-agent: Mediapartners-Google\nUser-agent: GoogleStructuredData\nUser-agent: Google-Read-Aloud\nUser-agent: bingbot\nUser-agent: Applebot\nUser-agent: DuckDuckBot\nUser-agent: DuckAssistBot\nUser-agent: Baiduspider\nUser-agent: Sogou\nUser-agent: MojeekBot\nUser-agent: facebookexternalhit\nUser-agent: Twitterbot\nUser-agent: Pinterestbot\nUser-agent: LinkedInBot\nUser-agent: WhatsApp\nUser-agent: TelegramBot\nUser-agent: Discordbot\nUser-agent: Snapchat\nUser-agent: Slackbot-LinkExpanding\nUser-agent: Amazonbot\nUser-agent: Amzn-SearchBot\nUser-agent: Amzn-User\nUser-agent: ClarityBot\nUser-agent: Citoid\nAllow: /\nDisallow: /register\nDisallow: /*/register\nDisallow: /registration/\nDisallow: /*/registration/\nDisallow: /search/title/*\nDisallow: /*/search/title/*\nDisallow: /search/name/*\nDisallow: /*/search/name/*\nDisallow: /find\nDisallow: /*/find\nDisallow: /find/\nDisallow: /*/find/\nDisallow: /contribute/*\nDisallow: /*/contribute/*\nDisallow: /title/*/review/*\nDisallow: /*/title/*/review/*\nDisallow: /title/*/reviews\nDisallow: /*/title/*/reviews\nDisallow: /user/ur*/reviews\nDisallow: /*/user/ur*/reviews\n\nUser-agent: AmazonAdBot\nAllow: /\n\nUser-agent: *\nDisallow: /\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":4,"text":"Use of any device, tool, or process designed to data mine or scrape the content"}],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["Googlebot","Googlebot-Image","Googlebot-Video","Google-PageRenderer","Google-InspectionTool","Google-Safety","Google-Site-Verification","AdsBot-Google","AdsBot-Google-Mobile","Chrome-Lighthouse","Storebot-Google","Mediapartners-Google","GoogleStructuredData","Google-Read-Aloud","bingbot","Applebot","DuckDuckBot","DuckAssistBot","Baiduspider","Sogou","MojeekBot","facebookexternalhit","Twitterbot","Pinterestbot","LinkedInBot","WhatsApp","TelegramBot","Discordbot","Snapchat","Slackbot-LinkExpanding","Amazonbot","Amzn-SearchBot","Amzn-User","ClarityBot","Citoid","AmazonAdBot","*"],"lines":70},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Google-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]},{"token":"DuckAssistBot","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]},{"token":"Amzn-User","deprecated":false,"matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":69},"any":{"allowed":false,"line":69},"search":{"allowed":false,"line":69},"cart":{"allowed":false,"line":69},"checkout":{"allowed":false,"line":69},"account":{"allowed":false,"line":69}},"disallows":1,"lines":[68,69],"groupRanges":[[68,69]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":43},"any":{"allowed":true,"line":43},"search":{"allowed":true,"line":43},"cart":{"allowed":true,"line":43},"checkout":{"allowed":true,"line":43},"account":{"allowed":true,"line":43}},"disallows":20,"lines":[8,63],"groupRanges":[[8,63]]}],"lanes":{"training":{"level":"none","none":9,"full":0,"restricted":0,"n":9},"search":{"level":"mixed","none":5,"full":3,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":6,"full":1,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":["Amazon CloudFront"],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":45,"first":"2023-01-01","archive_points":44,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-10","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-11","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2024-12","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":3},{"m":"2026-08","s":"blocked","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":21},{"m":"2026-09","s":"blocked","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20},{"m":"2026-10","s":"blocked","l":{"training":"none","search":"mixed","user_directed":"mixed"},"n":20}]},"events":[{"domain":"imdb.com","name":"IMDb","category":"reference","date":"2026-09","precision":"month","source":"archive","provisional":false,"between":["2026-05-01","2026-09-01"],"summary":"Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.","dir":"up","up":["ClaudeBot","Applebot-Extended","Meta-ExternalAgent","Bytespider","MistralAI-Training","AI2Bot","OAI-SearchBot","Claude-SearchBot","PerplexityBot","MistralAI-Index","Meta-WebIndexer","ChatGPT-User","Claude-User","Perplexity-User","Google-Agent","Meta-ExternalFetcher","MistralAI-User"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"blocked","evidence":["https://web.archive.org/web/20260501154213/https://www.imdb.com/robots.txt","https://web.archive.org/web/20260901222738/https://www.imdb.com/robots.txt"],"diff":[{"t":"del","s":"# robots.txt for https://www.imdb.com properties"},{"t":"del","s":"# See https://m.imdb.com/robots.txt for the mDot equivalent"},{"t":"del","s":" "},{"t":"del","s":"User-agent: *"},{"t":"del","s":"Disallow: /OnThisDay"},{"t":"del","s":"Disallow: /*/OnThisDay"},{"t":"del","s":"Disallow: /ads/"},{"t":"del","s":"Disallow: /*/ads/"},{"t":"del","s":"Disallow: /ap/"},{"t":"del","s":"Disallow: /*/ap/"},{"t":"del","s":"Disallow: /mymovies/"},{"t":"del","s":"Disallow: /*/mymovies/"},{"t":"del","s":"Disallow: /r/"},{"t":"del","s":"Disallow: /*/r/"},{"t":"add","s":"# The content on this site is owned by IMDb.com, Inc., an Amazon.com company"},{"t":"add","s":"# and is made available for your personal, non-commercial use subject"},{"t":"add","s":"# to our Conditions of Use here: https://www.imdb.com/conditions/"},{"t":"add","s":"# Use of any device, tool, or process designed to data mine or scrape the content"},{"t":"add","s":"# using automated means is prohibited without prior written permission from IMDb."},{"t":"add","s":"# For authorized data access and licensing inquiries, see: https://www.imdb.com/licensing/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Googlebot"},{"t":"add","s":"User-agent: Googlebot-Image"},{"t":"add","s":"User-agent: Googlebot-Video"},{"t":"add","s":"User-agent: Google-PageRenderer"},{"t":"add","s":"User-agent: Google-InspectionTool"},{"t":"add","s":"User-agent: Google-Safety"},{"t":"add","s":"User-agent: Google-Site-Verification"},{"t":"add","s":"User-agent: AdsBot-Google"},{"t":"add","s":"User-agent: AdsBot-Google-Mobile"},{"t":"add","s":"User-agent: Chrome-Lighthouse"},{"t":"add","s":"User-agent: Storebot-Google"},{"t":"add","s":"User-agent: Mediapartners-Google"},{"t":"add","s":"User-agent: GoogleStructuredData"},{"t":"add","s":"User-agent: Google-Read-Aloud"},{"t":"add","s":"User-agent: bingbot"},{"t":"add","s":"User-agent: Applebot"},{"t":"add","s":"User-agent: DuckDuckBot"},{"t":"add","s":"User-agent: DuckAssistBot"},{"t":"add","s":"User-agent: Baiduspider"},{"t":"add","s":"User-agent: Sogou"},{"t":"add","s":"User-agent: MojeekBot"},{"t":"add","s":"User-agent: facebookexternalhit"},{"t":"add","s":"User-agent: Twitterbot"},{"t":"add","s":"User-agent: Pinterestbot"},{"t":"add","s":"User-agent: LinkedInBot"},{"t":"add","s":"User-agent: WhatsApp"},{"t":"add","s":"User-agent: TelegramBot"},{"t":"add","s":"User-agent: Discordbot"},{"t":"add","s":"User-agent: Snapchat"},{"t":"add","s":"User-agent: Slackbot-LinkExpanding"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"User-agent: Amzn-SearchBot"},{"t":"add","s":"User-agent: Amzn-User"},{"t":"add","s":"User-agent: ClarityBot"},{"t":"add","s":"User-agent: Citoid"},{"t":"add","s":"Allow: /"},{"t":"ctx","s":"Disallow: /register"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*/registration/"},{"t":"gap","s":"… 90 more lines"}]},{"domain":"imdb.com","name":"IMDb","category":"reference","date":"2025-07","precision":"month","source":"archive","provisional":false,"between":["2025-06-01","2025-07-01"],"summary":"Now blocked: GPTBot, Google-Extended, CCBot and anthropic-ai.","dir":"up","up":["GPTBot","Google-Extended","CCBot","anthropic-ai"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20250601000128/https://www.imdb.com/robots.txt","https://web.archive.org/web/20250701003403/https://www.imdb.com/robots.txt"],"diff":[{"t":"ctx","s":"# robots.txt for https://www.imdb.com properties"},{"t":"add","s":"# See https://m.imdb.com/robots.txt for the mDot equivalent"},{"t":"add","s":" "},{"t":"ctx","s":"User-agent: *"},{"t":"ctx","s":"Disallow: /OnThisDay"},{"t":"add","s":"Disallow: /*/OnThisDay"},{"t":"ctx","s":"Disallow: /ads/"},{"t":"add","s":"Disallow: /*/ads/"},{"t":"ctx","s":"Disallow: /ap/"},{"t":"add","s":"Disallow: /*/ap/"},{"t":"ctx","s":"Disallow: /mymovies/"},{"t":"add","s":"Disallow: /*/mymovies/"},{"t":"ctx","s":"Disallow: /r/"},{"t":"add","s":"Disallow: /*/r/"},{"t":"ctx","s":"Disallow: /register"},{"t":"add","s":"Disallow: /*/register"},{"t":"ctx","s":"Disallow: /registration/"},{"t":"add","s":"Disallow: /*/registration/"},{"t":"ctx","s":"Disallow: /search/name-text"},{"t":"add","s":"Disallow: /*/search/name-text"},{"t":"ctx","s":"Disallow: /search/title-text"},{"t":"add","s":"Disallow: /*/search/title-text"},{"t":"ctx","s":"Disallow: /find"},{"t":"add","s":"Disallow: /*find"},{"t":"ctx","s":"Disallow: /find$"},{"t":"add","s":"Disallow: /*/find$"},{"t":"ctx","s":"Disallow: /find/"},{"t":"add","s":"Disallow: /*/find/"},{"t":"ctx","s":"Disallow: /tvschedule"},{"t":"add","s":"Disallow: /*/tvschedule"},{"t":"ctx","s":"Disallow: /updates"},{"t":"add","s":"Disallow: /*/updates"},{"t":"ctx","s":"Disallow: /watch/_ajax/option"},{"t":"add","s":"Disallow: /*/watch/_ajax/option"},{"t":"ctx","s":"Disallow: /_json/video/mon"},{"t":"add","s":"Disallow: /*/_json/video/mon"},{"t":"ctx","s":"Disallow: /_json/getAdsForMediaViewer/"},{"t":"add","s":"Disallow: /*/_json/getAdsForMediaViewer/"},{"t":"ctx","s":"Disallow: /list/ls*/_ajax"},{"t":"add","s":"Disallow: /*/list/ls*/_ajax"},{"t":"add","s":"Disallow: /list/ls*/export"},{"t":"add","s":"Disallow: /*/list/ls*/export"},{"t":"ctx","s":"Disallow: /*/*/rg*/mediaviewer/rm*/tr"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /title/tt*/mediaviewer/rm*/tr"},{"t":"add","s":"Disallow: /*/title/tt*/mediaviewer/rm*/tr"},{"t":"ctx","s":"Disallow: /name/nm*/mediaviewer/rm*/tr"},{"t":"add","s":"Disallow: /*/name/nm*/mediaviewer/rm*/tr"},{"t":"ctx","s":"Disallow: /gallery/rg*/mediaviewer/rm*/tr"},{"t":"add","s":"Disallow: /*/gallery/rg*/mediaviewer/rm*/tr"},{"t":"ctx","s":"Disallow: /tr/"},{"t":"add","s":"Disallow: /*/tr/"},{"t":"ctx","s":"Disallow: /title/tt*/watchoptions"},{"t":"add","s":"Disallow: /*/title/tt*/watchoptions"},{"t":"ctx","s":"Disallow: /search/title/?title_type=feature,tv_movie,tv_miniseries,documentary,short,video,tv_short&release_date=,2020-12-31&lists=%21ls538187658,%21ls539867036,%21ls538186228&view=simple&sort=num_votes,asc&aft"},{"t":"add","s":"Disallow: /*/search/title/?title_type=feature,tv_movie,tv_miniseries,documentary,short,video,tv_short&release_date=,2020-12-31&lists=%21ls538187658,%21ls539867036,%21ls538186228&view=simple&sort=num_votes,asc&aft"},{"t":"ctx","s":"Disallow: /name/nm*/filmotype/*"},{"t":"del","s":""},{"t":"add","s":"Disallow: /*/name/nm*/filmotype/*"},{"t":"add","s":"Disallow: /user/ur*/ratings"},{"t":"gap","s":"… 32 more lines"}]}]}