{"domain":"instagram.com","name":"Instagram","category":"social","subcategory":"network","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and PerplexityBot directly. A catch-all rule (<code>User-agent: *</code>) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.397Z","last_checked":"2026-10-19T13:36:23.998Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://www.instagram.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.instagram.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.instagram.com/robots.txt","hops":[{"from":"https://instagram.com/robots.txt","to":"https://www.instagram.com/robots.txt","status":301}],"blob":"fe84b3913976aa37e983fd44f2ace9cf62973a512ae694eb18f3cf02066602b1","text":"# Notice: Collection of data on Instagram through automated means is\n# prohibited unless you have express written permission from Instagram\n# and may only be conducted for the limited purpose contained in said\n# permission.\n# All authorized user-agents listed on this page must comply with Meta’s\n# Automated Data Collection Terms available at:\n# https://www.facebook.com/legal/automated_data_collection_terms\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Brightbot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: uptimerobot\nDisallow: /\n\nUser-agent: viberbot\nDisallow: /\n\nUser-agent: YaK\nDisallow: /\n\nUser-agent: Yandex\nDisallow: /\n\nUser-agent: Yeti\nDisallow: /\n\nUser-agent: Applebot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Bingbot\nDisallow: /*/c/\nDisallow: /*/comments/\nDisallow: /*/liked_by/\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: deepcrawl\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Discordbot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: DuckDuckBot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: facebookexternalhit\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Googlebot\nDisallow: /*/c/\nDisallow: /*/comments/\nDisallow: /*/liked_by/\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Googlebot-Image\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Googlebot-Video\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Googlebot-News\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Google-InspectionTool\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: lumar\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: LinkedInBot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: msnbot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Pinterestbot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Screaming Frog SEO Spider\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: seznambot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Slurp\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: teoma\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: TelegramBot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: Twitterbot\nDisallow: /accounts/login/*?next=\nDisallow: /ajax/\nDisallow: /client_error/\nDisallow: /direct/\nDisallow: /logging/\nDisallow: /publicapi/\nDisallow: /qp/batch_fetch_web/\nDisallow: /query/\nAllow: /places/c/\n\nUser-agent: *\nDisallow: /\n\nSitemap: https://www.instagram.com/sitemap/ig_places_sitemap.xml.gz\nSitemap: https://www.instagram.com/sitemap/ig_seo_profile_sitemap.xml.gz\nSitemap: https://www.instagram.com/sitemap/ig_seo_profile_sitemap_non_media_eligible.xml.gz\nSitemap: https://www.instagram.com/sitemap/dim_ig_logged_out_web_keyword_ent.xml.gz\n\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":5,"text":"All authorized user-agents listed on this page must comply with Meta’s"}],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["Amazonbot","Applebot-Extended","Brightbot","ClaudeBot","Google-Extended","GPTBot","PerplexityBot","PetalBot","Scrapy","uptimerobot","viberbot","YaK","Yandex","Yeti","Applebot","Bingbot","deepcrawl","Discordbot","DuckDuckBot","facebookexternalhit","Googlebot","Googlebot-Image","Googlebot-Video","Googlebot-News","Google-InspectionTool","lumar","LinkedInBot","msnbot","Pinterestbot","Screaming Frog SEO Spider","seznambot","Slurp","teoma","TelegramBot","Twitterbot","*"],"lines":295},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":24},"any":{"allowed":false,"line":24},"search":{"allowed":false,"line":24},"cart":{"allowed":false,"line":24},"checkout":{"allowed":false,"line":24},"account":{"allowed":false,"line":24}},"disallows":1,"lines":[23,24],"groupRanges":[[23,24]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":18},"any":{"allowed":false,"line":18},"search":{"allowed":false,"line":18},"cart":{"allowed":false,"line":18},"checkout":{"allowed":false,"line":18},"account":{"allowed":false,"line":18}},"disallows":1,"lines":[17,18],"groupRanges":[[17,18]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":21},"any":{"allowed":false,"line":21},"search":{"allowed":false,"line":21},"cart":{"allowed":false,"line":21},"checkout":{"allowed":false,"line":21},"account":{"allowed":false,"line":21}},"disallows":1,"lines":[20,21],"groupRanges":[[20,21]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":12},"any":{"allowed":false,"line":12},"search":{"allowed":false,"line":12},"cart":{"allowed":false,"line":12},"checkout":{"allowed":false,"line":12},"account":{"allowed":false,"line":12}},"disallows":1,"lines":[11,12],"groupRanges":[[11,12]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":27},"any":{"allowed":false,"line":27},"search":{"allowed":false,"line":27},"cart":{"allowed":false,"line":27},"checkout":{"allowed":false,"line":27},"account":{"allowed":false,"line":27}},"disallows":1,"lines":[26,27],"groupRanges":[[26,27]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":9},"any":{"allowed":false,"line":9},"search":{"allowed":false,"line":9},"cart":{"allowed":false,"line":9},"checkout":{"allowed":false,"line":9},"account":{"allowed":false,"line":9}},"disallows":1,"lines":[8,9],"groupRanges":[[8,9]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":288},"any":{"allowed":false,"line":288},"search":{"allowed":false,"line":288},"cart":{"allowed":false,"line":288},"checkout":{"allowed":false,"line":288},"account":{"allowed":false,"line":288}},"disallows":1,"lines":[287,288],"groupRanges":[[287,288]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[119,131],"groupRanges":[[119,131]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":11,"lines":[61,73],"groupRanges":[[61,73]]}],"lanes":{"training":{"level":"none","none":9,"full":0,"restricted":0,"n":9},"search":{"level":"none","none":8,"full":0,"restricted":0,"n":8},"user_directed":{"level":"none","none":7,"full":0,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":26,"first":"2023-01-01","archive_points":25,"series":[{"m":"2023-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24}]},"events":[{"domain":"instagram.com","name":"Instagram","category":"social","date":"2026-04","precision":"month","source":"archive","provisional":false,"between":["2025-01-01","2026-04-01"],"summary":"Named PerplexityBot explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20250101000504/https://www.instagram.com/robots.txt","https://web.archive.org/web/20260401004226/https://www.instagram.com/robots.txt"],"diff":[{"t":"ctx","s":"# permission."},{"t":"add","s":"# All authorized user-agents listed on this page must comply with Meta’s"},{"t":"add","s":"# Automated Data Collection Terms available at:"},{"t":"add","s":"# https://www.facebook.com/legal/automated_data_collection_terms"},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Brightbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: ClaudeBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Scrapy"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: uptimerobot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: viberbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Yandex"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Yeti"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /accounts/login/*?next="},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /direct/"},{"t":"ctx","s":"Disallow: /logging/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /*/liked_by/"},{"t":"add","s":"Disallow: /accounts/login/*?next="},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /direct/"},{"t":"ctx","s":"Disallow: /logging/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Discordbot"},{"t":"add","s":"Disallow: /accounts/login/*?next="},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /direct/"},{"t":"ctx","s":"Disallow: /logging/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: DuckDuckBot"},{"t":"add","s":"Disallow: /accounts/login/*?next="},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /direct/"},{"t":"gap","s":"… 144 more lines"}]},{"domain":"instagram.com","name":"Instagram","category":"social","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-07-01","2024-08-01"],"summary":"Named ClaudeBot, Applebot-Extended and Amazonbot explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20240701000115/https://www.instagram.com/robots.txt","https://web.archive.org/web/20240801001531/https://www.instagram.com/robots.txt"],"diff":[{"t":"add","s":"# Notice: Collection of data on Instagram through automated means is"},{"t":"add","s":"# prohibited unless you have express written permission from Instagram"},{"t":"add","s":"# and may only be conducted for the limited purpose contained in said"},{"t":"add","s":"# permission."},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PetalBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: viberbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: YaK"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Applebot"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"del","s":"Disallow: /publicapi/"},{"t":"del","s":"Disallow: /query/"},{"t":"del","s":"Disallow: /logging/"},{"t":"del","s":"Disallow: /qp/batch_fetch_web/"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"del","s":""},{"t":"del","s":"User-agent: baiduspider"},{"t":"del","s":"Disallow: /ajax/"},{"t":"del","s":"Disallow: /publicapi/"},{"t":"del","s":"Disallow: /query/"},{"t":"ctx","s":"Disallow: /logging/"},{"t":"add","s":"Disallow: /publicapi/"},{"t":"ctx","s":"Disallow: /qp/batch_fetch_web/"},{"t":"del","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /query/"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: Bingbot"},{"t":"add","s":"Disallow: /*/c/"},{"t":"add","s":"Disallow: /*/comments/"},{"t":"add","s":"Disallow: /*/liked_by/"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"add","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /logging/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"add","s":"Disallow: /qp/batch_fetch_web/"},{"t":"ctx","s":"Disallow: /query/"},{"t":"add","s":""},{"t":"add","s":"User-agent: Discordbot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"add","s":"Disallow: /client_error/"},{"t":"gap","s":"… 180 more lines"}]},{"domain":"instagram.com","name":"Instagram","category":"social","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-02-01","2024-03-01"],"summary":"Named GPTBot and Google-Extended explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20240201000013/https://www.instagram.com/robots.txt","https://web.archive.org/web/20240301000747/https://www.instagram.com/robots.txt"],"diff":[{"t":"ctx","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: baiduspider"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Bingbot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /*/comments/"},{"t":"add","s":"Disallow: /*/c/"},{"t":"add","s":"Disallow: /*/liked_by/"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: DuckDuckBot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Googlebot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /*/comments/"},{"t":"add","s":"Disallow: /*/c/"},{"t":"add","s":"Disallow: /*/liked_by/"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: ia_archiver"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: msnbot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Naverbot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /client_error/"},{"t":"add","s":"Disallow: /*/comments/"},{"t":"add","s":"Disallow: /*/c/"},{"t":"add","s":"Disallow: /*/liked_by/"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: seznambot"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"…"},{"t":"ctx","s":"User-agent: Slurp"},{"t":"add","s":"Disallow: /ajax/"},{"t":"ctx","s":"Disallow: /publicapi/"},{"t":"gap","s":"… 16 more lines"}]}]}