{"domain":"x.com","name":"X","category":"social","subcategory":"network","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names Google-Extended and Meta-ExternalAgent directly. A catch-all rule (<code>User-agent: *</code>) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.339Z","last_checked":"2026-10-19T13:36:11.927Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://x.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://x.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://x.com/robots.txt","hops":[],"blob":"14a72e33b3e9af07e1e8f5f645bfa00ed3fe9f9c87249d861d369d33d3f3ddc6","text":"# Google / Bing Search Engine Robots\n# ==================================\n# Shared group so Googlebot and Bingbot always get the same rules.\nUser-agent: Googlebot\nUser-agent: Bingbot\n\nAllow: /*?s=\nAllow: /*?t=\nAllow: /*?ref_src=\nAllow: /hashtag/*?src=\nAllow: /search?q=%23\nAllow: /i/api/\n# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in\n# favor of Allow, so /i/api/ fetches stay crawlable.\nAllow: /i/api/*\nDisallow: /*?lang=en-ss\nAllow: /*?lang=\nDisallow: /search/realtime\nDisallow: /search/users\nDisallow: /search/*/grid\n\nDisallow: /*/analytics\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/verified_followers\n\nDisallow: /account/deactivated\nDisallow: /settings/deactivated\n\n# Only `*` and `$` are metacharacters here (RFC 9309); regex classes match\n# literally. /photo is anchored so /status/*/photo/1 media pages stay open.\nDisallow: /*/likes\nDisallow: /*/likes?\nDisallow: /*/retweets\nDisallow: /*/retweets?\nDisallow: /*/media\nDisallow: /*/media?\nDisallow: /*/photo$\nDisallow: /*/photo?\nAllow: /*?\n\nUser-agent: facebookexternalhit\n\nAllow: /*?lang=\nAllow: /*?s=\nAllow: /*?t=\nAllow: /*?ref_src=\nAllow: /hashtag/*?src=\nAllow: /search?*cashtagRestId=\nAllow: /i/api/\nAllow: /i/api/*\nDisallow: /search?q=\nDisallow: /search/realtime\nDisallow: /search/users\nDisallow: /search/*/grid\n\nDisallow: /*?\nDisallow: /*/followers\nDisallow: /*/following\nDisallow: /*/verified_followers\n\nDisallow: /account/deactivated\nDisallow: /settings/deactivated\n\nDisallow: /*/likes\nDisallow: /*/likes?\nDisallow: /*/retweets\nDisallow: /*/retweets?\nDisallow: /*/media\nDisallow: /*/media?\nDisallow: /*/photo$\nDisallow: /*/photo?\n\nUser-Agent: Google-Extended\nDisallow: *\n\nUser-Agent: FacebookBot\nDisallow: *\n\nUser-agent: Discordbot\nDisallow: *\n\n# Meta AI / product crawlers (product tokens; versions like /1.1 are ignored\n# under RFC 9309 token matching).\nUser-agent: meta-webindexer\nDisallow: *\n\nUser-agent: meta-externalagent\nDisallow: *\n\nUser-agent: meta-externalads\nDisallow: *\n\nUser-agent: meta-externalfetcher\nDisallow: *\n\n# Every bot that might possibly read and respect this file\n# ========================================================\nUser-agent: *\nDisallow: /\n\n\n# WHAT-4882 - Keep notification-email links (/i/u) out of search results.\n# Named crawlers stay un-blocked on purpose: they must crawl /i/u to see its\n# X-Robots-Tag noindex (the robots.txt Noindex directive died in 2019).\nDisallow: /i/u\n\n# Wait 1 second between successive requests. See ONBOARD-2698 for details.\nCrawl-delay: 1\n\n# Independent of user agent. Links in the sitemap are full URLs using https://\n# and need to match the protocol of the sitemap.\nSitemap: https://x.com/sitemap.xml\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":83,"text":"Meta AI / product crawlers (product tokens; versions like /1.1 are ignored"},{"line":111,"text":"Independent of user agent. Links in the sitemap are full URLs using https://"}],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["Googlebot","Bingbot","facebookexternalhit","Google-Extended","FacebookBot","Discordbot","meta-webindexer","meta-externalagent","meta-externalads","meta-externalfetcher","*"],"lines":114},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"ClaudeBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":75},"any":{"allowed":false,"line":75},"search":{"allowed":false,"line":75},"cart":{"allowed":false,"line":75},"checkout":{"allowed":false,"line":75},"account":{"allowed":false,"line":75}},"disallows":1,"lines":[74,75],"groupRanges":[[74,75]]},{"token":"Applebot-Extended","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":89},"any":{"allowed":false,"line":89},"search":{"allowed":false,"line":89},"cart":{"allowed":false,"line":89},"checkout":{"allowed":false,"line":89},"account":{"allowed":false,"line":89}},"disallows":1,"lines":[88,89],"groupRanges":[[88,89]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"PerplexityBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":86},"any":{"allowed":false,"line":86},"search":{"allowed":false,"line":86},"cart":{"allowed":false,"line":86},"checkout":{"allowed":false,"line":86},"account":{"allowed":false,"line":86}},"disallows":1,"lines":[85,86],"groupRanges":[[85,86]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":95},"any":{"allowed":false,"line":95},"search":{"allowed":false,"line":95},"cart":{"allowed":false,"line":95},"checkout":{"allowed":false,"line":95},"account":{"allowed":false,"line":95}},"disallows":1,"lines":[94,95],"groupRanges":[[94,95]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":100},"any":{"allowed":false,"line":100},"search":{"allowed":false,"line":100},"cart":{"allowed":false,"line":100},"checkout":{"allowed":false,"line":100},"account":{"allowed":false,"line":100}},"disallows":2,"lines":[99,109],"groupRanges":[[99,109]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":18,"lines":[4,40],"groupRanges":[[4,40]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":18,"lines":[4,40],"groupRanges":[[4,40]]}],"lanes":{"training":{"level":"none","none":9,"full":0,"restricted":0,"n":9},"search":{"level":"none","none":8,"full":0,"restricted":0,"n":8},"user_directed":{"level":"none","none":7,"full":0,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":36,"first":"2023-12-06","archive_points":35,"series":[{"m":"2023-01","s":null},{"m":"2023-02","s":null},{"m":"2023-03","s":null},{"m":"2023-04","s":null},{"m":"2023-05","s":null},{"m":"2023-06","s":null},{"m":"2023-07","s":null},{"m":"2023-08","s":null},{"m":"2023-09","s":null},{"m":"2023-10","s":null},{"m":"2023-11","s":null},{"m":"2023-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24}]},"events":[{"domain":"x.com","name":"X","category":"social","date":"2026-09","precision":"month","source":"archive","provisional":false,"between":["2026-08-06","2026-09-03"],"summary":"Named Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20260806020259/https://x.com/robots.txt","https://web.archive.org/web/20260903213435/https://x.com/robots.txt"],"diff":[{"t":"del","s":"# Google Search Engine Robot"},{"t":"del","s":"# =========================="},{"t":"add","s":"# Google / Bing Search Engine Robots"},{"t":"add","s":"# =================================="},{"t":"add","s":"# Shared group so Googlebot and Bingbot always get the same rules."},{"t":"ctx","s":"User-agent: Googlebot"},{"t":"add","s":"User-agent: Bingbot"},{"t":"ctx","s":""},{"t":"del","s":"Allow: /*?lang="},{"t":"add","s":"Allow: /*?s="},{"t":"add","s":"Allow: /*?t="},{"t":"add","s":"Allow: /*?ref_src="},{"t":"ctx","s":"Allow: /hashtag/*?src="},{"t":"gap","s":"…"},{"t":"ctx","s":"Allow: /i/api/"},{"t":"add","s":"# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in"},{"t":"add","s":"# favor of Allow, so /i/api/ fetches stay crawlable."},{"t":"add","s":"Allow: /i/api/*"},{"t":"add","s":"Disallow: /*?lang=en-ss"},{"t":"add","s":"Allow: /*?lang="},{"t":"ctx","s":"Disallow: /search/realtime"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"Disallow: /*/analytics"},{"t":"add","s":"Disallow: /*/followers"},{"t":"add","s":"Disallow: /*/following"},{"t":"add","s":"Disallow: /*/verified_followers"},{"t":"add","s":""},{"t":"add","s":"Disallow: /account/deactivated"},{"t":"add","s":"Disallow: /settings/deactivated"},{"t":"add","s":""},{"t":"add","s":"# Only `*` and `$` are metacharacters here (RFC 9309); regex classes match"},{"t":"add","s":"# literally. /photo is anchored so /status/*/photo/1 media pages stay open."},{"t":"add","s":"Disallow: /*/likes"},{"t":"add","s":"Disallow: /*/likes?"},{"t":"add","s":"Disallow: /*/retweets"},{"t":"add","s":"Disallow: /*/retweets?"},{"t":"add","s":"Disallow: /*/media"},{"t":"add","s":"Disallow: /*/media?"},{"t":"add","s":"Disallow: /*/photo$"},{"t":"add","s":"Disallow: /*/photo?"},{"t":"add","s":"Allow: /*?"},{"t":"add","s":""},{"t":"add","s":"User-agent: facebookexternalhit"},{"t":"add","s":""},{"t":"add","s":"Allow: /*?lang="},{"t":"add","s":"Allow: /*?s="},{"t":"add","s":"Allow: /*?t="},{"t":"add","s":"Allow: /*?ref_src="},{"t":"add","s":"Allow: /hashtag/*?src="},{"t":"add","s":"Allow: /search?*cashtagRestId="},{"t":"add","s":"Allow: /i/api/"},{"t":"add","s":"Allow: /i/api/*"},{"t":"add","s":"Disallow: /search?q="},{"t":"add","s":"Disallow: /search/realtime"},{"t":"add","s":"Disallow: /search/users"},{"t":"add","s":"Disallow: /search/*/grid"},{"t":"add","s":""},{"t":"ctx","s":"Disallow: /*?"},{"t":"gap","s":"…"},{"t":"gap","s":"… 61 more lines"}]},{"domain":"x.com","name":"X","category":"social","date":"2024-04","precision":"month","source":"archive","provisional":false,"between":["2024-03-01","2024-04-01"],"summary":"Named Google-Extended explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20240301002301/https://x.com/robots.txt","https://web.archive.org/web/20240401004311/https://x.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-Agent: Google-Extended"},{"t":"add","s":"Disallow: *"},{"t":"ctx","s":""}]}]}