{"domain":"ft.com","name":"Financial Times","category":"news","subcategory":"business","country":"GB","tags":[],"notes":"","status":"partial","status_label":"Blocks some AI","rationale":"Its robots.txt (the public rules file for bots) treats AI bots differently from each other. It blocks 9 of the 24 AI bots we track from the whole site, including ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent. GPTBot, OAI-SearchBot, Claude-SearchBot, ChatGPT-User and 1 more can reach most of it. The homepage also showed our bot, which says who it is, a Cloudflare challenge page instead of the site.","computed_at":"2026-10-20T03:43:15.228Z","last_checked":"2026-10-19T13:34:49.183Z","checker":{"allowed":true,"vantage":"residential connection, United States (manual run)","origin":"https://www.ft.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://www.ft.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://www.ft.com/robots.txt","hops":[{"from":"https://ft.com/robots.txt","to":"https://www.ft.com/robots.txt","status":301}],"blob":"57ce1e58ae57b17699c48037deda0f549296f5c0bfe44e92cf7af6deb6767b4b","text":"# All use of FT content is subject to the Terms & Conditions and\n# Copyright Policy set out on FT.com\n\n# We expressly prohibit any use of our content or data for any\n# machine learning or artificial intelligence purposes, including\n# the training or development of AI technologies or machine learning\n# language models.\n#\n# For any licensing queries please contact content.licensing@ft.com\n\nSitemap: https://www.ft.com/sitemaps/index.xml\n\nUser-agent: *\nDisallow: /__\nDisallow: /_fs-ch\nDisallow: /article/e63d8640-eeb3-4473-b0ad-aa6aec16d9fe\nDisallow: /buy/\nDisallow: /cancel/\nDisallow: /join/\nDisallow: /login\nDisallow: /myft\nDisallow: /offline/\nDisallow: /page-resources\nDisallow: /preferences/manage-cookies\nDisallow: /register/\nDisallow: /search\nDisallow: /signup\nAllow: /__origami/\nAllow: /__assets/\nCrawl-Delay: 1\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\nAllow: /*?syn-25a6b1a6=1\n\nUser-agent: GoogleOther\nDisallow: /\n\nUser-agent: anthropic-ai\nDisallow: /\n\nUser-agent: omgili\nDisallow: /\n\nUser-agent: omgilibot\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: cohere-ai\nDisallow: /\n\nUser-agent: magpie-crawler\nDisallow: /\n\nUser-agent: NewsNow\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: Claude-Web\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Diffbot\nDisallow: /\n\nUser-agent: FacebookBot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: Meta-ExternalFetcher\nDisallow: /\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":5,"text":"machine learning or artificial intelligence purposes, including"},{"line":6,"text":"the training or development of AI technologies or machine learning"}],"platform":null,"wildcard":{"exists":true,"access":"full"},"named":["*","CCBot","Google-Extended","GoogleOther","anthropic-ai","omgili","omgilibot","Bytespider","cohere-ai","magpie-crawler","NewsNow","PerplexityBot","Perplexity-User","Claude-Web","ClaudeBot","Diffbot","FacebookBot","Applebot-Extended","meta-externalagent","Meta-ExternalFetcher"],"lines":89},"agents":[{"token":"GPTBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":73},"any":{"allowed":false,"line":73},"search":{"allowed":false,"line":73},"cart":{"allowed":false,"line":73},"checkout":{"allowed":false,"line":73},"account":{"allowed":false,"line":73}},"disallows":1,"lines":[72,73],"groupRanges":[[72,73]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":36},"any":{"allowed":false,"line":36},"search":{"allowed":false,"line":36},"cart":{"allowed":false,"line":36},"checkout":{"allowed":false,"line":36},"account":{"allowed":false,"line":36}},"disallows":1,"lines":[35,37],"groupRanges":[[35,37]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":82},"any":{"allowed":false,"line":82},"search":{"allowed":false,"line":82},"cart":{"allowed":false,"line":82},"checkout":{"allowed":false,"line":82},"account":{"allowed":false,"line":82}},"disallows":1,"lines":[81,82],"groupRanges":[[81,82]]},{"token":"CCBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":33},"any":{"allowed":false,"line":33},"search":{"allowed":false,"line":33},"cart":{"allowed":false,"line":33},"checkout":{"allowed":false,"line":33},"account":{"allowed":false,"line":33}},"disallows":1,"lines":[32,33],"groupRanges":[[32,33]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":85},"any":{"allowed":false,"line":85},"search":{"allowed":false,"line":85},"cart":{"allowed":false,"line":85},"checkout":{"allowed":false,"line":85},"account":{"allowed":false,"line":85}},"disallows":1,"lines":[84,85],"groupRanges":[[84,85]]},{"token":"Bytespider","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":52},"any":{"allowed":false,"line":52},"search":{"allowed":false,"line":52},"cart":{"allowed":false,"line":52},"checkout":{"allowed":false,"line":52},"account":{"allowed":false,"line":52}},"disallows":1,"lines":[51,52],"groupRanges":[[51,52]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"anthropic-ai","deprecated":true,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":43},"any":{"allowed":false,"line":43},"search":{"allowed":false,"line":43},"cart":{"allowed":false,"line":43},"checkout":{"allowed":false,"line":43},"account":{"allowed":false,"line":43}},"disallows":1,"lines":[42,43],"groupRanges":[[42,43]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":64},"any":{"allowed":false,"line":64},"search":{"allowed":false,"line":64},"cart":{"allowed":false,"line":64},"checkout":{"allowed":false,"line":64},"account":{"allowed":false,"line":64}},"disallows":1,"lines":[63,64],"groupRanges":[[63,64]]},{"token":"Amazonbot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Perplexity-User","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":67},"any":{"allowed":false,"line":67},"search":{"allowed":false,"line":67},"cart":{"allowed":false,"line":67},"checkout":{"allowed":false,"line":67},"account":{"allowed":false,"line":67}},"disallows":1,"lines":[66,67],"groupRanges":[[66,67]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":88},"any":{"allowed":false,"line":88},"search":{"allowed":false,"line":88},"cart":{"allowed":false,"line":88},"checkout":{"allowed":false,"line":88},"account":{"allowed":false,"line":88}},"disallows":1,"lines":[87,88],"groupRanges":[[87,88]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]}],"refs":[{"token":"Googlebot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]},{"token":"Bingbot","matched":"wildcard","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":false,"line":26},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":14,"lines":[13,30],"groupRanges":[[13,30]]}],"lanes":{"training":{"level":"mixed","none":6,"full":3,"restricted":0,"n":9},"search":{"level":"mixed","none":1,"full":7,"restricted":0,"n":8},"user_directed":{"level":"mixed","none":2,"full":5,"restricted":0,"n":7}},"door":{"door":"challenged","http":403,"detail":"Cloudflare challenge page"},"edge":{"cdn":["Cloudflare"],"botManagement":[{"vendor":"Cloudflare Bot Management","cookies":["__cf_bm"]}]},"llms":{"state":"absent","http":404,"url":"https://www.ft.com/llms.txt"},"ai_txt":{"state":"absent","http":404,"url":"https://www.ft.com/ai.txt"},"tdmrep":{"state":"absent","http":404,"url":"https://www.ft.com/.well-known/tdmrep.json"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":46,"first":"2023-01-01","archive_points":45,"series":[{"m":"2023-01","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-02","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-03","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-04","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-05","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-06","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-07","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-08","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-09","s":"allowed","l":{"training":"full","search":"full","user_directed":"full"},"n":0},{"m":"2023-10","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-11","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2023-12","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-01","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-02","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-03","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-04","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-05","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-06","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-07","s":"partial","l":{"training":"mixed","search":"full","user_directed":"full"},"n":1},{"m":"2024-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":6},{"m":"2024-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2024-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"full"},"n":7},{"m":"2025-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-11","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2025-12","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-01","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-02","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-03","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-04","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-05","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-06","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-07","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9},{"m":"2026-10","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"mixed"},"n":9}]},"events":[{"domain":"ft.com","name":"Financial Times","category":"news","date":"2025-05","precision":"month","source":"archive","provisional":false,"between":["2025-04-01","2025-05-01"],"summary":"Now blocked: Perplexity-User and Meta-ExternalFetcher.","dir":"up","up":["Perplexity-User","Meta-ExternalFetcher"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20250401021020/https://www.ft.com/robots.txt","https://web.archive.org/web/20250501052735/https://www.ft.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: GoogleOther"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: anthropic-ai"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Perplexity-User"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Claude-Web"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Diffbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: FacebookBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Meta-ExternalFetcher"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"ft.com","name":"Financial Times","category":"news","date":"2024-11","precision":"month","source":"archive","provisional":false,"between":["2024-10-01","2024-11-01"],"summary":"Now blocked: Meta-ExternalAgent.","dir":"up","up":["Meta-ExternalAgent"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20241001015441/https://www.ft.com/robots.txt","https://web.archive.org/web/20241101004400/https://www.ft.com/robots.txt"],"diff":[{"t":"ctx","s":""},{"t":"add","s":"User-agent: meta-externalagent"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"ft.com","name":"Financial Times","category":"news","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-07-01","2024-08-01"],"summary":"Now blocked: ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, anthropic-ai and PerplexityBot. Now allowed: GPTBot.","dir":"mixed","up":["ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","anthropic-ai","PerplexityBot"],"down":["GPTBot"],"major":true,"signals":false,"status_from":"partial","status_to":"partial","evidence":["https://web.archive.org/web/20240701005256/https://www.ft.com/robots.txt","https://web.archive.org/web/20240801004438/https://www.ft.com/robots.txt"],"diff":[{"t":"del","s":"# all use of FT content is subject to the Terms & Conditions and Copyright Policy set out on FT.com"},{"t":"add","s":"# All use of FT content is subject to the Terms & Conditions and"},{"t":"add","s":"# Copyright Policy set out on FT.com"},{"t":"add","s":""},{"t":"add","s":"# We expressly prohibit any use of our content or data for any"},{"t":"add","s":"# machine learning or artificial intelligence purposes, including"},{"t":"add","s":"# the training or development of AI technologies or machine learning"},{"t":"add","s":"# language models."},{"t":"add","s":"#"},{"t":"add","s":"# For any licensing queries please contact content.licensing@ft.com"},{"t":"add","s":""},{"t":"ctx","s":"Sitemap: https://www.ft.com/sitemaps/index.xml"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"add","s":"User-agent: CCBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: anthropic-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgili"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: omgilibot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Bytespider"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: cohere-ai"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: magpie-crawler"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: NewsNow"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Claude-Web"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: FacebookBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]},{"domain":"ft.com","name":"Financial Times","category":"news","date":"2023-10","precision":"month","source":"archive","provisional":false,"between":["2023-09-01","2023-10-01"],"summary":"Now blocked: GPTBot.","dir":"up","up":["GPTBot"],"down":[],"major":true,"signals":false,"status_from":"allowed","status_to":"partial","evidence":["https://web.archive.org/web/20230901012747/https://www.ft.com/robots.txt","https://web.archive.org/web/20231001000230/https://www.ft.com/robots.txt"],"diff":[{"t":"ctx","s":"Disallow: /preferences/cookies"},{"t":"del","s":"Allow: /myft/list/"},{"t":"add","s":"Disallow: /login"},{"t":"ctx","s":"Allow: /__origami/"},{"t":"ctx","s":"Allow: /__assets/"},{"t":"add","s":"Crawl-Delay: 1"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""}]}]}