{"domain":"facebook.com","name":"Facebook","category":"social","subcategory":"network","country":"US","tags":[],"notes":"","status":"blocked","status_label":"Blocks AI","rationale":"Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and PerplexityBot directly. A catch-all rule (<code>User-agent: *</code>) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.","computed_at":"2026-10-20T03:43:15.370Z","last_checked":"2026-10-19T13:36:14.355Z","checker":{"allowed":false,"vantage":"residential connection, United States (manual run)","origin":"https://facebook.com"},"robots":{"via":{"type":"live","date":"2026-10-19","url":"https://facebook.com/robots.txt"},"kind":"file","http":200,"error":null,"url":"https://facebook.com/robots.txt","hops":[],"blob":"ae34650d8199df7bcfbcd9da82b8fa911e522d3201af364ee6cae6e9872653d2","text":"# Notice: Collection of data on Facebook through automated means is\n# prohibited unless you have express written permission from Facebook\n# and may only be conducted for the limited purpose contained in said\n# permission.\n# All authorized user-agents listed on this page must comply with Meta’s\n# Automated Data Collection Terms available at:\n# https://www.facebook.com/legal/automated_data_collection_terms\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Brightbot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: PetalBot\nDisallow: /\n\nUser-agent: Scrapy\nDisallow: /\n\nUser-agent: uptimerobot\nDisallow: /\n\nUser-agent: viberbot\nDisallow: /\n\nUser-agent: YaK\nDisallow: /\n\nUser-agent: Yandex\nDisallow: /\n\nUser-agent: Yeti\nDisallow: /\n\nUser-agent: Applebot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Bingbot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: deepcrawl\nDisallow: /\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /business/\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Discordbot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: DuckDuckBot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: facebookexternalhit\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Googlebot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Googlebot-Image\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /map_tile.php\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /static_map.php\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Googlebot-Video\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Googlebot-News\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Google-InspectionTool\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: lumar\nDisallow: /\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /business/\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: LinkedInBot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: msnbot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Pinterestbot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Screaming Frog SEO Spider\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: seznambot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Slurp\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: teoma\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: TelegramBot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: Twitterbot\nDisallow: /*/plugins/*\nDisallow: /?*next=\nDisallow: /a/bz?\nDisallow: /ajax/\nDisallow: /album.php\nDisallow: /business/*&categories\nDisallow: /business/*?categories\nDisallow: /business/help/search*&query=\nDisallow: /business/help/search*?query=\nDisallow: /business/news*&tags\nDisallow: /business/news*?tags\nDisallow: /business/search*&q=\nDisallow: /business/search*?q=\nDisallow: /checkpoint/\nDisallow: /contact_importer/\nDisallow: /dialog/\nDisallow: /fbml/ajax/dialog/\nDisallow: /feeds/\nDisallow: /file_download.php\nDisallow: /job_application/\nDisallow: /l.php\nDisallow: /login.php*&next=\nDisallow: /login.php/?next=\nDisallow: /login.php?next=\nDisallow: /login/*&next=\nDisallow: /login/?next=\nDisallow: /login/device-based/regular/login/*&next=\nDisallow: /login/device-based/regular/login/?next=\nDisallow: /meta-ai-answers/\nDisallow: /moments_app/\nDisallow: /p.php\nDisallow: /photos.php\nDisallow: /plugins/\nDisallow: /share.php\nDisallow: /share/\nDisallow: /sharer.php\nDisallow: /sharer/\nDisallow: /tr/\nDisallow: /tr?\nDisallow: /ufi/reaction/profile/browser/\nDisallow: /x/oauth/\nAllow: /ajax/bootloader-endpoint/\nAllow: /ajax/pagelet/generic.php/PagePostsSectionPagelet\nAllow: /careers/\nAllow: /safetycheck/\n\nUser-agent: *\nDisallow: /\n\n\n","refused":null,"warnings":[],"contentSignals":[],"contentUsage":[],"licenses":[],"aiComments":[{"line":5,"text":"All authorized user-agents listed on this page must comply with Meta’s"}],"platform":null,"wildcard":{"exists":true,"access":"none"},"named":["Amazonbot","Applebot-Extended","Brightbot","ClaudeBot","Google-Extended","GPTBot","PerplexityBot","PetalBot","Scrapy","uptimerobot","viberbot","YaK","Yandex","Yeti","Applebot","Bingbot","deepcrawl","Discordbot","DuckDuckBot","facebookexternalhit","Googlebot","Googlebot-Image","Googlebot-Video","Googlebot-News","Google-InspectionTool","lumar","LinkedInBot","msnbot","Pinterestbot","Screaming Frog SEO Spider","seznambot","Slurp","teoma","TelegramBot","Twitterbot","*"],"lines":1047},"agents":[{"token":"GPTBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":24},"any":{"allowed":false,"line":24},"search":{"allowed":false,"line":24},"cart":{"allowed":false,"line":24},"checkout":{"allowed":false,"line":24},"account":{"allowed":false,"line":24}},"disallows":1,"lines":[23,24],"groupRanges":[[23,24]]},{"token":"ClaudeBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":18},"any":{"allowed":false,"line":18},"search":{"allowed":false,"line":18},"cart":{"allowed":false,"line":18},"checkout":{"allowed":false,"line":18},"account":{"allowed":false,"line":18}},"disallows":1,"lines":[17,18],"groupRanges":[[17,18]]},{"token":"Google-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":21},"any":{"allowed":false,"line":21},"search":{"allowed":false,"line":21},"cart":{"allowed":false,"line":21},"checkout":{"allowed":false,"line":21},"account":{"allowed":false,"line":21}},"disallows":1,"lines":[20,21],"groupRanges":[[20,21]]},{"token":"Applebot-Extended","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":12},"any":{"allowed":false,"line":12},"search":{"allowed":false,"line":12},"cart":{"allowed":false,"line":12},"checkout":{"allowed":false,"line":12},"account":{"allowed":false,"line":12}},"disallows":1,"lines":[11,12],"groupRanges":[[11,12]]},{"token":"CCBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Meta-ExternalAgent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Bytespider","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"MistralAI-Training","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"AI2Bot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"anthropic-ai","deprecated":true,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"OAI-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Claude-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"PerplexityBot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":27},"any":{"allowed":false,"line":27},"search":{"allowed":false,"line":27},"cart":{"allowed":false,"line":27},"checkout":{"allowed":false,"line":27},"account":{"allowed":false,"line":27}},"disallows":1,"lines":[26,27],"groupRanges":[[26,27]]},{"token":"Amazonbot","deprecated":false,"matched":"named","access":"none","probes":{"root":{"allowed":false,"line":9},"any":{"allowed":false,"line":9},"search":{"allowed":false,"line":9},"cart":{"allowed":false,"line":9},"checkout":{"allowed":false,"line":9},"account":{"allowed":false,"line":9}},"disallows":1,"lines":[8,9],"groupRanges":[[8,9]]},{"token":"Amzn-SearchBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"DuckAssistBot","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"MistralAI-Index","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Meta-WebIndexer","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"ChatGPT-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Claude-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Perplexity-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Google-Agent","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Meta-ExternalFetcher","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"Amzn-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]},{"token":"MistralAI-User","deprecated":false,"matched":"wildcard","access":"none","probes":{"root":{"allowed":false,"line":1044},"any":{"allowed":false,"line":1044},"search":{"allowed":false,"line":1044},"cart":{"allowed":false,"line":1044},"checkout":{"allowed":false,"line":1044},"account":{"allowed":false,"line":1044}},"disallows":1,"lines":[1043,1044],"groupRanges":[[1043,1044]]}],"refs":[{"token":"Googlebot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":41,"lines":[334,379],"groupRanges":[[334,379]]},{"token":"Bingbot","matched":"named","access":"full","probes":{"root":{"allowed":true,"line":null},"any":{"allowed":true,"line":null},"search":{"allowed":true,"line":null},"cart":{"allowed":true,"line":null},"checkout":{"allowed":true,"line":null},"account":{"allowed":true,"line":null}},"disallows":41,"lines":[97,142],"groupRanges":[[97,142]]}],"lanes":{"training":{"level":"none","none":9,"full":0,"restricted":0,"n":9},"search":{"level":"none","none":8,"full":0,"restricted":0,"n":8},"user_directed":{"level":"none","none":7,"full":0,"restricted":0,"n":7}},"door":{"door":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"edge":{"cdn":[],"botManagement":[]},"llms":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"ai_txt":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"tdmrep":{"state":"skipped","reason":"Disallowed for AgentAccessIndexBot by robots.txt"},"meta":null,"x_robots_tag":null,"tos":{"url":null,"source":null,"skipped":"No ToS URL curated or discovered"},"paid":null,"program":null,"statements":[],"claim":null,"reports":[],"history":{"points":46,"first":"2023-01-01","archive_points":45,"series":[{"m":"2023-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2023-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2024-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-08","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-09","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-11","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2025-12","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-01","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-02","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-03","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-04","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-05","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-06","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-07","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24},{"m":"2026-08","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":18},{"m":"2026-09","s":"partial","l":{"training":"mixed","search":"mixed","user_directed":"none"},"n":18},{"m":"2026-10","s":"blocked","l":{"training":"none","search":"none","user_directed":"none"},"n":24}]},"events":[{"domain":"facebook.com","name":"Facebook","category":"social","date":"2026-10","precision":"month","source":"archive+checker","provisional":true,"between":["2026-10-01","2026-10-19"],"summary":"Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.","dir":"up","up":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","PerplexityBot","Amazonbot"],"down":[],"major":true,"signals":false,"status_from":"partial","status_to":"blocked","evidence":["https://web.archive.org/web/20261001201122/https://www.facebook.com/robots.txt","https://facebook.com/robots.txt"],"diff":[{"t":"ctx","s":"# https://www.facebook.com/legal/automated_data_collection_terms"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Brightbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Discordbot"},{"t":"del","s":"Disallow: /*/plugins/*"},{"t":"del","s":"Disallow: /?*next="},{"t":"del","s":"Disallow: /a/bz?"},{"t":"del","s":"Disallow: /ajax/"},{"t":"del","s":"Disallow: /album.php"},{"t":"del","s":"Disallow: /business/*&categories"},{"t":"del","s":"Disallow: /business/*?categories"},{"t":"del","s":"Disallow: /business/help/search*&query="},{"t":"del","s":"Disallow: /business/help/search*?query="},{"t":"del","s":"Disallow: /business/news*&tags"},{"t":"del","s":"Disallow: /business/news*?tags"},{"t":"del","s":"Disallow: /business/search*&q="},{"t":"del","s":"Disallow: /business/search*?q="},{"t":"del","s":"Disallow: /checkpoint/"},{"t":"del","s":"Disallow: /contact_importer/"},{"t":"del","s":"Disallow: /dialog/"},{"t":"del","s":"Disallow: /fbml/ajax/dialog/"},{"t":"del","s":"Disallow: /feeds/"},{"t":"del","s":"Disallow: /file_download.php"},{"t":"del","s":"Disallow: /job_application/"},{"t":"del","s":"Disallow: /l.php"},{"t":"del","s":"Disallow: /login.php*&next="},{"t":"del","s":"Disallow: /login.php/?next="},{"t":"del","s":"Disallow: /login.php?next="},{"t":"del","s":"Disallow: /login/*&next="},{"t":"del","s":"Disallow: /login/?next="},{"t":"del","s":"Disallow: /login/device-based/regular/login/*&next="},{"t":"del","s":"Disallow: /login/device-based/regular/login/?next="},{"t":"del","s":"Disallow: /meta-ai-answers/"},{"t":"del","s":"Disallow: /moments_app/"},{"t":"del","s":"Disallow: /p.php"},{"t":"del","s":"Disallow: /photos.php"},{"t":"del","s":"Disallow: /plugins/"},{"t":"del","s":"Disallow: /share.php"},{"t":"gap","s":"… 313 more lines"}]},{"domain":"facebook.com","name":"Facebook","category":"social","date":"2026-08","precision":"month","source":"archive","provisional":false,"between":["2026-07-29","2026-08-03"],"summary":"Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.","dir":"down","up":[],"down":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","PerplexityBot","Amazonbot"],"major":true,"signals":false,"status_from":"blocked","status_to":"partial","evidence":["https://web.archive.org/web/20260729020032/https://www.facebook.com/robots.txt","https://web.archive.org/web/20260803020630/https://facebook.com/robots.txt"],"diff":[{"t":"ctx","s":"# https://www.facebook.com/legal/automated_data_collection_terms"},{"t":"del","s":"User-agent: Amazonbot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Applebot-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: ClaudeBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: Google-Extended"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: GPTBot"},{"t":"del","s":"Disallow: /"},{"t":"del","s":""},{"t":"del","s":"User-agent: PerplexityBot"},{"t":"add","s":"User-agent: PetalBot"},{"t":"ctx","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: PetalBot"},{"t":"add","s":"User-agent: Scrapy"},{"t":"ctx","s":"Disallow: /"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Yeti"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Applebot"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /album.php"},{"t":"add","s":"Disallow: /business/*&categories"},{"t":"add","s":"Disallow: /business/*?categories"},{"t":"add","s":"Disallow: /business/help/search*&query="},{"t":"add","s":"Disallow: /business/help/search*?query="},{"t":"add","s":"Disallow: /business/news*&tags"},{"t":"add","s":"Disallow: /business/news*?tags"},{"t":"add","s":"Disallow: /business/search*&q="},{"t":"add","s":"Disallow: /business/search*?q="},{"t":"ctx","s":"Disallow: /checkpoint/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /login/device-based/regular/login/?next="},{"t":"add","s":"Disallow: /meta-ai-answers/"},{"t":"ctx","s":"Disallow: /moments_app/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /album.php"},{"t":"add","s":"Disallow: /business/*&categories"},{"t":"add","s":"Disallow: /business/*?categories"},{"t":"add","s":"Disallow: /business/help/search*&query="},{"t":"add","s":"Disallow: /business/help/search*?query="},{"t":"add","s":"Disallow: /business/news*&tags"},{"t":"add","s":"Disallow: /business/news*?tags"},{"t":"add","s":"Disallow: /business/search*&q="},{"t":"add","s":"Disallow: /business/search*?q="},{"t":"ctx","s":"Disallow: /checkpoint/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /login/device-based/regular/login/?next="},{"t":"add","s":"Disallow: /meta-ai-answers/"},{"t":"ctx","s":"Disallow: /moments_app/"},{"t":"gap","s":"…"},{"t":"gap","s":"… 585 more lines"}]},{"domain":"facebook.com","name":"Facebook","category":"social","date":"2025-03","precision":"month","source":"archive","provisional":false,"between":["2025-02-01","2025-03-01"],"summary":"Named PerplexityBot explicitly, without changing what it may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20250201005733/https://www.facebook.com/robots.txt","https://web.archive.org/web/20250301001510/https://www.facebook.com/robots.txt"],"diff":[{"t":"ctx","s":"# permission."},{"t":"del","s":"# See: http://www.facebook.com/apps/site_scraping_tos_terms.php"},{"t":"add","s":"# All authorized user-agents listed on this page must comply with Meta’s"},{"t":"add","s":"# Automated Data Collection Terms available at:"},{"t":"add","s":"# https://www.facebook.com/legal/automated_data_collection_terms"},{"t":"ctx","s":"User-agent: Amazonbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: PerplexityBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: PetalBot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: uptimerobot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: viberbot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"add","s":"User-agent: Yandex"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"ctx","s":"User-agent: Applebot"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: LinkedInBot"},{"t":"add","s":"User-agent: Googlebot-Video"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: msnbot"},{"t":"add","s":"User-agent: Googlebot-News"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Naverbot"},{"t":"add","s":"User-agent: Google-InspectionTool"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Pinterestbot"},{"t":"add","s":"User-agent: LinkedInBot"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Screaming Frog SEO Spider"},{"t":"add","s":"User-agent: msnbot"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: seznambot"},{"t":"add","s":"User-agent: Naverbot"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: Slurp"},{"t":"add","s":"User-agent: Pinterestbot"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"gap","s":"…"},{"t":"gap","s":"… 117 more lines"}]},{"domain":"facebook.com","name":"Facebook","category":"social","date":"2024-08","precision":"month","source":"archive","provisional":false,"between":["2024-07-01","2024-08-01"],"summary":"Named ClaudeBot, Applebot-Extended and Amazonbot explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20240701001135/https://www.facebook.com/robots.txt","https://web.archive.org/web/20240801000043/https://facebook.com/robots.txt"],"diff":[{"t":"ctx","s":"# See: http://www.facebook.com/apps/site_scraping_tos_terms.php"},{"t":"del","s":"User-agent: Applebot"},{"t":"del","s":"Disallow: /*/plugins/*"},{"t":"del","s":"Disallow: /a/bz?"},{"t":"del","s":"Disallow: /ajax/"},{"t":"del","s":"Disallow: /album.php"},{"t":"del","s":"Disallow: /checkpoint/"},{"t":"del","s":"Disallow: /contact_importer/"},{"t":"del","s":"Disallow: /dialog/"},{"t":"del","s":"Disallow: /fbml/ajax/dialog/"},{"t":"del","s":"Disallow: /feeds/"},{"t":"del","s":"Disallow: /file_download.php"},{"t":"del","s":"Disallow: /job_application/"},{"t":"del","s":"Disallow: /l.php"},{"t":"del","s":"Disallow: /login.php?next="},{"t":"del","s":"Disallow: /login/?next="},{"t":"del","s":"Disallow: /moments_app/"},{"t":"del","s":"Disallow: /p.php"},{"t":"del","s":"Disallow: /photos.php"},{"t":"del","s":"Disallow: /plugins/"},{"t":"del","s":"Disallow: /share.php"},{"t":"del","s":"Disallow: /share/"},{"t":"del","s":"Disallow: /sharer.php"},{"t":"del","s":"Disallow: /sharer/"},{"t":"del","s":"Disallow: /tr/"},{"t":"del","s":"Disallow: /tr?"},{"t":"del","s":"Disallow: /ufi/reaction/profile/browser/"},{"t":"del","s":"Disallow: /x/oauth/"},{"t":"add","s":"User-agent: Amazonbot"},{"t":"add","s":"Disallow: /"},{"t":"ctx","s":""},{"t":"del","s":"User-agent: baiduspider"},{"t":"add","s":"User-agent: Applebot-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: ClaudeBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Google-Extended"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: GPTBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: PetalBot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: viberbot"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: YaK"},{"t":"add","s":"Disallow: /"},{"t":"add","s":""},{"t":"add","s":"User-agent: Applebot"},{"t":"ctx","s":"Disallow: /*/plugins/*"},{"t":"add","s":"Disallow: /?*next="},{"t":"ctx","s":"Disallow: /a/bz?"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /l.php"},{"t":"add","s":"Disallow: /login.php*&next="},{"t":"gap","s":"… 621 more lines"}]},{"domain":"facebook.com","name":"Facebook","category":"social","date":"2024-03","precision":"month","source":"archive","provisional":false,"between":["2024-01-01","2024-03-01"],"summary":"Named GPTBot and Google-Extended explicitly, without changing what they may reach.","dir":"other","up":[],"down":[],"major":false,"signals":false,"status_from":"blocked","status_to":"blocked","evidence":["https://web.archive.org/web/20240101000356/https://www.facebook.com/robots.txt","https://web.archive.org/web/20240301000033/https://www.facebook.com/robots.txt"],"diff":[{"t":"ctx","s":"# See: http://www.facebook.com/apps/site_scraping_tos_terms.php"},{"t":"del","s":""},{"t":"ctx","s":"User-agent: Applebot"},{"t":"add","s":"Disallow: /*/plugins/*"},{"t":"add","s":"Disallow: /a/bz?"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /l.php"},{"t":"add","s":"Disallow: /login.php?next="},{"t":"add","s":"Disallow: /login/?next="},{"t":"ctx","s":"Disallow: /moments_app/"},{"t":"ctx","s":"Disallow: /p.php"},{"t":"del","s":"Disallow: /photo.php"},{"t":"ctx","s":"Disallow: /photos.php"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /tr?"},{"t":"add","s":"Disallow: /ufi/reaction/profile/browser/"},{"t":"add","s":"Disallow: /x/oauth/"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: baiduspider"},{"t":"add","s":"Disallow: /*/plugins/*"},{"t":"add","s":"Disallow: /a/bz?"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /l.php"},{"t":"add","s":"Disallow: /login.php?next="},{"t":"add","s":"Disallow: /login/?next="},{"t":"ctx","s":"Disallow: /moments_app/"},{"t":"ctx","s":"Disallow: /p.php"},{"t":"del","s":"Disallow: /photo.php"},{"t":"ctx","s":"Disallow: /photos.php"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /tr?"},{"t":"add","s":"Disallow: /ufi/reaction/profile/browser/"},{"t":"add","s":"Disallow: /x/oauth/"},{"t":"ctx","s":""},{"t":"ctx","s":"User-agent: Bingbot"},{"t":"add","s":"Disallow: /*/plugins/*"},{"t":"add","s":"Disallow: /a/bz?"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /l.php"},{"t":"add","s":"Disallow: /login.php?next="},{"t":"add","s":"Disallow: /login/?next="},{"t":"ctx","s":"Disallow: /moments_app/"},{"t":"ctx","s":"Disallow: /p.php"},{"t":"del","s":"Disallow: /photo.php"},{"t":"ctx","s":"Disallow: /photos.php"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /tr?"},{"t":"add","s":"Disallow: /ufi/reaction/profile/browser/"},{"t":"add","s":"Disallow: /x/oauth/"},{"t":"ctx","s":""},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /"},{"t":"add","s":"Disallow: /*/plugins/*"},{"t":"add","s":"Disallow: /a/bz?"},{"t":"ctx","s":"Disallow: /ajax/"},{"t":"gap","s":"…"},{"t":"ctx","s":"Disallow: /l.php"},{"t":"gap","s":"… 389 more lines"}]}]}