The Walled Web Newsletter

All sites / Social and forums

Facebook

facebook.com
Blocks AI

Its robots.txt (the public rules file for bots) tells all 24 AI bots we track to stay out of the whole site. It names GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and PerplexityBot directly. A catch-all rule (User-agent: *) keeps the rest out too. The same rules ask our bot to stay out, so we read only robots.txt.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Blocked

9 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Blocked

8 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Blocked

7 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI botAllowed?Where the rule isCompany says it follows robots.txt?
Bots that train AI · Blocked
GPTBotOpenAI Blocked Named, line 23 Says yes
ClaudeBotAnthropic Blocked Named, line 17 Says yes
Google-ExtendedGoogle · opt-out name, not a bot Blocked Named, line 20 Says yes
Applebot-ExtendedApple · opt-out name, not a bot Blocked Named, line 11 Says yes
CCBotCommon Crawl Blocked Not named; the catch-all rule applies Says yes
Meta-ExternalAgentMeta Blocked Not named; the catch-all rule applies Suggests yes
BytespiderByteDance Blocked Not named; the catch-all rule applies No statement found
MistralAI-TrainingMistral AI Blocked Not named; the catch-all rule applies Says yes
AI2BotAllen Institute for AI Blocked Not named; the catch-all rule applies Does not say
anthropic-aiAnthropic (legacy) · old name Blocked Not named; the catch-all rule applies Unknown
Bots for AI search · Blocked
OAI-SearchBotOpenAI Blocked Not named; the catch-all rule applies Says yes
Claude-SearchBotAnthropic Blocked Not named; the catch-all rule applies Says yes
PerplexityBotPerplexity Blocked Named, line 26 Suggests yes
AmazonbotAmazon Blocked Named, line 8 Says yes
Amzn-SearchBotAmazon Blocked Not named; the catch-all rule applies Says yes
DuckAssistBotDuckDuckGo Blocked Not named; the catch-all rule applies Says yes
MistralAI-IndexMistral AI Blocked Not named; the catch-all rule applies Suggests yes
Meta-WebIndexerMeta Blocked Not named; the catch-all rule applies Suggests yes
AI assistants · Blocked
ChatGPT-UserOpenAI Blocked Not named; the catch-all rule applies Says not always
Claude-UserAnthropic Blocked Not named; the catch-all rule applies Says yes
Perplexity-UserPerplexity Blocked Not named; the catch-all rule applies Says it mostly ignores it
Google-AgentGoogle Blocked Not named; the catch-all rule applies Says it mostly ignores it
Meta-ExternalFetcherMeta Blocked Not named; the catch-all rule applies Says not always
Amzn-UserAmazon Blocked Not named; the catch-all rule applies Says not always
MistralAI-UserMistral AI Blocked Not named; the catch-all rule applies Suggests yes
For comparison
GooglebotRegular search engine, for comparisonAllowedNamed
BingbotRegular search engine, for comparisonAllowedNamed

Other AI names in this file

Names in this file that are not among the 24 bots we track. A rule for a name no real bot uses blocks nothing, but it shows which names the site has added.

PetalBotBlockedline 29Not in our verified list
ApplebotHas rulesline 50Documented by its operator

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 1047 lines · 60 lines about AI bots Source
# Notice: Collection of data on Facebook through automated means is# prohibited unless you have express written permission from Facebook# and may only be conducted for the limited purpose contained in said# permission.# All authorized user-agents listed on this page must comply with Meta’s# Automated Data Collection Terms available at:… 6 lines not about AI bots# https://www.facebook.com/legal/automated_data_collection_termsUser-agent: AmazonbotDisallow: / User-agent: Applebot-ExtendedDisallow: / User-agent: BrightbotDisallow: /… 2 lines not about AI bots User-agent: ClaudeBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: GPTBotDisallow: / User-agent: PerplexityBotDisallow: / User-agent: PetalBotDisallow: / User-agent: ScrapyDisallow: / User-agent: uptimerobotDisallow: / User-agent: viberbotDisallow: / User-agent: YaKDisallow: / User-agent: YandexDisallow: / User-agent: YetiDisallow: /… 17 lines not about AI bots User-agent: ApplebotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: BingbotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: deepcrawlDisallow: /Disallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /business/Allow: /careers/Allow: /safetycheck/ User-agent: DiscordbotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: DuckDuckBotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: facebookexternalhitDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: GooglebotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: Googlebot-ImageDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /map_tile.phpDisallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /static_map.phpDisallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: Googlebot-VideoDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: Googlebot-NewsDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: Google-InspectionToolDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: lumarDisallow: /Disallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /business/Allow: /careers/Allow: /safetycheck/ User-agent: LinkedInBotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: msnbotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: PinterestbotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: Screaming Frog SEO SpiderDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: seznambotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: SlurpDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: teomaDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: TelegramBotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: TwitterbotDisallow: /*/plugins/*Disallow: /?*next=Disallow: /a/bz?Disallow: /ajax/Disallow: /album.phpDisallow: /business/*&categoriesDisallow: /business/*?categoriesDisallow: /business/help/search*&query=Disallow: /business/help/search*?query=Disallow: /business/news*&tagsDisallow: /business/news*?tagsDisallow: /business/search*&q=Disallow: /business/search*?q=Disallow: /checkpoint/Disallow: /contact_importer/Disallow: /dialog/Disallow: /fbml/ajax/dialog/Disallow: /feeds/Disallow: /file_download.phpDisallow: /job_application/Disallow: /l.phpDisallow: /login.php*&next=Disallow: /login.php/?next=Disallow: /login.php?next=Disallow: /login/*&next=Disallow: /login/?next=Disallow: /login/device-based/regular/login/*&next=Disallow: /login/device-based/regular/login/?next=Disallow: /meta-ai-answers/Disallow: /moments_app/Disallow: /p.phpDisallow: /photos.phpDisallow: /plugins/Disallow: /share.phpDisallow: /share/Disallow: /sharer.phpDisallow: /sharer/Disallow: /tr/Disallow: /tr?Disallow: /ufi/reaction/profile/browser/Disallow: /x/oauth/Allow: /ajax/bootloader-endpoint/Allow: /ajax/pagelet/generic.php/PagePostsSectionPageletAllow: /careers/Allow: /safetycheck/ User-agent: *Disallow: /   … 951 lines not about AI bots

Notes about AI written in the file: "All authorized user-agents listed on this page must comply with Meta’s"

Everything else we checked

Front doorWe did not visit: the rules ask our bot to stay out.
llms.txtDisallowed for AgentAccessIndexBot by robots.txt
ai.txtDisallowed for AgentAccessIndexBot by robots.txt
TDMRepDisallowed for AgentAccessIndexBot by robots.txt/.well-known/tdmrep.json (W3C Community Group report)
Page tagsHomepage not read.
Anti-bot softwareNo sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AIHomepage not read.Homepage only. Useful for site owners; not part of the result.
Terms of ServiceNo Terms of Service page found.
Not foundNo Content-Signal or Content-Usage lines or licensing signals (RSL License, HTTP 402).

History

46 versions on record since 1 Jan 2023

OverallJan 2023: Overall blockedFeb 2023: Overall blockedMar 2023: Overall blockedApr 2023: Overall blockedMay 2023: Overall blockedJun 2023: Overall blockedJul 2023: Overall blockedAug 2023: Overall blockedSep 2023: Overall blockedOct 2023: Overall blockedNov 2023: Overall blockedDec 2023: Overall blockedJan 2024: Overall blockedFeb 2024: Overall blockedMar 2024: Overall blockedApr 2024: Overall blockedMay 2024: Overall blockedJun 2024: Overall blockedJul 2024: Overall blockedAug 2024: Overall blockedSep 2024: Overall blockedOct 2024: Overall blockedNov 2024: Overall blockedDec 2024: Overall blockedJan 2025: Overall blockedFeb 2025: Overall blockedMar 2025: Overall blockedApr 2025: Overall blockedMay 2025: Overall blockedJun 2025: Overall blockedJul 2025: Overall blockedAug 2025: Overall blockedSep 2025: Overall blockedOct 2025: Overall blockedNov 2025: Overall blockedDec 2025: Overall blockedJan 2026: Overall blockedFeb 2026: Overall blockedMar 2026: Overall blockedApr 2026: Overall blockedMay 2026: Overall blockedJun 2026: Overall blockedJul 2026: Overall blockedAug 2026: Overall partialSep 2026: Overall partialOct 2026: Overall blockedTrainingJan 2023: Training BlockedFeb 2023: Training BlockedMar 2023: Training BlockedApr 2023: Training BlockedMay 2023: Training BlockedJun 2023: Training BlockedJul 2023: Training BlockedAug 2023: Training BlockedSep 2023: Training BlockedOct 2023: Training BlockedNov 2023: Training BlockedDec 2023: Training BlockedJan 2024: Training BlockedFeb 2024: Training BlockedMar 2024: Training BlockedApr 2024: Training BlockedMay 2024: Training BlockedJun 2024: Training BlockedJul 2024: Training BlockedAug 2024: Training BlockedSep 2024: Training BlockedOct 2024: Training BlockedNov 2024: Training BlockedDec 2024: Training BlockedJan 2025: Training BlockedFeb 2025: Training BlockedMar 2025: Training BlockedApr 2025: Training BlockedMay 2025: Training BlockedJun 2025: Training BlockedJul 2025: Training BlockedAug 2025: Training BlockedSep 2025: Training BlockedOct 2025: Training BlockedNov 2025: Training BlockedDec 2025: Training BlockedJan 2026: Training BlockedFeb 2026: Training BlockedMar 2026: Training BlockedApr 2026: Training BlockedMay 2026: Training BlockedJun 2026: Training BlockedJul 2026: Training BlockedAug 2026: Training Some blockedSep 2026: Training Some blockedOct 2026: Training BlockedSearchJan 2023: Search BlockedFeb 2023: Search BlockedMar 2023: Search BlockedApr 2023: Search BlockedMay 2023: Search BlockedJun 2023: Search BlockedJul 2023: Search BlockedAug 2023: Search BlockedSep 2023: Search BlockedOct 2023: Search BlockedNov 2023: Search BlockedDec 2023: Search BlockedJan 2024: Search BlockedFeb 2024: Search BlockedMar 2024: Search BlockedApr 2024: Search BlockedMay 2024: Search BlockedJun 2024: Search BlockedJul 2024: Search BlockedAug 2024: Search BlockedSep 2024: Search BlockedOct 2024: Search BlockedNov 2024: Search BlockedDec 2024: Search BlockedJan 2025: Search BlockedFeb 2025: Search BlockedMar 2025: Search BlockedApr 2025: Search BlockedMay 2025: Search BlockedJun 2025: Search BlockedJul 2025: Search BlockedAug 2025: Search BlockedSep 2025: Search BlockedOct 2025: Search BlockedNov 2025: Search BlockedDec 2025: Search BlockedJan 2026: Search BlockedFeb 2026: Search BlockedMar 2026: Search BlockedApr 2026: Search BlockedMay 2026: Search BlockedJun 2026: Search BlockedJul 2026: Search BlockedAug 2026: Search Some blockedSep 2026: Search Some blockedOct 2026: Search BlockedAssistantsJan 2023: Assistants BlockedFeb 2023: Assistants BlockedMar 2023: Assistants BlockedApr 2023: Assistants BlockedMay 2023: Assistants BlockedJun 2023: Assistants BlockedJul 2023: Assistants BlockedAug 2023: Assistants BlockedSep 2023: Assistants BlockedOct 2023: Assistants BlockedNov 2023: Assistants BlockedDec 2023: Assistants BlockedJan 2024: Assistants BlockedFeb 2024: Assistants BlockedMar 2024: Assistants BlockedApr 2024: Assistants BlockedMay 2024: Assistants BlockedJun 2024: Assistants BlockedJul 2024: Assistants BlockedAug 2024: Assistants BlockedSep 2024: Assistants BlockedOct 2024: Assistants BlockedNov 2024: Assistants BlockedDec 2024: Assistants BlockedJan 2025: Assistants BlockedFeb 2025: Assistants BlockedMar 2025: Assistants BlockedApr 2025: Assistants BlockedMay 2025: Assistants BlockedJun 2025: Assistants BlockedJul 2025: Assistants BlockedAug 2025: Assistants BlockedSep 2025: Assistants BlockedOct 2025: Assistants BlockedNov 2025: Assistants BlockedDec 2025: Assistants BlockedJan 2026: Assistants BlockedFeb 2026: Assistants BlockedMar 2026: Assistants BlockedApr 2026: Assistants BlockedMay 2026: Assistants BlockedJun 2026: Assistants BlockedJul 2026: Assistants BlockedAug 2026: Assistants BlockedSep 2026: Assistants BlockedOct 2026: Assistants BlockedBotsblockedJan 2023: 24 of 24 AI bots blockedFeb 2023: 24 of 24 AI bots blockedMar 2023: 24 of 24 AI bots blockedApr 2023: 24 of 24 AI bots blockedMay 2023: 24 of 24 AI bots blockedJun 2023: 24 of 24 AI bots blockedJul 2023: 24 of 24 AI bots blockedAug 2023: 24 of 24 AI bots blockedSep 2023: 24 of 24 AI bots blockedOct 2023: 24 of 24 AI bots blockedNov 2023: 24 of 24 AI bots blockedDec 2023: 24 of 24 AI bots blockedJan 2024: 24 of 24 AI bots blockedFeb 2024: 24 of 24 AI bots blockedMar 2024: 24 of 24 AI bots blockedApr 2024: 24 of 24 AI bots blockedMay 2024: 24 of 24 AI bots blockedJun 2024: 24 of 24 AI bots blockedJul 2024: 24 of 24 AI bots blockedAug 2024: 24 of 24 AI bots blockedSep 2024: 24 of 24 AI bots blockedOct 2024: 24 of 24 AI bots blockedNov 2024: 24 of 24 AI bots blockedDec 2024: 24 of 24 AI bots blockedJan 2025: 24 of 24 AI bots blockedFeb 2025: 24 of 24 AI bots blockedMar 2025: 24 of 24 AI bots blockedApr 2025: 24 of 24 AI bots blockedMay 2025: 24 of 24 AI bots blockedJun 2025: 24 of 24 AI bots blockedJul 2025: 24 of 24 AI bots blockedAug 2025: 24 of 24 AI bots blockedSep 2025: 24 of 24 AI bots blockedOct 2025: 24 of 24 AI bots blockedNov 2025: 24 of 24 AI bots blockedDec 2025: 24 of 24 AI bots blockedJan 2026: 24 of 24 AI bots blockedFeb 2026: 24 of 24 AI bots blockedMar 2026: 24 of 24 AI bots blockedApr 2026: 24 of 24 AI bots blockedMay 2026: 24 of 24 AI bots blockedJun 2026: 24 of 24 AI bots blockedJul 2026: 24 of 24 AI bots blockedAug 2026: 18 of 24 AI bots blockedSep 2026: 18 of 24 AI bots blockedOct 2026: 24 of 24 AI bots blocked242023202420252026

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

  • Oct 2026
    More blocking Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Changed between the Internet Archive's copy from 1 Oct 2026 and our own check on 19 Oct 2026. Not confirmed yet: seen once so far. Evidence: before · after
    Show exactly what changed
    # https://www.facebook.com/legal/automated_data_collection_terms
    + User-agent: Amazonbot
    + Disallow: /
    +
    + User-agent: Applebot-Extended
    + Disallow: /
    +
    + User-agent: Brightbot
    + Disallow: /
    +
    + User-agent: ClaudeBot
    + Disallow: /
    +
    + User-agent: Google-Extended
    + Disallow: /
    +
    + User-agent: GPTBot
    + Disallow: /
    +
    + User-agent: PerplexityBot
    + Disallow: /
    +
    User-agent: PetalBot
    …
    - User-agent: Discordbot
    - Disallow: /*/plugins/*
    - Disallow: /?*next=
    - Disallow: /a/bz?
    - Disallow: /ajax/
    - Disallow: /album.php
    - Disallow: /business/*&categories
    - Disallow: /business/*?categories
    - Disallow: /business/help/search*&query=
    - Disallow: /business/help/search*?query=
    - Disallow: /business/news*&tags
    - Disallow: /business/news*?tags
    - Disallow: /business/search*&q=
    - Disallow: /business/search*?q=
    - Disallow: /checkpoint/
    - Disallow: /contact_importer/
    - Disallow: /dialog/
    - Disallow: /fbml/ajax/dialog/
    - Disallow: /feeds/
    - Disallow: /file_download.php
    - Disallow: /job_application/
    - Disallow: /l.php
    - Disallow: /login.php*&next=
    - Disallow: /login.php/?next=
    - Disallow: /login.php?next=
    - Disallow: /login/*&next=
    - Disallow: /login/?next=
    - Disallow: /login/device-based/regular/login/*&next=
    - Disallow: /login/device-based/regular/login/?next=
    - Disallow: /meta-ai-answers/
    - Disallow: /moments_app/
    - Disallow: /p.php
    - Disallow: /photos.php
    - Disallow: /plugins/
    - Disallow: /share.php
    … 313 more lines
  • Aug 2026
    Less blocking Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.
    Found in the Internet Archive's copies: changed between 29 Jul 2026 and 3 Aug 2026. Evidence: before · after
    Show exactly what changed
    # https://www.facebook.com/legal/automated_data_collection_terms
    - User-agent: Amazonbot
    - Disallow: /
    -
    - User-agent: Applebot-Extended
    - Disallow: /
    -
    - User-agent: ClaudeBot
    - Disallow: /
    -
    - User-agent: Google-Extended
    - Disallow: /
    -
    - User-agent: GPTBot
    - Disallow: /
    -
    - User-agent: PerplexityBot
    + User-agent: PetalBot
    Disallow: /
    - User-agent: PetalBot
    + User-agent: Scrapy
    Disallow: /
    …
    + User-agent: Yeti
    + Disallow: /
    +
    User-agent: Applebot
    …
    Disallow: /album.php
    + Disallow: /business/*&categories
    + Disallow: /business/*?categories
    + Disallow: /business/help/search*&query=
    + Disallow: /business/help/search*?query=
    + Disallow: /business/news*&tags
    + Disallow: /business/news*?tags
    + Disallow: /business/search*&q=
    + Disallow: /business/search*?q=
    Disallow: /checkpoint/
    …
    Disallow: /login/device-based/regular/login/?next=
    + Disallow: /meta-ai-answers/
    Disallow: /moments_app/
    …
    Disallow: /album.php
    + Disallow: /business/*&categories
    + Disallow: /business/*?categories
    + Disallow: /business/help/search*&query=
    + Disallow: /business/help/search*?query=
    + Disallow: /business/news*&tags
    + Disallow: /business/news*?tags
    + Disallow: /business/search*&q=
    + Disallow: /business/search*?q=
    Disallow: /checkpoint/
    …
    Disallow: /login/device-based/regular/login/?next=
    + Disallow: /meta-ai-answers/
    Disallow: /moments_app/
    …
    … 585 more lines
  • Mar 2025
    Signals Named PerplexityBot explicitly, without changing what it may reach.
    Found in the Internet Archive's copies: changed between 1 Feb 2025 and 1 Mar 2025. Evidence: before · after
    Show exactly what changed
    # permission.
    - # See: http://www.facebook.com/apps/site_scraping_tos_terms.php
    + # All authorized user-agents listed on this page must comply with Meta’s
    + # Automated Data Collection Terms available at:
    + # https://www.facebook.com/legal/automated_data_collection_terms
    User-agent: Amazonbot
    …
    + User-agent: PerplexityBot
    + Disallow: /
    +
    User-agent: PetalBot
    …
    + User-agent: uptimerobot
    + Disallow: /
    +
    User-agent: viberbot
    …
    + User-agent: Yandex
    + Disallow: /
    +
    User-agent: Applebot
    …
    - User-agent: LinkedInBot
    + User-agent: Googlebot-Video
    Disallow: /*/plugins/*
    …
    - User-agent: msnbot
    + User-agent: Googlebot-News
    Disallow: /*/plugins/*
    …
    - User-agent: Naverbot
    + User-agent: Google-InspectionTool
    Disallow: /*/plugins/*
    …
    - User-agent: Pinterestbot
    + User-agent: LinkedInBot
    Disallow: /*/plugins/*
    …
    - User-agent: Screaming Frog SEO Spider
    + User-agent: msnbot
    Disallow: /*/plugins/*
    …
    - User-agent: seznambot
    + User-agent: Naverbot
    Disallow: /*/plugins/*
    …
    - User-agent: Slurp
    + User-agent: Pinterestbot
    Disallow: /*/plugins/*
    …
    … 117 more lines
  • Aug 2024
    Signals Named ClaudeBot, Applebot-Extended and Amazonbot explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Jul 2024 and 1 Aug 2024. Evidence: before · after
    Show exactly what changed
    # See: http://www.facebook.com/apps/site_scraping_tos_terms.php
    - User-agent: Applebot
    - Disallow: /*/plugins/*
    - Disallow: /a/bz?
    - Disallow: /ajax/
    - Disallow: /album.php
    - Disallow: /checkpoint/
    - Disallow: /contact_importer/
    - Disallow: /dialog/
    - Disallow: /fbml/ajax/dialog/
    - Disallow: /feeds/
    - Disallow: /file_download.php
    - Disallow: /job_application/
    - Disallow: /l.php
    - Disallow: /login.php?next=
    - Disallow: /login/?next=
    - Disallow: /moments_app/
    - Disallow: /p.php
    - Disallow: /photos.php
    - Disallow: /plugins/
    - Disallow: /share.php
    - Disallow: /share/
    - Disallow: /sharer.php
    - Disallow: /sharer/
    - Disallow: /tr/
    - Disallow: /tr?
    - Disallow: /ufi/reaction/profile/browser/
    - Disallow: /x/oauth/
    + User-agent: Amazonbot
    + Disallow: /
    - User-agent: baiduspider
    + User-agent: Applebot-Extended
    + Disallow: /
    +
    + User-agent: ClaudeBot
    + Disallow: /
    +
    + User-agent: Google-Extended
    + Disallow: /
    +
    + User-agent: GPTBot
    + Disallow: /
    +
    + User-agent: PetalBot
    + Disallow: /
    +
    + User-agent: viberbot
    + Disallow: /
    +
    + User-agent: YaK
    + Disallow: /
    +
    + User-agent: Applebot
    Disallow: /*/plugins/*
    + Disallow: /?*next=
    Disallow: /a/bz?
    …
    Disallow: /l.php
    + Disallow: /login.php*&next=
    … 621 more lines
  • Mar 2024
    Signals Named GPTBot and Google-Extended explicitly, without changing what they may reach.
    Found in the Internet Archive's copies: changed between 1 Jan 2024 and 1 Mar 2024. Evidence: before · after
    Show exactly what changed
    # See: http://www.facebook.com/apps/site_scraping_tos_terms.php
    -
    User-agent: Applebot
    + Disallow: /*/plugins/*
    + Disallow: /a/bz?
    Disallow: /ajax/
    …
    Disallow: /l.php
    + Disallow: /login.php?next=
    + Disallow: /login/?next=
    Disallow: /moments_app/
    Disallow: /p.php
    - Disallow: /photo.php
    Disallow: /photos.php
    …
    Disallow: /tr?
    + Disallow: /ufi/reaction/profile/browser/
    + Disallow: /x/oauth/
    User-agent: baiduspider
    + Disallow: /*/plugins/*
    + Disallow: /a/bz?
    Disallow: /ajax/
    …
    Disallow: /l.php
    + Disallow: /login.php?next=
    + Disallow: /login/?next=
    Disallow: /moments_app/
    Disallow: /p.php
    - Disallow: /photo.php
    Disallow: /photos.php
    …
    Disallow: /tr?
    + Disallow: /ufi/reaction/profile/browser/
    + Disallow: /x/oauth/
    User-agent: Bingbot
    + Disallow: /*/plugins/*
    + Disallow: /a/bz?
    Disallow: /ajax/
    …
    Disallow: /l.php
    + Disallow: /login.php?next=
    + Disallow: /login/?next=
    Disallow: /moments_app/
    Disallow: /p.php
    - Disallow: /photo.php
    Disallow: /photos.php
    …
    Disallow: /tr?
    + Disallow: /ufi/reaction/profile/browser/
    + Disallow: /x/oauth/
    …
    Disallow: /
    + Disallow: /*/plugins/*
    + Disallow: /a/bz?
    Disallow: /ajax/
    …
    Disallow: /l.php
    … 389 more lines

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.