Craigslist

craigslist.org

Allows AI

Its robots.txt (the public rules file for bots) has no rules for any AI bot we track. They are covered by its catch-all rule (User-agent: *), which lets them into the site.

Last checked 19 Oct 2026. Rules file read today. Everything we checked · History

Bots that train AI Allowed

0 of 9 blocked

Blocking these keeps the site out of future AI training. It does not stop AI from reading the site to answer questions.

Bots for AI search Allowed

0 of 8 blocked

Blocking these usually means the site stops showing up as a source in AI answers.

AI assistants Allowed

0 of 7 blocked

5 of these 7 companies say their assistants may ignore robots.txt.

Bot by bot

AI bot	Allowed?	Where the rule is	Company says it follows robots.txt?
Bots that train AI · Allowed
GPTBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
ClaudeBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Google-ExtendedGoogle · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
Applebot-ExtendedApple · opt-out name, not a bot	Allowed	Not named; the catch-all rule applies	Says yes
CCBotCommon Crawl	Allowed	Not named; the catch-all rule applies	Says yes
Meta-ExternalAgentMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
BytespiderByteDance	Allowed	Not named; the catch-all rule applies	No statement found
MistralAI-TrainingMistral AI	Allowed	Not named; the catch-all rule applies	Says yes
AI2BotAllen Institute for AI	Allowed	Not named; the catch-all rule applies	Does not say
anthropic-aiAnthropic (legacy) · old name	Allowed	Not named; the catch-all rule applies	Unknown
Bots for AI search · Allowed
OAI-SearchBotOpenAI	Allowed	Not named; the catch-all rule applies	Says yes
Claude-SearchBotAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
PerplexityBotPerplexity	Allowed	Not named; the catch-all rule applies	Suggests yes
AmazonbotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
Amzn-SearchBotAmazon	Allowed	Not named; the catch-all rule applies	Says yes
DuckAssistBotDuckDuckGo	Allowed	Not named; the catch-all rule applies	Says yes
MistralAI-IndexMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
Meta-WebIndexerMeta	Allowed	Not named; the catch-all rule applies	Suggests yes
AI assistants · Allowed
ChatGPT-UserOpenAI	Allowed	Not named; the catch-all rule applies	Says not always
Claude-UserAnthropic	Allowed	Not named; the catch-all rule applies	Says yes
Perplexity-UserPerplexity	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Google-AgentGoogle	Allowed	Not named; the catch-all rule applies	Says it mostly ignores it
Meta-ExternalFetcherMeta	Allowed	Not named; the catch-all rule applies	Says not always
Amzn-UserAmazon	Allowed	Not named; the catch-all rule applies	Says not always
MistralAI-UserMistral AI	Allowed	Not named; the catch-all rule applies	Suggests yes
For comparison
GooglebotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies
BingbotRegular search engine, for comparison	Allowed	Not named; the catch-all rule applies

We test two pages: the homepage, and a made-up page that stands in for the rest of the site. "Blocked" means both are off limits. "Some pages blocked" means only one is. Rules for search results, carts, checkout and accounts don't change these labels, because almost every site keeps all bots out of those pages.

The rules file (robots.txt)

robots.txt · 11 linesSource

User-agent: *Disallow: /replyDisallow: /fb/Disallow: /suggestDisallow: /flagDisallow: /mfDisallow: /mailflagDisallow: /eaf Disallow: /sitemap/

Everything else we checked

Front door	Let in The homepage loaded normally for our bot, which says who it is (HTTP 200).GET https://craigslist.org/ as WalledWebBot, from a residential connection in the United States
Anti-bot software	No sign of anti-bot software in the response.From response headers and cookie names only. A cookie shows the product is there, not how it is set up.
Readable by AI	Structured data types: WebSite, SearchAction. Readable text is present in the HTML.Homepage only. Useful for site owners; not part of the result.
Terms of Service	www.craigslist.org/about/terms.of.use (found on the homepage) Sentences flagged by keyword. These are machine-selected excerpts, not a legal reading; open the source for context. You agree not to copy/collect CL content via robots, spiders, scripts, scrapers, crawlers, or any automated or manual equivalent (e.g., by hand).matched "robots" · reads like a restriction To the full extent permitted by law, craigslist, Inc., and its officers, directors, employees, agents, licensors, affiliates, and successors in interest ("CL Entities") (1) make no promises, warranties, or representations as to CL, including its completeness, accuracy, availability, timeliness, prop…matched "agents" · reads like a restriction …ur use of CL; and (3) be liable and responsible for any Claims we may have against your officers, directors, employees, agents, affiliates, or any other party, directly or indirectly, paid, directed or controlled by you, or acting for your benefit.matched "agents"
Not found	No llms.txt, ai.txt, tdmrep.json, Content-Signal or Content-Usage lines, licensing signals (RSL License, HTTP 402) or robots meta tags.

History

37 versions on record since 1 Jan 2023

One square per month, from Internet Archive copies and our own daily checks. Dark: blocked. Gray: some blocked. Light: allowed. Striped: no copy that month.

Dec 2025 to Jul 2026
Less blocking Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.
Found in the Internet Archive's copies: changed between 11 Dec 2025 and 4 Jul 2026. Evidence: before · after

Show exactly what changed

- User-agent: Diffbot
- User-agent: FacebookBot
- User-agent: FriendlyCrawler
- User-agent: Google-Extended
- User-agent: ImagesiftBot
- User-agent: img2dataset
- User-agent: Meta-ExternalAgent
- User-agent: omgili
- User-agent: omgilibot
- User-agent: PerplexityBot
- User-agent: Timpibot
- User-agent: VelenPublicWebCrawler
- User-agent: YouBot
- Disallow: /
-
+ Sitemap: https://www.craigslist.org/sitemap-index-categories-and-hubs.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-00.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-01.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-02.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-03.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-04.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-05.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-06.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-07.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-08.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-09.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-10.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-11.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-12.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-13.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-14.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-15.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-16.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-17.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-18.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-19.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-20.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-21.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-22.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-23.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-24.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-25.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-26.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-27.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-28.xml
+ Sitemap: https://www.craigslist.org/sitemap-index-postings-29.xml
Sep 2025
Less blocking Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.
Found in the Internet Archive's copies: changed between 2 Aug 2025 and 17 Sep 2025. Evidence: before · after

Show exactly what changed

- User-agent: Amazonbot
- User-agent: anthropic-ai
- User-agent: Applebot-Extended
- User-agent: Bytespider
- User-agent: CCBot
- User-agent: ClaudeBot
- User-agent: Claude-Web
- User-agent: cohere-ai
User-agent: Diffbot
Jun 2025
Less blocking Now allowed: GPTBot, OAI-SearchBot and ChatGPT-User.
Found in the Internet Archive's copies: changed between 22 May 2025 and 19 Jun 2025. Evidence: before · after

Show exactly what changed
User-agent: CCBot
- User-agent: ChatGPT-User
User-agent: ClaudeBot
…
User-agent: Google-Extended
- User-agent: GPTBot
User-agent: ImagesiftBot
…
User-agent: Meta-ExternalAgent
- User-agent: OAI-SearchBot
User-agent: omgili
Aug 2024
More blocking Now blocked: GPTBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, anthropic-ai, OAI-SearchBot and 3 more.
Found in the Internet Archive's copies: changed between 1 Jul 2024 and 4 Aug 2024. Evidence: before · after

Show exactly what changed

- User-agent: claudebot
+ User-agent: Amazonbot
+ User-agent: anthropic-ai
+ User-agent: Applebot-Extended
+ User-agent: Bytespider
+ User-agent: CCBot
+ User-agent: ChatGPT-User
+ User-agent: ClaudeBot
+ User-agent: Claude-Web
+ User-agent: cohere-ai
+ User-agent: Diffbot
+ User-agent: FacebookBot
+ User-agent: FriendlyCrawler
+ User-agent: Google-Extended
+ User-agent: GPTBot
+ User-agent: ImagesiftBot
+ User-agent: img2dataset
+ User-agent: Meta-ExternalAgent
+ User-agent: OAI-SearchBot
+ User-agent: omgili
+ User-agent: omgilibot
+ User-agent: PerplexityBot
+ User-agent: Timpibot
+ User-agent: VelenPublicWebCrawler
+ User-agent: YouBot
Disallow: /
May 2024
More blocking Now blocked: ClaudeBot.
Found in the Internet Archive's copies: changed between 1 Apr 2024 and 1 May 2024. Evidence: before · after

Show exactly what changed
Disallow: /eaf
- Disallow: /*.html?lang=
- Sitemap: https://craigslist.org/sitemap_index.xml

+ User-agent: claudebot
+ Disallow: /
+
+

Reports from readers

No reports yet. Tell us what happened when your AI tool tried this site, even if it worked fine.