https://walledweb.com/changes/AI-relevant changes to the robots.txt files of tracked sites.https://walledweb.com/site/airbnb.com/#historyairbnb.com:2026-10:GPTBot,ClaudeBot,Applebot-Extended,MistralAI-Training,AI2Bot:Invalid DateAirbnb: Now blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.https://walledweb.com/site/aliexpress.com/#historyaliexpress.com:2026-10:GPTBot,ClaudeBot,Google-Extended,CCBot,Bytespider,Amazonbot:Invalid DateAliExpress: Now blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.https://walledweb.com/site/bestbuy.com/#historybestbuy.com:2026-10::Invalid DateBest Buy: Named GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, anthropic-ai and 6 more explicitly, without changing what they may reach.https://walledweb.com/site/bloomberg.com/#historybloomberg.com:2026-10::OAI-SearchBot,Claude-SearchBot,ChatGPT-User,Claude-UserInvalid DateBloomberg: Now partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.https://walledweb.com/site/capitalone.com/#historycapitalone.com:2026-10::Invalid DateCapital One: Stopped naming ClaudeBot and Amazonbot; access unchanged.https://walledweb.com/site/cnn.com/#historycnn.com:2026-10::Google-ExtendedInvalid DateCNN: Now allowed: Google-Extended.https://walledweb.com/site/ebay.com/#historyebay.com:2026-10::Invalid DateeBay: Named OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User explicitly, without changing what they may reach.https://walledweb.com/site/facebook.com/#historyfacebook.com:2026-10:GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,PerplexityBot,Amazonbot:Invalid DateFacebook: Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.https://walledweb.com/site/github.com/#historygithub.com:2026-10:Bytespider:Invalid DateGitHub: Now blocked: Bytespider.https://walledweb.com/site/glassdoor.com/#historyglassdoor.com:2026-10:ClaudeBot:Invalid DateGlassdoor: Now blocked: ClaudeBot.https://walledweb.com/site/humana.com/#historyhumana.com:2026-10::Invalid DateHumana: Named ClaudeBot, OAI-SearchBot and Claude-SearchBot explicitly, without changing what they may reach.https://walledweb.com/site/mountsinai.org/#historymountsinai.org:2026-10::Invalid DateMount Sinai: Stopped naming CCBot; access unchanged.https://walledweb.com/site/nordstrom.com/#historynordstrom.com:2026-10::Invalid DateNordstrom: Named Meta-ExternalAgent and Meta-WebIndexer explicitly, without changing what they may reach.https://walledweb.com/site/npr.org/#historynpr.org:2026-10:Claude-SearchBot,Claude-User:Invalid DateNPR: Now blocked: Claude-SearchBot and Claude-User.https://walledweb.com/site/pinterest.com/#historypinterest.com:2026-10:Meta-ExternalFetcher:Invalid DatePinterest: Now blocked: Meta-ExternalFetcher.https://walledweb.com/site/imdb.com/#historyimdb.com:2026-09:ClaudeBot,Applebot-Extended,Meta-ExternalAgent,Bytespider,MistralAI-Training,AI2Bot,OAI-SearchBot,Claude-SearchBot,PerplexityBot,MistralAI-Index,Meta-WebIndexer,ChatGPT-User,Claude-User,Perplexity-User,Google-Agent,Meta-ExternalFetcher,MistralAI-User:Invalid DateIMDb: Now blocked: ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot and 9 more.https://walledweb.com/site/indeed.com/#historyindeed.com:2026-09::Invalid DateIndeed: Named ClaudeBot, Meta-ExternalAgent, AI2Bot, Claude-SearchBot, PerplexityBot, DuckAssistBot and 5 more explicitly, without changing what they may reach.https://walledweb.com/site/reuters.com/#historyreuters.com:2026-09::Invalid DateReuters: Named Applebot-Extended explicitly, without changing what it may reach.https://walledweb.com/site/webmd.com/#historywebmd.com:2026-09::ChatGPT-UserInvalid DateWebMD: Now allowed: ChatGPT-User.https://walledweb.com/site/x.com/#historyx.com:2026-09::Invalid DateX: Named Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher explicitly, without changing what they may reach.https://walledweb.com/site/economist.com/#historyeconomist.com:2026-08:ClaudeBot,OAI-SearchBot,Claude-SearchBot,Claude-User:Invalid DateThe Economist: Now blocked: ClaudeBot, OAI-SearchBot, Claude-SearchBot and Claude-User.https://walledweb.com/site/facebook.com/#historyfacebook.com:2026-08::GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,PerplexityBot,AmazonbotInvalid DateFacebook: Now allowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.https://walledweb.com/site/axios.com/#historyaxios.com:2026-07::Invalid DateAxios: Named Google-Extended explicitly, without changing what it may reach.https://walledweb.com/site/craigslist.org/#historycraigslist.org:2026-07::Google-Extended,Meta-ExternalAgent,PerplexityBotInvalid DateCraigslist: Now allowed: Google-Extended, Meta-ExternalAgent and PerplexityBot.https://walledweb.com/site/linkedin.com/#historylinkedin.com:2026-07:Bytespider:Claude-SearchBotInvalid DateLinkedIn: Now blocked: Bytespider. Now allowed: Claude-SearchBot.https://walledweb.com/site/twitch.tv/#historytwitch.tv:2026-07::Invalid DateTwitch: Named Amazonbot, Amzn-SearchBot and Amzn-User explicitly, without changing what they may reach.https://walledweb.com/site/amazon.com/#historyamazon.com:2026-06:Meta-WebIndexer:Invalid DateAmazon: Now blocked: Meta-WebIndexer.https://walledweb.com/site/bloomberg.com/#historybloomberg.com:2026-06:ClaudeBot,Applebot-Extended,OAI-SearchBot,Claude-SearchBot,DuckAssistBot,Claude-User,Meta-ExternalFetcher:Invalid DateBloomberg: Now blocked: ClaudeBot, Applebot-Extended, OAI-SearchBot, Claude-SearchBot, DuckAssistBot, Claude-User and Meta-ExternalFetcher.https://walledweb.com/site/cnn.com/#historycnn.com:2026-06:Claude-SearchBot,Claude-User,Perplexity-User,MistralAI-User:Invalid DateCNN: Now blocked: Claude-SearchBot, Claude-User, Perplexity-User and MistralAI-User.https://walledweb.com/site/reuters.com/#historyreuters.com:2026-06:GPTBot,MistralAI-Training,Amazonbot,Amzn-SearchBot,DuckAssistBot,MistralAI-Index,Claude-User,Perplexity-User,Google-Agent:Invalid DateReuters: Now blocked: GPTBot, MistralAI-Training, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Claude-User, Perplexity-User and 1 more.https://walledweb.com/site/theverge.com/#historytheverge.com:2026-06:Meta-ExternalAgent,AI2Bot,Claude-SearchBot,DuckAssistBot,Claude-User,Perplexity-User,Meta-ExternalFetcher,MistralAI-User:Invalid DateThe Verge: Now blocked: Meta-ExternalAgent, AI2Bot, Claude-SearchBot, DuckAssistBot, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User.https://walledweb.com/site/usatoday.com/#historyusatoday.com:2026-06:Amzn-SearchBot,Meta-WebIndexer,Perplexity-User,Google-Agent,Amzn-User:Invalid DateUSA Today: Now blocked: Amzn-SearchBot, Meta-WebIndexer, Perplexity-User, Google-Agent and Amzn-User.https://walledweb.com/site/wsj.com/#historywsj.com:2026-06:MistralAI-Training,AI2Bot,Claude-SearchBot,Amazonbot,Amzn-SearchBot,DuckAssistBot,MistralAI-Index,Meta-WebIndexer,Claude-User,Perplexity-User,Google-Agent,Amzn-User,MistralAI-User:Invalid DateThe Wall Street Journal: Now blocked: MistralAI-Training, AI2Bot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, DuckAssistBot, MistralAI-Index, Meta-WebIndexer and 5 more.https://walledweb.com/site/docs.stripe.com/#historydocs.stripe.com:2026-05::Invalid DateStripe Docs: Added Content-Signal: ai-train=yes, search=yes, ai-input=yes.https://walledweb.com/site/gov.uk/#historygov.uk:2026-05::Invalid DateGOV.UK: Named Meta-ExternalAgent explicitly, without changing what it may reach.https://walledweb.com/site/instacart.com/#historyinstacart.com:2026-05::Invalid DateInstacart: Named Meta-ExternalAgent explicitly, without changing what it may reach.https://walledweb.com/site/macys.com/#historymacys.com:2026-05::Invalid DateMacy's: Named ClaudeBot explicitly, without changing what it may reach.https://walledweb.com/site/pinterest.com/#historypinterest.com:2026-05:MistralAI-Training,AI2Bot,OAI-SearchBot,Claude-SearchBot,Amazonbot,Amzn-SearchBot,MistralAI-Index,Meta-WebIndexer,Claude-User,Perplexity-User,Google-Agent,Amzn-User,MistralAI-User:Invalid DatePinterest: Now blocked: MistralAI-Training, AI2Bot, OAI-SearchBot, Claude-SearchBot, Amazonbot, Amzn-SearchBot, MistralAI-Index, Meta-WebIndexer and 5 more.https://walledweb.com/site/supabase.com/#historysupabase.com:2026-05::Invalid DateSupabase: Named GPTBot, ClaudeBot, Google-Extended and PerplexityBot explicitly, without changing what they may reach. Added Content-Signal: ai-train=yes, search=yes, ai-input=yes.https://walledweb.com/site/vercel.com/#historyvercel.com:2026-05::Invalid DateVercel: Added Content-Signal: search=yes, ai-input=yes, ai-train=no.https://walledweb.com/site/ebay.com/#historyebay.com:2026-04:GPTBot,Applebot-Extended,Meta-ExternalAgent:Invalid DateeBay: Now blocked: GPTBot, Applebot-Extended and Meta-ExternalAgent.https://walledweb.com/site/instagram.com/#historyinstagram.com:2026-04::Invalid DateInstagram: Named PerplexityBot explicitly, without changing what it may reach.https://walledweb.com/site/nytimes.com/#historynytimes.com:2026-04:Meta-WebIndexer:Invalid DateThe New York Times: Now blocked: Meta-WebIndexer.https://walledweb.com/site/theatlantic.com/#historytheatlantic.com:2026-04:Meta-ExternalAgent,Claude-SearchBot,Claude-User,Meta-ExternalFetcher,MistralAI-User:Invalid DateThe Atlantic: Now blocked: Meta-ExternalAgent, Claude-SearchBot, Claude-User, Meta-ExternalFetcher and MistralAI-User. Added Content-Signal: search=yes, ai-input=no, ai-train=no.https://walledweb.com/site/zillow.com/#historyzillow.com:2026-04:Applebot-Extended,Meta-ExternalAgent,AI2Bot,Amazonbot:Invalid DateZillow: Now blocked: Applebot-Extended, Meta-ExternalAgent, AI2Bot and Amazonbot.https://walledweb.com/site/axios.com/#historyaxios.com:2026-03::ClaudeBot,Google-ExtendedInvalid DateAxios: Now allowed: ClaudeBot and Google-Extended.https://walledweb.com/site/hotels.com/#historyhotels.com:2026-03::Invalid DateHotels.com: Named Perplexity-User explicitly, without changing what it may reach.https://walledweb.com/site/instacart.com/#historyinstacart.com:2026-03:PerplexityBot:Invalid DateInstacart: Now blocked: PerplexityBot.https://walledweb.com/site/reuters.com/#historyreuters.com:2026-03:AI2Bot,MistralAI-User:Claude-UserInvalid DateReuters: Now blocked: AI2Bot and MistralAI-User. Now allowed: Claude-User.https://walledweb.com/site/reddit.com/#historyreddit.com:2026-02:GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot,Meta-ExternalAgent,Bytespider,MistralAI-Training,AI2Bot,anthropic-ai,OAI-SearchBot,Claude-SearchBot,PerplexityBot,Amazonbot,Amzn-SearchBot,DuckAssistBot,MistralAI-Index,Meta-WebIndexer,ChatGPT-User,Claude-User,Perplexity-User,Google-Agent,Meta-ExternalFetcher,Amzn-User,MistralAI-User:Invalid DateReddit: Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.https://walledweb.com/site/stackoverflow.com/#historystackoverflow.com:2026-02::Invalid DateStack Overflow: Named GPTBot, Google-Extended, Amazonbot and ChatGPT-User explicitly, without changing what they may reach. Removed its RSL License: line.https://walledweb.com/site/tripadvisor.com/#historytripadvisor.com:2026-02:CCBot,Meta-ExternalAgent,Amazonbot:Invalid DateTripadvisor: Now blocked: CCBot, Meta-ExternalAgent and Amazonbot.https://walledweb.com/site/zocdoc.com/#historyzocdoc.com:2026-02::Invalid DateZocdoc: Named ClaudeBot, Google-Extended, Claude-SearchBot, PerplexityBot, Claude-User and Perplexity-User explicitly, without changing what they may reach.https://walledweb.com/site/bbc.com/#historybbc.com:2026-01:Amzn-SearchBot,Amzn-User:Invalid DateBBC: Now blocked: Amzn-SearchBot and Amzn-User.https://walledweb.com/site/cnn.com/#historycnn.com:2026-01:Amzn-SearchBot,Amzn-User:Meta-ExternalAgent,Meta-ExternalFetcherInvalid DateCNN: Now blocked: Amzn-SearchBot and Amzn-User. Now allowed: Meta-ExternalAgent and Meta-ExternalFetcher.https://walledweb.com/site/ebay.com/#historyebay.com:2026-01:ClaudeBot,CCBot,anthropic-ai,PerplexityBot,Amazonbot:Invalid DateeBay: Now blocked: ClaudeBot, CCBot, anthropic-ai, PerplexityBot and Amazonbot.https://walledweb.com/site/amazon.com/#historyamazon.com:2025-12:OAI-SearchBot,Claude-SearchBot,ChatGPT-User,Claude-User,Perplexity-User,Meta-ExternalFetcher:Invalid DateAmazon: Now blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher.https://walledweb.com/site/linkedin.com/#historylinkedin.com:2025-12::OAI-SearchBotInvalid DateLinkedIn: Now allowed: OAI-SearchBot.https://walledweb.com/site/stackoverflow.com/#historystackoverflow.com:2025-12::Invalid DateStack Overflow: Added Content-Signal: search=yes. Added an RSL License: line pointing to licensing terms.https://walledweb.com/site/ihg.com/#historyihg.com:2025-11::Invalid DateIHG Hotels: Stopped naming OAI-SearchBot, PerplexityBot and ChatGPT-User; access unchanged.https://walledweb.com/site/zocdoc.com/#historyzocdoc.com:2025-11::Invalid DateZocdoc: Named GPTBot, OAI-SearchBot and ChatGPT-User explicitly, without changing what they may reach.https://walledweb.com/site/developers.cloudflare.com/#historydevelopers.cloudflare.com:2025-10::Invalid DateCloudflare Docs: Added Content-Signal: ai-train=yes, search=yes, ai-input=yes.https://walledweb.com/site/ebay.com/#historyebay.com:2025-10:Bytespider:Invalid DateeBay: Now blocked: Bytespider.https://walledweb.com/site/newyorker.com/#historynewyorker.com:2025-10:ClaudeBot,Claude-SearchBot,DuckAssistBot,Meta-WebIndexer,Claude-User,Perplexity-User,Meta-ExternalFetcher,MistralAI-User:anthropic-aiInvalid DateThe New Yorker: Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.https://walledweb.com/site/nytimes.com/#historynytimes.com:2025-10::AmazonbotInvalid DateThe New York Times: Now allowed: Amazonbot.https://walledweb.com/site/tiktok.com/#historytiktok.com:2025-10:GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot,Meta-ExternalAgent,AI2Bot,anthropic-ai,OAI-SearchBot,Claude-SearchBot,PerplexityBot,DuckAssistBot,ChatGPT-User,Claude-User,Perplexity-User,MistralAI-User:Invalid DateTikTok: Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, AI2Bot, anthropic-ai and 8 more.https://walledweb.com/site/tripadvisor.com/#historytripadvisor.com:2025-10:GPTBot:Invalid DateTripadvisor: Now blocked: GPTBot.https://walledweb.com/site/wired.com/#historywired.com:2025-10:ClaudeBot,Claude-SearchBot,DuckAssistBot,Meta-WebIndexer,Claude-User,Perplexity-User,Meta-ExternalFetcher,MistralAI-User:anthropic-aiInvalid DateWIRED: Now blocked: ClaudeBot, Claude-SearchBot, DuckAssistBot, Meta-WebIndexer, Claude-User, Perplexity-User, Meta-ExternalFetcher and MistralAI-User. Now allowed: anthropic-ai.https://walledweb.com/site/amazon.com/#historyamazon.com:2025-09:ClaudeBot,Google-Extended,Meta-ExternalAgent,Bytespider,AI2Bot,DuckAssistBot,MistralAI-User:Invalid DateAmazon: Now blocked: ClaudeBot, Google-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, DuckAssistBot and MistralAI-User.https://walledweb.com/site/chewy.com/#historychewy.com:2025-09:Amazonbot:Invalid DateChewy: Now blocked: Amazonbot.https://walledweb.com/site/craigslist.org/#historycraigslist.org:2025-09::ClaudeBot,Applebot-Extended,CCBot,Bytespider,anthropic-ai,AmazonbotInvalid DateCraigslist: Now allowed: ClaudeBot, Applebot-Extended, CCBot, Bytespider, anthropic-ai and Amazonbot.https://walledweb.com/site/economist.com/#historyeconomist.com:2025-09:Perplexity-User:Invalid DateThe Economist: Now blocked: Perplexity-User.https://walledweb.com/site/expedia.com/#historyexpedia.com:2025-09::Invalid DateExpedia: Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, Claude-User and Perplexity-User explicitly, without changing what they may reach.https://walledweb.com/site/hotels.com/#historyhotels.com:2025-09::Invalid DateHotels.com: Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User explicitly, without changing what they may reach.https://walledweb.com/site/ikea.com/#historyikea.com:2025-09::GPTBot,Google-Extended,CCBotInvalid DateIKEA: Now allowed: GPTBot, Google-Extended and CCBot.https://walledweb.com/site/linkedin.com/#historylinkedin.com:2025-09::Invalid DateLinkedIn: Named GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent, anthropic-ai and 7 more explicitly, without changing what they may reach.https://walledweb.com/site/medium.com/#historymedium.com:2025-09:Meta-ExternalAgent,Amazonbot:Invalid DateMedium: Now blocked: Meta-ExternalAgent and Amazonbot.https://walledweb.com/site/pinterest.com/#historypinterest.com:2025-09:Google-Extended,Applebot-Extended,CCBot,Meta-ExternalAgent,anthropic-ai,PerplexityBot,ChatGPT-User:Invalid DatePinterest: Now blocked: Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, anthropic-ai, PerplexityBot and ChatGPT-User.https://walledweb.com/site/politico.com/#historypolitico.com:2025-09:GPTBot,Google-Extended,Applebot-Extended,Meta-ExternalAgent,Bytespider,AI2Bot,anthropic-ai,OAI-SearchBot,Claude-SearchBot,PerplexityBot,Amazonbot,DuckAssistBot,ChatGPT-User,Claude-User,Perplexity-User,Meta-ExternalFetcher,MistralAI-User:Invalid DatePolitico: Now blocked: GPTBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, AI2Bot, anthropic-ai, OAI-SearchBot and 9 more.https://walledweb.com/site/reddit.com/#historyreddit.com:2025-09::GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot,Meta-ExternalAgent,Bytespider,MistralAI-Training,AI2Bot,anthropic-ai,OAI-SearchBot,Claude-SearchBot,PerplexityBot,Amazonbot,Amzn-SearchBot,DuckAssistBot,MistralAI-Index,Meta-WebIndexer,ChatGPT-User,Claude-User,Perplexity-User,Google-Agent,Meta-ExternalFetcher,Amzn-User,MistralAI-UserInvalid DateReddit: Now partly blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 17 more.https://walledweb.com/site/temu.com/#historytemu.com:2025-09::Invalid DateTemu: Named GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot, Claude-SearchBot, PerplexityBot and 2 more explicitly, without changing what they may reach.https://walledweb.com/site/theatlantic.com/#historytheatlantic.com:2025-09:Perplexity-User:Invalid DateThe Atlantic: Now blocked: Perplexity-User.https://walledweb.com/site/usatoday.com/#historyusatoday.com:2025-09:MistralAI-User:Invalid DateUSA Today: Now blocked: MistralAI-User.https://walledweb.com/site/vrbo.com/#historyvrbo.com:2025-09::Invalid DateVrbo: Named OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User explicitly, without changing what they may reach.https://walledweb.com/site/aa.com/#historyaa.com:2025-08::GPTBotInvalid DateAmerican Airlines: Now allowed: GPTBot.https://walledweb.com/site/apnews.com/#historyapnews.com:2025-08:Claude-SearchBot,Claude-User:Invalid DateAssociated Press: Now blocked: Claude-SearchBot and Claude-User.https://walledweb.com/site/bbc.com/#historybbc.com:2025-08:Amazonbot,Perplexity-User:Invalid DateBBC: Now blocked: Amazonbot and Perplexity-User.https://walledweb.com/site/bloomberg.com/#historybloomberg.com:2025-08:Meta-ExternalAgent,Bytespider,Amazonbot:Invalid DateBloomberg: Now blocked: Meta-ExternalAgent, Bytespider and Amazonbot.https://walledweb.com/site/glassdoor.com/#historyglassdoor.com:2025-08::ClaudeBot,PerplexityBot,ChatGPT-UserInvalid DateGlassdoor: Now allowed: ClaudeBot, PerplexityBot and ChatGPT-User.https://walledweb.com/site/latimes.com/#historylatimes.com:2025-08:ClaudeBot,Bytespider:Invalid DateLos Angeles Times: Now blocked: ClaudeBot and Bytespider.https://walledweb.com/site/nytimes.com/#historynytimes.com:2025-08:Claude-SearchBot,Claude-User,Perplexity-User:Invalid DateThe New York Times: Now blocked: Claude-SearchBot, Claude-User and Perplexity-User.https://walledweb.com/site/opentable.com/#historyopentable.com:2025-08:GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot,Meta-ExternalAgent,Bytespider,MistralAI-Training,AI2Bot,anthropic-ai,Amzn-SearchBot,DuckAssistBot,MistralAI-Index,Meta-WebIndexer,Google-Agent,Meta-ExternalFetcher,Amzn-User,MistralAI-User:Invalid DateOpenTable: Now blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, MistralAI-Training and 10 more.https://walledweb.com/site/reuters.com/#historyreuters.com:2025-08:Claude-SearchBot,Claude-User:Invalid DateReuters: Now blocked: Claude-SearchBot and Claude-User.https://walledweb.com/site/washingtonpost.com/#historywashingtonpost.com:2025-08::Google-ExtendedInvalid DateThe Washington Post: Now allowed: Google-Extended.https://walledweb.com/site/imdb.com/#historyimdb.com:2025-07:GPTBot,Google-Extended,CCBot,anthropic-ai:Invalid DateIMDb: Now blocked: GPTBot, Google-Extended, CCBot and anthropic-ai.https://walledweb.com/site/medium.com/#historymedium.com:2025-07::Google-ExtendedInvalid DateMedium: Now allowed: Google-Extended. Added an RSL License: line pointing to licensing terms.https://walledweb.com/site/npr.org/#historynpr.org:2025-07:OAI-SearchBot:Invalid DateNPR: Now blocked: OAI-SearchBot.https://walledweb.com/site/ticketmaster.com/#historyticketmaster.com:2025-07::GPTBotInvalid DateTicketmaster: Now allowed: GPTBot.https://walledweb.com/site/usatoday.com/#historyusatoday.com:2025-07:AI2Bot,DuckAssistBot,Meta-ExternalFetcher:Invalid DateUSA Today: Now blocked: AI2Bot, DuckAssistBot and Meta-ExternalFetcher.https://walledweb.com/site/zillow.com/#historyzillow.com:2025-07::GPTBotInvalid DateZillow: Now allowed: GPTBot.