ClaudeBot: Anthropic's Crawler, Its User-Agent, and Whether to Allow It

ClaudeBot is Anthropic's web crawler for Claude. Here's the user-agent, what it's used for, how Anthropic treats robots.txt, and whether a Shopify store should let it in.

Claude is one of the assistants shoppers increasingly ask for recommendations, so whether Claude can read your store matters. The crawler behind it is ClaudeBot.

What ClaudeBot is

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +http://www.anthropic.com/claude-bot

ClaudeBot crawls the open web to gather data for Anthropic's models. Anthropic also operates a separate, user-triggered fetch for Claude's web features — which retrieves a specific page in real time when someone's question requires it, rather than crawling broadly.

Anthropic publishes its crawler and respects robots.txt, so your rules are honoured here — unlike some crawlers where compliance has been questioned.

Should a store allow it?

ClaudeBot sits closer to GPTBot than to OAI-SearchBot: it's primarily about model data rather than a live search index. Allowing it means Claude's understanding of the web — and your brand within it — can include you. Blocking it is a defensible content/IP choice with the same tradeoff as blocking GPTBot: you opt out of the training/knowledge side, which is a real decision, not an obvious win.

For a store that wants to be known to Claude, the default (allow) is usually right.

The robots.txt

# Allow Anthropic's crawler
User-agent: ClaudeBot
Allow: /

# To opt out instead:
# User-agent: ClaudeBot
# Disallow: /

Check it by name

• Run your store through the AI crawler & robot checker — see, per bot, whether ClaudeBot and the rest can reach you.
• Scan your store's AI readiness — crawler access is one piece of whether AI engines can read and recommend your products.

See also: GPTBot · OAI-SearchBot · PerplexityBot · Google-Extended · the complete AI crawler & user-agent list.

FoundGPT helps Shopify stores get read, recommended and sold by AI engines across ChatGPT, Gemini, Google AI Mode, Perplexity and Claude. Start with a free AI-readiness scan.