The Complete List of AI Crawlers & User-Agents (2026)

Every major AI crawler and user-agent in one table — who runs it, what it's for, whether it obeys robots.txt, and whether a Shopify store should let it in. If you block the wrong ones, you're invisible to the AI assistants your buyers use.

AI systems read your store through crawlers — automated agents that fetch your pages so an engine can index, train on, or answer questions about them. Each identifies itself with a user-agent string, and you control access to each one through your robots.txt.

Here's the catch most merchants miss: these crawlers do three different jobs, and you often want to treat them differently.

• Training crawlers collect text to train a model. Blocking them protects your content from being used as training data; it does not affect whether you show up in AI answers.
• Search / indexing crawlers build the index an AI assistant retrieves from when it answers a shopping question. Block these and you become invisible in AI answers.
• User-triggered fetchers load a page in real time because a user asked the assistant about it. Blocking these can stop the assistant from reading your page when a shopper is actively looking at it.

For a store that wants AI visibility, the rule is simple: allow the search and user-triggered agents; decide on training separately.

The complete AI crawler & user-agent list (2026)

Notes: "control token" agents (Google-Extended, Applebot-Extended) aren't crawlers you'll see in logs — they're names you put in robots.txt to opt out of that company's AI training without affecting search. The "contested" entries are the ones worth watching: a user-triggered fetcher may load your page even when a robots rule would block the indexing crawler.

Which ones a Shopify store should allow

If your goal is to be recommended by AI when shoppers ask what to buy, allow every search and answer engine your buyers use — blocking them is self-sabotage:

• Must-allow (these drive shopping answers): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot.
• Fine to allow: Applebot, Amazonbot, YouBot, DuckAssistBot.
• Your call (training-only, no visibility impact): Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider — block these if you don't want your content used for model training, and you'll still appear in AI answers.

The mistake we see most: a store copies a "block AI bots" robots.txt snippet from a privacy-focused blog, blocks GPTBot and ClaudeBot, and then wonders why ChatGPT never recommends it. Those snippets are written for publishers protecting content — not for stores that want to be found.

How to control them in robots.txt

To welcome the answer engines (recommended for stores):

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

To opt out of training while staying visible in answers, disallow only the training/control agents:

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

The gotcha: "allowed in robots.txt" ≠ "actually reachable"

A permissive robots.txt and a permissive edge configuration are not the same thing. A Cloudflare "block AI bots" toggle, a WAF rule, or bot-protection on your hosting can return a challenge or a 403 to a crawler your robots.txt technically allows — and you'll never see it happen. When we scan stores, "robots.txt says yes but the edge says no" is a common, invisible way to be blocked from AI.

Our free robots.txt checker shows which crawlers your file allows in one click. But checking the file isn't enough; you have to confirm the crawler can actually fetch a page. Run a free scan — it checks whether these AI user-agents can reach your store, across all the readiness factors AI engines rely on.

FAQ

Should I block AI crawlers on my Shopify store?

Only the training ones, and only if you specifically don't want your content used to train models. Never block the search/answer engines (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot) — that's what makes ChatGPT and friends unable to recommend you.

Does blocking GPTBot stop me appearing in ChatGPT?

It can hurt. GPTBot is OpenAI's crawler; OAI-SearchBot and ChatGPT-User handle search and live fetches. If you want ChatGPT to be able to read and recommend your store, allow all three.

What's the difference between Google-Extended and Googlebot?

Googlebot crawls for Search (and feeds AI Overviews / AI Mode). Google-Extended is not a crawler — it's a robots.txt token that controls only whether your content is used for Gemini training/grounding. Blocking Google-Extended does not remove you from Google Search.

Do all AI crawlers respect robots.txt?

Most of the major ones do. The contested cases are user-triggered fetchers (like Perplexity-User) that some vendors treat as exempt because a human initiated the request, and crawlers like Bytespider that historically ignored robots.txt. Treat robots.txt as necessary but not a guarantee.

How do I know which crawlers are actually reaching my store?

Check your server/edge logs for these user-agents — and remember your CDN or firewall may be blocking them before they reach your app. The free scanner tests reachability for you.

Sources & maintenance

Compiled from operators' published crawler documentation and our own fleet scanning across 107 Shopify stores. "Contested" entries reflect documented disputes about robots.txt compliance. Crawler names and behaviors change — we review this list quarterly; last updated August 2026.

About the author

Rahul — Founder, FoundGPT

Rahul built FoundGPT and its AI-readiness scanner. Across 111 Shopify stores he has run 23,000+ real AI-search checks on ChatGPT, Gemini, Google AI Mode and Perplexity, and applied ~99,000 structured-data fixes.

→ About FoundGPT