GPTBot and OpenAI's Other Crawlers: User-Agents, and Which Ones You Should Never Block

OpenAI doesn't run one crawler — it runs three, and they do completely different jobs. Block the wrong one and you quietly remove your store from ChatGPT's shopping answers. Here's each user-agent, what it's for, and the robots.txt that keeps you visible.

If you've looked at your server logs or your robots.txt and seen GPTBot, you've met one of OpenAI's crawlers. But GPTBot is the one everyone knows about, and it's arguably the least important for a store that wants to show up in ChatGPT. OpenAI runs three separate bots, each with its own user-agent and its own job — and the difference between them matters more than almost anything else in your AI-visibility setup.

The three OpenAI user-agents

GPTBot — model training.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

GPTBot crawls the open web to gather data that may be used to train future OpenAI models. It's the one with a genuine tradeoff: allowing it may help future models recognise your brand and products; blocking it is a reasonable choice if you don't want your content used for training. Match on the token GPTBot — the version number increments over time.

OAI-SearchBot — search and shopping surfacing.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

This is the crawler that indexes the web so ChatGPT can surface and link to sites when it answers a question — including shopping questions. It is not a training crawler; it's how you get found. For a store, this is the one you want to be visible to.

ChatGPT-User — live, user-triggered fetches.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

This fires when a ChatGPT user (or an action they take) causes ChatGPT to fetch a specific URL in real time — for example, following a link to read your product page while answering a shopper. Block it and ChatGPT can't open your page even when a customer explicitly asks about you.

The mistake that makes stores invisible

Here's the trap. Most "how to block AI crawlers" advice treats all of these as one thing — block the AI bots to protect your content. Merchants paste a rule that disallows everything OpenAI, feel protected, and don't realise they've just told ChatGPT it may not surface them, link to them, or even open their page when a customer asks.

• Blocking GPTBot stops training use. That's a defensible choice.
• Blocking OAI-SearchBot removes you from ChatGPT's search and shopping results.
• Blocking ChatGPT-User stops ChatGPT from reading your page when a shopper is actively looking at you.

The last two are self-sabotage for any store that wants AI-driven traffic. You can decline to feed the training machine and still be fully visible in the answers — but only if you keep the three separate.

The robots.txt that keeps you visible

# ChatGPT search/shopping surfacing — keep this open
User-agent: OAI-SearchBot
Allow: /

# Live fetches when a shopper asks ChatGPT about you — keep this open
User-agent: ChatGPT-User
Allow: /

# Model training — allow or disallow, your call
User-agent: GPTBot
Allow: /

To opt out of training only, change the last block to Disallow: / and leave the first two as Allow: /. That's the entire decision.

How to check what you're actually doing

robots.txt is easy to get subtly wrong — a broad Disallow higher up the file, a wildcard user-agent, a leftover rule from an SEO plugin. The only way to know is to test it against each bot by name.

• Run your store through the robots.txt checker — it tells you, per crawler, whether GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot and the rest can actually reach you.
• Scan your store's AI readiness — crawler access is one of fourteen things that decide whether ChatGPT can read, and recommend, your products.

And if you're not sure how your products get into ChatGPT in the first place, start here: the two lanes that connect a Shopify store to ChatGPT.

Being crawlable is necessary, not sufficient — but being accidentally blocked is the one failure that makes everything else you do invisible. Check it first.

FoundGPT helps Shopify stores get read, recommended and sold by AI engines. Start with a free AI-readiness scan.