Googlebot and AI: Why You Must Never Block It (and What Google-Extended Does Instead)
Googlebot powers Google Search — and Google's AI surfaces draw on that same index. Blocking it makes you invisible to Google entirely. Here's the user-agent, how it relates to Google's AI, and the one token to use if you only want to opt out of the generative part.
Googlebot powers Google Search — and Google's AI surfaces draw on that same index. Blocking it makes you invisible to Google entirely. Here's the user-agent, how it relates to Google's AI, and the one token you'd use if you only want to opt out of the generative part.
Of all the crawlers in the AI conversation, Googlebot is the one you absolutely must not block — because it's not just an AI crawler, it's the crawler for Google Search, and Google's AI answers are built on the same foundation.
What Googlebot is
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot crawls and indexes the web for Google Search. Crucially, Google's AI surfaces — AI Overviews and AI Mode — are grounded in that Search index. There is no separate "Google AI crawler" fetching your pages for those; they use what Googlebot already indexed.
So being readable by Googlebot is the prerequisite for appearing in both ordinary Google results and Google's AI answers.
The mistake people make
Some stores, trying to "opt out of AI," add a rule blocking Google. That doesn't opt you out of AI — it opts you out of Google Search, which is almost always a catastrophic mistake for a store.
The control you actually want for AI is Google-Extended — a robots.txt token that governs whether Google may use your content for its generative models (Gemini, Vertex), without affecting Search or Googlebot at all. If your goal is "stay in Search, opt out of generative training," that's the lever — not blocking Googlebot.
The robots.txt
# Keep Googlebot open — this is Search AND the basis of Google's AI answers
User-agent: Googlebot
Allow: /
Leave Googlebot allowed. Make any AI-specific choice with Google-Extended, separately.
Check it by name
- Run your store through the AI crawler & robot checker — confirm Googlebot (and every AI bot) can reach you.
- Scan your store's AI readiness — crawler access is one piece of whether engines can read and recommend your products.
See also: Google-Extended · GPTBot · OAI-SearchBot · the complete AI crawler & user-agent list.
FoundGPT helps Shopify stores get read, recommended and sold by AI engines across ChatGPT, Gemini, Google AI Mode, Perplexity and Claude. Start with a free AI-readiness scan.