Published 2026-10-31

Google-Extended: The robots.txt Token That Controls Your Gemini Visibility

Google-Extended isn't a crawler with its own user-agent — it's a robots.txt control that decides whether Google can use your content for Gemini and its AI answers. Blocking it doesn't touch your Search ranking, but it can quietly opt you out of Google's AI.

Most of the "AI crawler" list is made of bots with user-agent strings. Google-Extended is the odd one out: it doesn't crawl anything. It's a robots.txt product token you use to tell Google whether it may use the content it already crawls for generative AI. Understanding the difference matters, because people block it expecting one thing and get another.

What Google-Extended actually controls

Google crawls your site with Googlebot, as always, for Search. Google-Extended is a separate switch layered on top that governs whether that content can be used to train and ground Google's generative models — Gemini and the Vertex AI generative APIs.

# Opt OUT of Google's generative-AI use (Gemini / Vertex)
User-agent: Google-Extended
Disallow: /

Key points:

There is no Google-Extended user-agent in your logs. It's a control token, not a crawler. You won't see it fetch pages.
Blocking it does not affect Google Search. Your ranking, indexing and crawl via Googlebot are untouched. This is the most common misunderstanding — people block Google-Extended thinking they're hiding from Google; they're only opting out of the AI use.
It governs Gemini and Vertex generative use. If you want Google's AI products to be able to draw on your content, you leave it open.

Should a store block it?

For most stores that want to be visible when shoppers ask Google's AI about products, blocking Google-Extended is counterproductive — you'd be opting your content out of exactly the generative surface you're trying to appear in, while gaining nothing on the Search side.

The honest reason to block it is a content/IP preference: you don't want your material used in Google's generative models. That's a legitimate choice — just make it deliberately, knowing it's separate from Search and separate from whether Googlebot crawls you.

The default

If you do nothing, Google-Extended is allowed — Google may use your crawlable content for its generative AI. So for a store chasing AI visibility, the "do nothing" default is usually the right one. The mistake is adding a blanket Disallow that accidentally sweeps it in.

Check what your robots.txt is really saying

Run your store through the AI crawler & robot checker — it shows, per bot and token, what your robots.txt actually allows, including a blanket rule you didn't mean to apply.
Scan your store's AI readiness — crawler and generative-AI access is one piece of whether engines can read and recommend your products.

FoundGPT helps Shopify stores get read, recommended and sold by AI engines across ChatGPT, Gemini, Google AI Mode and Perplexity. Start with a free AI-readiness scan.