Gemini Crawler Controls: Google-Extended, Googlebot and Firewalls

Google-Extended is a product token you name in robots.txt. Google's common crawlers documentation (updated July 2026) says disallowing it stops your content being used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. The same page says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

That gives you a clean split, as of October 2026:

• Google-Extended governs Gemini training and Gemini Apps grounding.
• Googlebot governs Google Search, which includes AI Overviews and AI Mode.

The practical meaning: you can refuse Gemini training without touching your Search presence. You can't use Google-Extended to leave AI Overviews. For the full definition, see what Google-Extended controls. For the side-by-side, see Google-Extended vs Googlebot.

Google-Extended does not crawl anything. Google's documentation describes it as a token with no separate HTTP user agent. Googlebot and Google's other crawlers fetch your pages, and Google then applies your Google-Extended rule to how that content is used.

Two consequences follow.

• You won't find Google-Extended in your server logs. Seeing Googlebot after you block Google-Extended is expected.
• Googlebot rules control far more than Gemini. Google says "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control" for AI features. There is no separate crawler for AI Overviews.

Googlebot also renders JavaScript, unlike the ChatGPT and Claude crawlers Vercel studied. Content that appears only after scripts run is still readable to Google, though not to most other AI crawlers.

To refuse Gemini training and Gemini Apps grounding while staying in Google Search, add a group for Google-Extended:

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

The Googlebot group is optional, since allowing is the default. Writing it out makes the intent visible to anyone who edits the file later. Do not add User-agent: * with Disallow: / thinking it only affects AI bots. A wildcard block also matches Googlebot.

If you want the same split across vendors, the block training, keep AI search playbook covers OpenAI, Anthropic and Perplexity tokens. Each vendor's tokens are independent of Google's.

robots.txt is a request about use. A firewall or CDN rule is a hard stop at the network. OpenAI documents the general mechanism: a host or CDN can block a crawler that robots.txt allows, often with a 403.

For Google, this cuts deep. Because Google-Extended has no user agent, a firewall can't target it. Any rule that blocks "Google bots" blocks Googlebot. That removes the page from Google's index, and Google states a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode.

Common ways this happens without anyone deciding it:

• A bot-management rule or challenge page that treats crawlers as suspicious.
• A security plugin with an aggressive bot list. See whether WordPress security plugins block AI crawlers.
• A CDN default change. Since 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites and existing free-tier customers, while Search stays allowed.

The fix is to express training preferences in robots.txt and keep the firewall open to verified Google crawlers. The AI Crawler Checker requests a page as each crawler and shows which ones your firewall turns away.

Google documents:

• Google-Extended's scope: Gemini training and Gemini Apps and Vertex AI grounding, with no effect on Search inclusion or ranking.
• That Googlebot rules are the robots.txt control for AI Overviews and AI Mode.
• On-page controls for AI feature display: noindex, nosnippet, max-snippet and data-nosnippet.
• A Search Console control for generative AI features, which is opt-out. Inclusion is the default, child properties inherit from the parent, and it is still reaching only some owners as of October 2026.

Google does not document, in the sources we hold:

• Whether blocking Google-Extended changes what a Gemini model already knows. Training effects land a model generation later, and no study measures them. See whether blocking Google-Extended changes Gemini's brand answers.
• How the Gemini API's developer-invoked Grounding with Google Search relates to your Google-Extended setting. It is a separate, separately billed retrieval path.
• Any ranking weight for being crawlable. Allowing Googlebot buys eligibility, not inclusion.

1. Read your robots.txt. Confirm Google-Extended has the rule you intend, and that no wildcard group blocks Googlebot.
2. Confirm the file itself is reachable. A firewall that blocks the robots.txt fetch hides every rule in it. The robots.txt reachability check tests this.
3. Test the network layer. Request a real page as Googlebot and look for 403s, challenges or CAPTCHAs.
4. Scan for snippet controls. A template-wide nosnippet or stray data-nosnippet removes content from AI Overviews display.
5. Check Search Console. If the generative AI control is visible, confirm the property and its parent are not opted out.

Everything above is in your control. Whether Gemini or AI Overviews then use your content is not.

Does blocking Google-Extended remove my site from AI Overviews? No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search. Their robots.txt control is Googlebot. To limit how your content appears there, Google documents noindex, nosnippet, max-snippet and data-nosnippet, plus a Search Console control that some owners can see as of October 2026. More detail is in [does blocking Google-Extended remove my site from AI Overviews](/questions/google-extended-ai-overviews).

Can I block Google-Extended in my firewall instead of robots.txt? Not precisely. Google-Extended has no user agent of its own, so a firewall has nothing distinct to match. Rules aimed at Google's crawlers hit Googlebot, which removes pages from Search and with it AI Overviews and AI Mode eligibility. Express a training opt-out in robots.txt and keep the network layer open to verified Google crawlers.

Will blocking Google-Extended change what Gemini says about my brand? Not this quarter, and possibly never in a measurable way. A model's built-in knowledge is frozen at its training cutoff, and any training effect lands a model generation later. Nothing published measures whether excluding content changes brand recall. Treat the Google-Extended decision as a question of how your content is used, not as a visibility lever.

Is there a separate robots.txt token for Google AI Mode? No. As of October 2026, Google offers no robots.txt token that targets AI Overviews or AI Mode alone. Googlebot rules apply to all of Google Search. Blocking Googlebot to leave AI Mode would also remove you from regular results. The Search Console generative AI control and the snippet directives are the narrower tools Google documents.

Does Googlebot read content loaded by JavaScript? Yes. Googlebot renders pages, so client-side content can reach Google. Most other AI crawlers do not. Vercel found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. If your key facts only appear after scripts run, Google may see them while ChatGPT and Claude search miss them. Server-side rendering closes that gap.