Google-Extended vs Googlebot: What Each One Controls

Googlebot is the crawler that fetches pages for Google Search. As of October 2026, that one crawler feeds classic results, AI Overviews and AI Mode. Google's crawler documentation puts it plainly: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control."

That has a direct consequence. Google's AI features page says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. It adds that there are "no additional technical requirements." So Googlebot access is the gate. If Googlebot can't fetch and index a page, Google's AI surfaces can't cite it.

Two things Googlebot does differently from most AI crawlers:

• It renders JavaScript, so client-side content that other AI crawlers miss can still reach Google.
• It is not a separate AI bot. There is no separate crawler for AI Overviews to allow or block.

The next move: treat Googlebot access as non-negotiable unless you want out of Google Search entirely.

Google-Extended is a product token you use in robots.txt. It doesn't crawl on its own. Google's common crawlers documentation (updated July 2026) describes it as the control over whether your content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. The same page says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

That makes Google-Extended a training crawler control with one addition: it also covers grounding in Gemini Apps and Vertex. Grounding means the model pulls in live web content to answer a question. Blocking the token stops your content being used for those purposes on those products.

A minimal rule looks like this:

User-agent: Google-Extended
Disallow: /

Your Googlebot rules stay untouched, so Search is unaffected. For the full definition and edge cases, see the Google-Extended glossary entry or our notes on Google's common crawlers doc.

One scope note. The Gemini API's developer feature "Grounding with Google Search" is a separate, separately billed path that grounds on Google Search results. Those results come from what Googlebot indexed.

The pattern: Googlebot is a Search decision. Google-Extended is a Gemini decision. They are independent, and you can set each one without touching the other.

Blocking Googlebot

• Does: removes your pages from Google's index over time, which removes them from classic results, AI Overviews and AI Mode.
• Does not: give you a way to stay in Search while leaving AI features. Googlebot is all or nothing for Search.

Blocking Google-Extended

• Does: tell Google not to use your content to train Gemini or to ground Gemini Apps and Vertex answers.
• Does not: remove you from AI Overviews or AI Mode. That question gets asked often, and the answer is covered in does blocking Google-Extended remove my site from AI Overviews.
• Does not: change your Search rankings, per Google.
• Does not: erase what a model already learned. Training effects land a model generation later, if at all. Nothing published measures whether excluding a site from training changes how a model describes that brand. The honest label is unmeasured, not zero.

Allowing Googlebot also isn't the same as allowing Gemini training. Those are separate switches, explained in does allowing Googlebot let Google train Gemini.

One more gate sits outside robots.txt. A firewall or CDN can block a crawler that robots.txt allows, often with a 403. If you change these rules, confirm your network layer agrees. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.

Since no robots.txt token separates AI Overviews from Search, Google documents page-level controls instead. As of October 2026 these are the on-page levers:

• noindex removes the page from Search, and with it from AI features.
• nosnippet stops Google showing a snippet, which also makes the page ineligible as an AI Overviews or AI Mode supporting link.
• max-snippet:0 has the same effect as nosnippet.
• data-nosnippet on an HTML element keeps that section out of snippets and AI answers while the rest of the page stays eligible. Bing supports it too.

Google is also rolling out a Search Console control for its generative AI features. It is opt-out, inclusion is the default, and it covers AI Overviews, AI Mode and generative AI in Discover. Child properties inherit the parent's setting, so a property can be excluded without anyone touching it directly. Google says opting out does not stop content being used to help power Search as a whole. More on that in if I opt out of Google's AI features, can Google still use my content.

The risk runs the other way too. Accidental nosnippet or data-nosnippet tags inherited from a template can quietly remove content from AI answers. Audit them before you add new ones.

Pick the control that matches the outcome you want. Say you run a B2B software company and want to appear in AI Overviews but keep your docs out of Gemini training. Here is how each goal maps:

1. You want to be in Google Search and its AI features, and you don't mind Gemini training. Allow Googlebot. Leave Google-Extended allowed or unset.
2. You want Search and AI Overviews, but not Gemini training or Gemini Apps grounding. Allow Googlebot. Disallow Google-Extended. This is the common setup, and the block AI training but stay in AI search playbook extends it to other vendors' bots.
3. You want to keep specific content out of AI Overviews and AI Mode. Keep Googlebot allowed. Use nosnippet, max-snippet or data-nosnippet on that content, or the Search Console control if your property has it.
4. You want out of Google entirely. Disallow Googlebot or use noindex. Expect to lose classic results and AI features together.

Whatever you choose, none of these settings promises a citation. Allowing Googlebot buys eligibility, not inclusion. Google's own page says indexing and serving "isn't guaranteed."

Does Google penalize sites that block Google-Extended? Google says Google-Extended is not used as a ranking signal and does not affect inclusion in Search. So blocking it should not change how your pages rank in classic results or whether they can appear in AI Overviews. What it does change is whether Gemini can train on or ground with your content in Gemini Apps and Vertex AI. For the wider question across other AI bots, see [does Google penalize sites that block AI crawlers](/questions/google-penalize-blocking-ai-crawlers).

Is Google-Extended a separate crawler I'll see in my logs? No. Google describes Google-Extended as a product token, not a separate user agent that fetches pages. Google's existing crawlers do the fetching, and the token tells Google how it may use what they collect. That's why you won't see Google-Extended requests in server logs. You control it only through robots.txt rules that name the token.

Will blocking Google-Extended change what Gemini says about my brand? It stops new use of your content for Gemini training and for grounding in Gemini Apps and Vertex. It doesn't remove what an existing model already learned, and nobody has published a measurement of how training exclusion affects brand answers later. If Gemini gets a fact about you wrong, the more direct fix is making sure the correct fact is stated clearly on pages Google can fetch.

Can I block AI Mode but keep AI Overviews? Not with robots.txt. Googlebot is the single control for Search, and Google says AI Overviews and AI Mode share the same eligibility rule: indexed and snippet-eligible. Snippet controls such as nosnippet and data-nosnippet apply to both. The Search Console generative AI control, where available, covers both surfaces plus Discover. Google does say the two surfaces may use different models and return different links.

Does Google-Extended affect Google-CloudVertexBot? They are different things. Google-CloudVertexBot fetches only when a site owner sets up a Vertex AI agent that uses their own content, and Google says it has no effect on Search or other products. Google-Extended is the broader opt-out for Gemini training and grounding. Most sites only need to decide on Googlebot and Google-Extended.