Google's Common Crawlers Documentation and Google-Extended, Digested

Google publishes a reference page listing its common crawlers and the product tokens site owners can name in robots.txt. The version digested here is dated 2026-07-14. It sits alongside Google's other site-owner documentation, such as Google's AI features documentation and the generative AI optimization guide.

The page matters because two tokens on it get confused constantly: Googlebot and Google-Extended. Many sites block one when they meant the other. The result is either a site that vanishes from Search or a site that thinks it has left AI answers when it has not. For a side-by-side view, see Google-Extended vs Googlebot.

This is not a study. Nobody measured anything. It is Google describing its own systems, which makes it first-party documentation rather than evidence from an experiment.

That distinction cuts two ways:

• It is the best available source for what each Google token governs. No outside party can see inside Google's pipeline.
• It is an assertion, not a measurement. Statements like "not used as a ranking signal" cannot be tested from outside. Treat them as Google's stated policy as of the page date.

The 99Visibility evidence base recorded the page's wording on 2026-08-22. Crawler documentation changes, so re-read the live page before acting on a specific sentence.

The document draws clear lines between tokens. The table below restates each one as the decision it actually represents.

A Google-Extended block is short:

User-agent: Google-Extended
Disallow: /

That rule leaves Googlebot untouched. Your Search listings and AI Overviews eligibility stay as they were. If you want more detail on the token itself, the Google-Extended definition covers it.

This is the part most robots.txt advice gets wrong. As of October 2026, there is no token that removes a site from AI Overviews or AI Mode while keeping it in regular Search. Google's stated reason: AI is built into Search, so robots.txt directives for Googlebot are the control.

Google's AI features documentation says a page must be indexed and eligible to be shown in Search with a snippet to appear as a supporting link. It adds that there are no additional technical requirements. So eligibility for AI Overviews follows normal Search eligibility.

The controls that do act on Google's AI surfaces are:

• Googlebot rules in robots.txt. Blocking Googlebot removes you from Search entirely, not just AI features.
• Snippet controls. noindex, nosnippet, max-snippet and data-nosnippet are the documented on-page controls for limiting display.
• The Search Console generative AI control. Inclusion is the default for all properties, so it is an opt-out. It covers AI Overviews, AI Mode and generative AI features in Discover. Child properties inherit from the parent by default, and it is still rolling out to a subset of owners.

Google also says opting out of display does not stop content being used to power Search as a whole, such as understanding the language of queries. More on this in AI Overviews and Googlebot.

The page defines scope. It says nothing about outcomes.

• It does not show that allowing Google-Extended helps you. Nothing in it says allowing the token improves how Gemini describes your brand or how often it cites you.
• It does not measure what blocking costs. Training exclusion affects what future models know, and no study in the evidence base measures that effect in either direction. The honest label is unmeasured.
• It does not cover every Google AI path. The Gemini API's developer-invoked Grounding with Google Search is a separate, separately billed retrieval path.
• It does not prove Google honours every directive in practice. Compliance is stated, not externally verified.
• It does not cover network blocks. A firewall or CDN can turn away a crawler that robots.txt allows. That gate sits outside this document entirely.

Make each choice on its own terms. Here is the order that avoids the common mistakes.

1. Decide on training separately from visibility. If you don't want Gemini Apps or Vertex AI training on or grounding in your content, block Google-Extended. Know that this does not remove you from AI Overviews. The block AI training but stay in AI search playbook walks through this across vendors.
2. Never block Googlebot to escape AI Overviews. You would lose Search with it.
3. Audit your snippet directives. Look for nosnippet, max-snippet:0 and data-nosnippet inherited from a template. Each one can hide content from AI answers without anyone meaning it to.
4. Check the Search Console generative AI control on every property, including child properties that inherit from a parent. It is invisible from outside.
5. Read your actual robots.txt per token. The robots.txt Checker for AI Crawlers shows how each named bot resolves, so you can confirm Google-Extended and Googlebot are set the way you intended.
6. Apply the same logic to other vendors. OpenAI separates training and search bots in a similar way. See OpenAI's crawler documentation.

None of these steps guarantees you appear in an AI answer. They remove the blocks you control.