AI Overviews and robots.txt: Googlebot Is the Control, Not a Separate Crawler

Google does not publish a separate crawler for AI Overviews or AI Mode. Its position is that AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot are the control.

That means you can't write a rule that keeps a page in classic results but out of AI Overviews at the robots.txt level. The two are fetched by the same crawler, Googlebot, and stored in the same index.

Google's AI features documentation sets the eligibility bar: to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. Google adds that there are no additional technical requirements. It also says indexing and serving isn't guaranteed. Eligibility gets you considered, nothing more.

What this means for you: if you block Googlebot, you leave Search entirely. If you allow it, you are eligible for AI features unless you use one of the narrower controls below. For the full picture on blocked pages, see whether a robots.txt-blocked page can appear in AI Overviews.

Google-Extended is the token most people reach for when they want to block Google's AI. It does something real, just not what many expect.

According to Google's common crawlers documentation (updated July 2026), disallowing Google-Extended stops content being used for training and grounding in Gemini Apps and Vertex AI. The same page says it does not impact a site's inclusion in Google Search, nor is it used as a ranking signal.

User-agent: Google-Extended
Disallow: /

This rule leaves AI Overviews and AI Mode untouched. Google-Extended is not a crawler with its own fetches either. It is a control token read against content Google's existing crawlers collect.

The split matters if your goal is to limit model training while staying visible in Google's answers. Our comparison of Google-Extended vs Googlebot walks through each surface, and Gemini crawler controls covers the Gemini app side.

Google documents on-page controls that apply to AI features because they apply to Search snippets. As of October 2026, these are the tools Google names:

• noindex removes the page from the index, and so from AI features.
• nosnippet stops any snippet, which also blocks use as a supporting link.
• max-snippet:0 has the same effect, and a larger number caps snippet length.
• data-nosnippet on an HTML element excludes that section from snippets.

A meta tag looks like this:

<meta name="robots" content="max-snippet:0">

These are suppression tools only. No markup makes a section more likely to be used in an AI answer. The common failure is accidental: a template applies nosnippet or data-nosnippet site-wide, often to hide a banner, and the content you wanted quoted disappears from AI answers.

The next move: search your templates and headers for these four directives and confirm each one is intentional.

Google also offers a generative AI control in Search Console. It covers AI Overviews, AI Mode and generative AI features in Google Discover, and Google says it expects to update that list over time.

Three details matter:

1. Inclusion is the default control for all properties. You do not need to opt in to be eligible.
2. Child properties default to inheriting the control from their parent. A setting on a parent property can silently apply to sections beneath it.
3. As of October 2026, Google's help page still describes the control as rolling out to a subset of website owners, so some properties show no toggle yet.

Google states the setting isn't used as a ranking or inclusion signal for other parts of Search. It also notes that opting out of display does not stop content being used to power Search as a whole, such as understanding query language.

No crawler can see this setting. If a property is excluded here, nothing else you fix will put it into AI Overviews. Check it in Search Console, on every parent and child property.

Googlebot behaves differently from most AI crawlers in two ways.

First, it renders JavaScript. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so client-rendered content is invisible to them. Googlebot does execute JavaScript, per Google's JavaScript SEO documentation. A client-rendered page can be readable to Google's AI features while staying blank to other engines. That is not a reason to skip server-side rendering, because the other crawlers still matter.

Second, Googlebot has a payload ceiling. Google's March 2026 crawler post states it fetches up to 2MB for any individual URL, excluding PDFs, which have a 64MB limit. Bytes past the threshold are not fetched, rendered or indexed. Google also says the vast majority of the web will never hit this limit.

The practical check: measure raw HTML size on your heaviest templates. Act only if a page actually exceeds 2MB.

robots.txt is a request, not a gate. A web application firewall, CDN bot rule or IP filter can refuse Googlebot even when robots.txt allows it.

Because AI Overviews and AI Mode draw on the same index as classic Search, a block at the network layer has no partial outcome. Google cannot index what it cannot fetch, and an unindexed page is not eligible for any Google surface. The failure is silent: robots.txt looks fine and nothing in your own dashboards flags it.

CDN defaults change without you editing anything. From 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed by default. Check how your provider classifies each bot.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. For per-bot rules beyond Google, see the robots.txt guide for AI crawlers.

Can I block AI Overviews without leaving Google Search? Not through robots.txt. The documented options are snippet controls and the Search Console generative AI setting. Using nosnippet or max-snippet:0 on a page stops it appearing as a supporting link, but it also removes the classic text snippet. The Search Console control targets AI Overviews, AI Mode and Discover generative features, though as of October 2026 not every property has it. Neither option guarantees anything about what Google's models already learned elsewhere.

Does blocking Google-Extended remove my site from AI Overviews? No. Google's common crawlers documentation says Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. It governs training and grounding for Gemini Apps and Vertex AI. AI Overviews and AI Mode run on Googlebot's index, so a Google-Extended disallow leaves them unchanged. If you want to limit training while keeping AI search visibility, see [how to block AI training but stay in AI search](/playbooks/block-ai-training-keep-ai-search).

Do I need special markup or an llms.txt file for AI Overviews? Google says no. Its generative AI optimization guide states you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features, because Google Search ignores them. The ai-features documentation says there are no additional technical requirements beyond being indexed and snippet-eligible. Existing structured data does no harm, but it is not a lever for AI Overviews.

Will Google penalize me for blocking other AI crawlers? Rules for GPTBot, ClaudeBot or PerplexityBot have no documented effect on Google. Google says the Google-Extended setting is not a ranking signal, and the Search Console generative AI control isn't used as a ranking signal for other parts of Search. The one block that does remove you from Google is a block on Googlebot itself, whether in robots.txt or at the firewall.