Googlebot: definition and its role in Google's AI features

Googlebot is the user agent Google uses to crawl the web for Google Search. It requests your pages, renders them, and passes what it finds to Google's index. Everything Google shows in Search starts there, including its generative features.

That makes Googlebot different from most crawlers named in AI search discussions. OpenAI, Anthropic and Perplexity run separate bots for training and for search. Google does not split Search that way. If you want the full picture of how this plays out, read why AI Overviews have no separate crawler.

Googlebot fetches the raw HTML, then renders the page in a browser environment so JavaScript-generated content can be indexed. Google documents this rendering step in its JavaScript SEO basics. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them, and it named Googlebot as an exception.

Two practical limits apply:

• Payload size. Google's March 2026 crawler post says Googlebot fetches up to 2MB per URL (64MB for PDFs) and ignores bytes past that point. Google also says most sites never hit this. See whether your page is too large for Google.
• Rendering is not a guarantee. Server-rendered HTML is still the safer default, mostly because other AI crawlers read only the raw HTML. The trade-offs are covered in whether AI Overviews need server-side rendering.

Google's AI features documentation says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. It adds: "There are no additional technical requirements." Eligibility is still not inclusion. Google also says indexing and serving isn't guaranteed.

So your levers on Google are access and suppression, not optimization:

1. Let Googlebot crawl. A Googlebot disallow in robots.txt removes the page from Search and, with it, from AI features.
2. Check snippet controls. noindex, nosnippet, max-snippet:0 and data-nosnippet keep content out of AI answers. They are often inherited from a template by accident. See how noindex and nosnippet affect AI Overviews.
3. Check Search Console. Google's generative AI inclusion control is opt-out and on by default, and child properties inherit it from the parent. It is rolling out to a subset of owners and cannot be seen from outside.

If you want to keep a page in Search but out of AI answers, use snippet controls:

<meta name="robots" content="nosnippet">

What you cannot control is whether an AI Overview triggers for a query, or which eligible pages Google picks.

• "There's a robots.txt token for AI Overviews." There isn't. Google says robots.txt directives for Googlebot are the control because AI is built into Search.
• "Blocking Google-Extended removes me from AI Overviews." It doesn't. Google's common crawlers doc says Google-Extended governs Gemini Apps and Vertex training and grounding, and does not affect Search inclusion or ranking. The detail is in Google's common crawlers doc and Google-Extended.
• "Allowing Googlebot means Google trains Gemini on my site." Training use is a separate setting. See whether allowing Googlebot lets Google train Gemini.
• "Special markup or an llms.txt file helps Googlebot feed AI features." Google's AI optimization guide says Search doesn't use AI text files or special markup, and structured data isn't required for generative AI search.

• Google-Extended: the robots.txt token for Gemini training and grounding, separate from Search.
• Supporting link: the cited source link shown in AI Overviews and AI Mode.
• Claude-SearchBot: Anthropic's search crawler. Unlike Googlebot, it does not execute JavaScript, so it reads only server-rendered content.
• Snippet controls: nosnippet, max-snippet and data-nosnippet, the on-page controls for AI answer display.