Crawl payload limit: definition and Googlebot's 2MB cap

A crawl payload limit is a ceiling on how many bytes a crawler downloads from one URL. Once it reaches that ceiling, it stops reading.

The best-documented example is Googlebot. In a March 31, 2026 post on the Google Search Central blog ("Inside Googlebot: demystifying crawling, fetching, and the bytes we process"), Gary Illyes wrote that Googlebot currently fetches up to 2MB for any individual URL, excluding PDFs. For PDFs, the limit is 64MB. The same post gives a 15MB default for crawlers that don't specify their own limit.

This entry is a short definition. For the full procedure, see the guide on checking pages against crawler payload limits.

Googlebot counts bytes as it downloads the raw response for a URL. Per Google's post, any bytes after the 2MB threshold are "entirely ignored. They aren't fetched, they aren't rendered, and they aren't indexed."

Three details matter:

• It is per URL, not per site. A 1.5MB page and a 1.8MB page on the same domain are both fine.
• Sub-resources have separate counters. Linked JavaScript, CSS and images are fetched as their own URLs, each with its own limit.
• PDFs are treated differently. They get 64MB. If you publish long documents as PDFs, the PDF question covers how AI engines handle them.

The number that counts is the size of the raw HTML response. To check it, measure the byte size of each URL as served. Then flag only pages over 2MB. The free page size check does this for a single page.

As of October 2026, Google says a page must be indexed and eligible to show with a snippet before it can appear as a supporting link in AI Overviews or AI Mode. Content past the 2MB cap is never indexed. So a fact sitting below the cutoff on an oversized page cannot support a Google AI answer, however relevant it is.

That is the whole effect. It is a block, not a ranking factor. Google's own framing is blunt: for the vast majority of the web, a 2MB HTML payload is massive, and you will never hit this limit.

What this means in practice:

1. If your pages are ordinary marketing or product pages, you can likely set this aside.
2. If you run very long pages (huge catalogs, inlined data, giant single-page docs), measure them. The page too large for Google answer walks through the symptoms.
3. If a page is over the cap, move load-bearing content earlier or split the page so the facts you care about fall inside the first 2MB.

For non-Google AI crawlers, a more common problem is rendering. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. A small page whose content only appears after JavaScript runs is worse off than a large server-rendered one. Start with server-side rendering and a raw HTML vs rendered DOM check before worrying about bytes.

• "Smaller pages get cited more." No evidence supports this. Getting under the cap only matters if you were over it.
• "2MB applies to every AI crawler." The 2MB figure is Googlebot's. It is not a documented ceiling for ChatGPT, Claude or Perplexity retrieval.
• "Images and scripts count toward the 2MB." Sub-resources are separate URLs with separate counters.
• "The whole site shares one budget." The limit applies to each URL on its own.
• "Google warns you when a page is cut off." Nothing in Google's post describes a warning. You have to measure the page yourself.

• Googlebot: the crawler the 2MB limit applies to, and how it feeds Google's AI features.
• Server-side rendering: putting content in the initial HTML so crawlers that don't run JavaScript can read it.
• Large robots.txt files: a related question about size limits on the robots.txt file itself.