Checking Pages Against Crawler Payload Limits: the Googlebot 2MB Limit

In a March 2026 Search Central post, Google's Gary Illyes stated that Googlebot currently fetches up to 2MB for any individual URL, excluding PDFs. For PDF files the limit is 64MB. Bytes after the 2MB threshold are, in Google's words, entirely ignored: they aren't fetched, they aren't rendered, and they aren't indexed.

Three details matter for anyone running a check:

• It is per URL. The HTML document has its own counter. Images, stylesheets and script files are sub-resources with separate counters, so a heavy image does not eat into your HTML budget.
• The 15MB figure is a default. The same post says any other crawler that doesn't specify a limit defaults to 15MB. That statement covers Google's crawlers. It is not a published limit for GPTBot, ClaudeBot or PerplexityBot, and none of those operators document one in the evidence we hold.
• It rarely bites. Google's own framing is that for the vast majority of the web, a 2MB HTML payload is massive and you will never hit this limit.

So this is a check you run to rule something out. For the full definition, see the crawl payload limit glossary entry.

AI visibility has four stages: what the model already knows (parametric), whether a crawler can fetch and store the page (access), whether the page is pulled in for a query (retrieval), and what the model uses and cites (generation). The payload limit is an access problem. You control it completely.

On Google, the link to AI features is direct. Google's AI features documentation says a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in AI Overviews or AI Mode. Text past the 2MB cutoff is never indexed, so it cannot be eligible. If your pricing table or product specs sit after a huge inline data block, they may simply not exist as far as Google is concerned.

For non-Google AI crawlers, the bigger issue is usually rendering, not size. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they only see what is in the raw HTML. That is why size and rendering checks belong together. Pair this page with the raw HTML vs rendered DOM check.

What fixing a size problem does not do: it doesn't make a page more likely to be retrieved or cited. It removes a block. Nobody outside the engines controls what happens after that.

You need the size of the HTML document as the server sends it, before any JavaScript runs. Follow these steps.

1. List your heaviest templates. Long category or listing pages, documentation hubs, pages with big inline data, and any page built by a page builder that inlines styles. Check one URL per template, not every URL.
2. Fetch the raw HTML with one request. curl doesn't execute JavaScript and, by default, doesn't ask for compression, so it reports the uncompressed body size:

curl -s -L -o page.html -w "body: %{size_download} bytes\nheaders: %{size_header} bytes\n" https://example.com/your-page

3. Record the body and header bytes together. Google's evidence on this limit doesn't state whether it is measured before or after compression. Using the uncompressed figure gives you the larger number, so your check errs on the safe side.
4. Compare against the thresholds. 2MB for Googlebot on HTML. 64MB for PDFs. 15MB for Google crawlers that specify no limit.
5. If a page is over, find where the bytes are. Open page.html and search for long inline script blocks, embedded JSON state, base64-encoded images, and inline SVG. These are the usual culprits.
6. Find where your key facts sit. If the page is over the limit, check whether your prices, definitions and product names appear before or after the 2MB mark. Content past the mark is what you lose.

If you'd rather not use a terminal, the free page size crawler limit check runs the same measurement for a single URL.

Most results will be boring. That is the good outcome.

That last row catches people out. A firewall or CDN can block a crawler that robots.txt allows, and the HTML you measure will be tiny because it is a block page. Confirm reachability with the free AI crawler checker before trusting any size number.

Do not report "page is 1.2MB" as a problem. Google's own statement is that most pages never come near the limit. Flag only pages that exceed it.

Run through this list for each template you check.

• I measured the raw HTML, not the rendered DOM or the total page weight with images.
• I recorded body bytes and header bytes from a single request.
• The response was a 200 with real page content, not a redirect loop, challenge page or 403.
• HTML pages are under 2MB, or I have flagged the ones that aren't.
• PDFs I care about are under 64MB.
• For any oversized page, I know which inline block is causing it.
• For any oversized page, I know whether key facts fall before or after the 2MB mark.
• I checked that the content that matters is in the raw HTML at all, not only after JavaScript runs.
• I did not flag pages under the limit as a size problem.
• I noted the date of the check, so I can compare after the next release.

The fix is almost always about inline weight, not about writing less.

Move inline payloads out of the HTML

• Base64 images: serve them as normal image files. Sub-resources have their own counters.
• Large inline scripts and styles: move them into external files.
• Embedded data blobs: frameworks sometimes inline the full data set a page uses for hydration. Trim it to what the page needs, or load the rest separately.
• Endless listings: split very long lists across paginated URLs, each well under the limit.

Put the important facts early

If you can't shrink a page quickly, at least make sure the content that matters appears before any large block. That way the first 2MB contains your facts, not your scripts.

Don't trade one problem for another

Moving content behind client-side JavaScript to cut HTML size can hide it from AI crawlers that don't execute JavaScript. Keep load-bearing text in the server-rendered HTML. The free JavaScript rendering check confirms nothing disappeared after the change.

After the fix, re-run the measurement and confirm the page returns 200 with the facts present.

Page size only changes when your templates or data change, so tie the check to those events rather than a calendar alone.

• After any redesign, framework upgrade or new page builder. These are the changes most likely to add inline weight.
• When a listing or catalog page grows. A page that lists every product can cross the line as the catalog expands.
• Quarterly spot check. Re-measure one URL per heavy template.
• When Google updates its crawling documentation. The limits above are as stated in Google's March 2026 post and current as of October 2026. Google describes them as what Googlebot "currently" fetches, so check the source again if you see reports of a change.

If every template has been comfortably under the limit for several checks, you can drop to a check only after major releases. Spend the saved time on access and rendering, where problems are far more common.