Free Check: Is Your Page Too Big for Crawlers?

This check answers one question: does the HTML response for a URL fit inside the bytes a crawler will actually read?

Google set out the numbers in a Search Central post dated March 31, 2026 ("Inside Googlebot: demystifying crawling, fetching, and the bytes we process"):

• Googlebot fetches up to 2MB for any individual URL, excluding PDFs.
• For PDFs, the limit is 64MB.
• Bytes after the threshold are ignored. Google says they aren't fetched, rendered or indexed.
• For any other crawler that doesn't specify a limit, the default is 15MB.

The cap is per URL. Sub-resources such as images, CSS and JavaScript files have separate counters. A page with 10MB of images can still have a tiny HTML document.

Why this matters for AI search: Google states a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. Text past the 2MB mark is not indexed, so it can't support anything. For a fuller definition, see the crawl payload limit entry.

The honest framing: Google itself says a 2MB HTML payload is massive for most of the web. This is a rare failure, not a routine one. The check is cheap, so run it once and move on unless the number is close.

You need a terminal or a desktop browser. Both methods measure the HTML document only, which is what the limit applies to.

Option A: curl (most precise)

1. Open a terminal on macOS, Linux or Windows (curl ships with recent Windows versions).
2. Save the headers and the decompressed body to files:

curl -s --compressed -D headers.txt -o page.html https://example.com/your-page

3. Count the bytes in both:

wc -c headers.txt page.html

4. Add the two numbers. That total is your raw HTML payload including headers.
5. Repeat for your heaviest templates: long category pages, documentation hubs, pages with big inline data.

The --compressed flag makes curl decompress the response, so you measure the larger, uncompressed size. If that number is under 2MB, you're under either way.

Option B: Chrome DevTools (no terminal)

1. Open the page in Chrome and press F12 (Cmd+Option+I on Mac).
2. Go to the Network tab and pick the Doc filter.
3. Reload the page.
4. Find the first row, the HTML document itself.
5. Hover the Size column. Chrome shows bytes transferred over the network and the resource size. Use the resource size, the decompressed figure.

Ignore the page's total weight at the bottom of the panel. That figure adds up every sub-resource, and the limit doesn't. The crawler payload limit playbook covers checking many URLs at once.

Compare your number with the thresholds:

If a page is over 2MB, the question is what lives past the cutoff. A product description at the top is safe. Pricing, FAQs or a footer link block near the bottom may not be read. Our answer on whether Google can read a large page in full walks through that.

Common causes of oversized HTML, when it happens:

• Large inline JSON blobs used to hydrate a JavaScript framework.
• Images or fonts embedded inline as base64.
• Big inline SVGs repeated across a page.
• Every product, variant or comment printed into one document.

The fix is usually moving that data into separate files, which have their own counters, or paginating the content.

Be clear about what a byte count can and can't tell you.

• It's a Googlebot limit. The 2MB figure is Google's. The 15MB figure is Google's default for crawlers that don't state one. Neither is a published limit for OpenAI, Anthropic or Perplexity crawlers, and the evidence we hold has none for them.
• Passing it confers nothing. A page under 2MB is eligible to be read in full. That isn't a ranking, a citation or a guarantee of indexing.
• Your server may answer bots differently. Some sites serve different HTML to crawlers, and some firewalls block them outright. Your browser's response may not match what a crawler receives. The free AI crawler checker covers the reachability side.
• Size isn't visibility. A small page can still hide its key content behind JavaScript. As Vercel's crawler study reported, ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so text that appears only after scripts run may never be seen. Googlebot renders JavaScript; most AI crawlers, as of October 2026, don't.

If your page is small but your important content is missing from the raw HTML, the problem is rendering, not size.

For one URL, the curl command above is the fastest free tool there is. For a site, script it: loop over your sitemap URLs, run the same curl and wc -c, and sort by size. Only the top few results need a human look.

While you have the raw HTML file open, use it twice. Search it for your price, your product name and your one-line description. If they're missing, a crawler that doesn't run JavaScript won't see them either. The raw HTML vs rendered DOM check shows how to diff the two properly.

That second check is the one more likely to find a real problem. Rendering gaps are common. Pages over 2MB are not.

Does the 2MB limit include images and scripts? No. Google's March 2026 post says the limit applies per URL, and sub-resources have separate counters. Images, CSS and JavaScript files are each fetched as their own URL. Only the bytes of the HTML document itself count toward its 2MB. So a page with heavy images can have a small HTML payload. Measure the document row, not the total page weight your browser reports.

What happens to content after the 2MB mark? Google says bytes past the 2MB threshold are entirely ignored: not fetched, not rendered and not indexed. The first 2MB is processed as normal. Anything that lives only in the cut-off part can't be indexed, and Google states AI Overviews and AI Mode only use indexed pages eligible for a snippet. Read more on [whether AI reads your whole page](/questions/does-ai-read-whole-page).

Do ChatGPT, Claude or Perplexity crawlers have the same limit? There's no published figure for them in the evidence we rely on. The 2MB cap is Googlebot's. Google's post gives 15MB as a default for crawlers that don't specify a limit, but that isn't a statement about other companies' bots. Don't assume a number for them. Keeping HTML well under 2MB covers Google and leaves plenty of headroom elsewhere.

Are PDFs treated differently? Yes. Google's post sets a 64MB limit for PDF files, far above the 2MB HTML cap. Most PDFs fit comfortably. If you publish very large PDFs, such as scanned reports or image-heavy catalogues, check the file size directly. The same rule applies: bytes past the limit are ignored, so put the important text early.

Should I shrink my pages to improve AI visibility? Only if a page actually exceeds the limit. Google says most of the web never hits 2MB, and trimming a 300KB page further changes nothing about whether it's read. Smaller HTML isn't a ranking or citation lever. It just removes a block when one exists. Spend the time checking crawler access and whether key content is in the raw HTML.