Raw HTML vs Rendered DOM: A Server Side Rendering Check for AI Crawlers

Most AI crawlers read the HTML your server sends and stop there. Vercel's study of AI crawler traffic (published December 2024, updated February 2025) found that ChatGPT and Claude crawlers do fetch JavaScript files (11.50% and 23.84% of their requests) but "don't execute them. They can't read client-side rendered content." You can read our summary of Vercel's AI crawler study for the detail.

Independent re-testing as of June 2026 found the same for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider. None of them rendered JavaScript. Googlebot and Applebot are the exceptions: both render pages.

Two caveats are worth knowing. Vercel's sample came from one month on nextjs.org, and Vercel sells server rendering. The finding has since replicated, but the incentive is real.

What this means: the raw HTML is a fair stand-in for what a non-rendering crawler receives. The rendered DOM is what a human, Googlebot or Applebot sees. The difference between the two is your exposure. If you need the background terms first, see server-side rendering.

This takes about fifteen minutes per page once you have the tools set up.

1. Pick the pages that matter. Start with pricing, your main product or service pages, the about page and your top three landing pages. These carry the facts an AI answer would quote.
2. Fetch the raw HTML. Request the page with no JavaScript execution and save the response. A plain request works:

curl -sL -o raw.html https://example.com/pricing

3. Render the same URL. Load it in a headless browser (Puppeteer, Playwright or headless Chrome), wait for network activity to settle, and save the final DOM as rendered.html.
4. Extract visible text from both. Strip script, style and noscript blocks, then keep the text nodes. Any HTML-to-text library will do. Use the same extractor on both files so the comparison is fair.
5. Diff the two text files. Any diff tool works. You are looking for lines present in the rendered text and missing from the raw text.
6. Search for your load-bearing facts. Grep the raw text for your price, product names, H1 and H2 headings, and any one-line definition of what you do. Note each one as present or missing.
7. Record the result. Write down the share of primary content that appears only after rendering, plus the list of missing facts. That list is your fix brief.

If you would rather not script it, the free JavaScript rendering check runs the same comparison for one URL.

You can get a rough answer in two minutes without any tooling.

• View page source. In Chrome or Firefox, open View Page Source (not Inspect). That shows the raw HTML the server sent. Use find-in-page to search for your price and your main heading.
• Disable JavaScript. Open DevTools, turn off JavaScript in the settings, and reload. What remains on screen is roughly what a non-rendering crawler can read.
• Compare with Inspect. The Elements panel in DevTools shows the rendered DOM. If a fact appears there but not in View Source, it depends on JavaScript.

This method misses subtle gaps, such as one tab of a pricing table loaded on click. Use it for triage. Use the full diff before you hand a fix to engineering.

Good result

• The raw HTML contains your H1, section headings and body copy.
• Prices, plan names and product names appear as plain text in the raw response.
• The diff shows only cosmetic additions: a cookie banner, a chat widget, analytics markup, personalised greetings.

Bad result

• The raw HTML is a near-empty shell, often a single container element and a list of script tags.
• Headings exist but the body copy, pricing table or feature list arrives later through a data request.
• Content sits inside tabs, accordions or "load more" sections that fetch text only on interaction.
• Key facts appear only in images or in text injected by a third-party widget.

Ambiguous result

Some frameworks embed page data as JSON inside the initial HTML. Vercel notes that JSON or React Server Component payloads in the initial response may still be indexed. Treat this as a partial pass. The data is in the response, but not as readable page text. If the fact matters, put it in the visible HTML as well.

One more thing to check while you are here: if a robots meta tag such as noindex or nosnippet is added only by JavaScript, crawlers that do not render will never see it. See whether AI crawlers can see robots meta tags added by JavaScript.

Run your diff against this list. Each item should be present as text in the raw response, before any JavaScript runs.

• Page title and H1
• Section headings (H2 and H3)
• A one-sentence description of what the product or business does
• Prices, plan names and what each plan includes
• Product and feature names, spelled the way you want them quoted
• Location, service area or opening hours, if relevant
• Founding year, ownership and other facts AI tools tend to get wrong
• Contact details and the canonical company name
• Robots meta directives and canonical tags
• Internal links to your other key pages

Stating a specific fact on a fetchable page gives an engine a sentence it can quote. That is the only on-page content signal with any positive experimental support in our evidence base. For a deeper sweep of individual facts, follow the playbook to find key facts hidden behind JavaScript.

Closing a rendering gap determines whether your content exists in a crawler's copy of the page at all. It does not make an engine more likely to retrieve, rank or cite you. Every later stage (retrieval for a query, then selection and citation) is controlled by the engine, not by you.

The broader evidence backs this caution. A July 2026 critical survey of 45 GEO studies (arXiv 2607.14035) found that no reviewed technique shows a stable, longitudinal, cross-platform causal effect on discoverability. Server rendering is not a growth tactic in that sense. It removes a block.

It also does nothing for Google AI Overviews or AI Mode, because Googlebot renders JavaScript. Google's eligibility rule is that a page must be indexed and eligible for a snippet, with no additional technical requirements. More on that in whether AI Overviews need server-side rendering.

The honest framing for your team: this is hygiene for ChatGPT, Claude, Perplexity and Meta crawlers. If the diff shows a gap, fix it. If it does not, move on to other access checks such as robots.txt and firewall rules. When you are ready to fix, the SSR and prerendering rollout plan covers sequencing by page. For the expected outcome in detail, read whether switching to SSR increases AI citations.

A clean result today can break with one deploy. Rendering gaps usually arrive through changes nobody linked to AI crawlers.

• After any framework, theme or CMS migration. These change how pages are built and are the most common source of new gaps.
• After template changes to pricing, product or landing pages.
• When a new component goes live, such as a pricing calculator, tabbed feature list or reviews widget.
• On a fixed schedule. Quarterly is a sensible default for most sites. Monthly fits sites that ship weekly.

Keep the saved raw and rendered text files from each run. Diffing this quarter's raw HTML against last quarter's shows regressions quickly. If you want a separate snapshot of how AI answers describe you, the free checker on this site shows whether engines name you for up to three buyer questions. Treat it as a snapshot, not a verdict on rendering.

Does this matter if Google is my main channel? For Google's AI surfaces, rendering is not the issue, because Googlebot renders JavaScript. Google says pages need to be indexed and snippet-eligible for AI Overviews and AI Mode, with no additional technical requirements. The gap matters for ChatGPT, Claude, Perplexity and Meta crawlers, which did not execute JavaScript in testing as of June 2026. Most brands care about at least one of those, so the check is still worth running.

Should I fetch the page with an AI crawler user agent? For the rendering check, a plain request is enough, because the question is what the server sends before JavaScript runs. Sending a crawler user agent tests something different: whether your firewall or CDN treats that bot differently. That is a separate access check and worth running too. Use the user agent strings OpenAI, Anthropic and Perplexity publish, and compare status codes, not just content.

Is a perfect match between raw and rendered text required? No. Cookie banners, chat widgets, personalised greetings and analytics will always differ. What matters is whether load-bearing content appears in the raw HTML: headings, body copy, prices, product names and the facts you want quoted. A diff that shows only cosmetic additions is a pass. A diff that shows your pricing table or feature list is missing is a fail.

My site is built with React. Am I automatically affected? Not automatically. React can render on the server, at build time, or only in the browser. Frameworks such as Next.js often send full HTML by default, while a plain single-page app may send an empty shell. The framework name tells you little. The fetch-twice diff tells you what crawlers actually receive, page by page, which is why you run it instead of guessing.