Vercel's AI Crawler Study: AI Crawlers Fetch JavaScript but Don't Execute It

Vercel, the hosting company behind Next.js, published "The rise of the AI crawler" on 17 December 2024 and updated it on 6 February 2025. The post looked at how AI crawlers behave when they request pages, and one finding has outlasted the rest: they fetch JavaScript but don't run it.

That matters because many modern sites use client-side rendering. The server sends a near-empty HTML shell, and the browser builds the visible page by executing JavaScript. A crawler that skips that step sees the shell, not your pricing, product names or headings.

The study is the strongest single piece of evidence behind any on-site AI visibility fix. It explains a failure that is common, invisible from a normal browser, and fully in the publisher's control.

Vercel's data is a one-month snapshot from December 2024, taken on nextjs.org. Crawlers were identified by user agent. That is why Microsoft Copilot was left out: it did not have a unique user agent to separate it from other traffic.

The analysis counted what crawlers requested, including how often those requests were for JavaScript files, and checked whether the crawlers acted on that JavaScript to read rendered content.

The evidence base records no further method detail, such as sample counts per crawler. Read the figures below as one site's traffic over one month, not a web-wide census.

Vercel's headline sentence, quoted exactly: "The data indicates that while ChatGPT and Claude crawlers do fetch JavaScript files (ChatGPT: 11.50%, Claude: 23.84% of requests), they don't execute them. They can't read client-side rendered content."

The download is the puzzling part. If a crawler won't run JavaScript, why fetch it? We cover the likely reasons in why AI crawlers download JavaScript files they don't run. For your site, the practical point is simple: a request for a .js file in your logs does not mean the crawler saw what that file renders.

Re-confirmed in 2026

Independent re-testing in June 2026 found that none of these crawlers render JavaScript:

• GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI)
• ClaudeBot and Claude-SearchBot (Anthropic)
• PerplexityBot (Perplexity)
• Meta-ExternalAgent (Meta)
• Bytespider (ByteDance)

The exceptions

Googlebot renders JavaScript, and Applebot renders through a browser-based crawler. So a client-side rendering gap is mostly irrelevant to Google AI Overviews, AI Mode and Gemini's Google-backed paths. It is decisive for ChatGPT search, Claude, Perplexity and Meta AI.

One nuance from Vercel: JSON or React Server Component payloads included in the initial HTML response may still be indexed. Content loaded later, by a separate request after the page boots, is the risk.

The finding is solid. The study still has limits worth stating plainly.

• One site, one month. nextjs.org is a developer documentation site built on Next.js. Its traffic is not a random sample of the web.
• Copilot is missing. Without a unique user agent, Microsoft's crawler could not be measured.
• The vendor has an interest. Vercel sells server-side rendering and hosting. That doesn't make the data wrong, and the result has since replicated independently, but the incentive is real.
• It is a snapshot of behavior. Crawler capabilities can change. As of October 2026, the June 2026 re-test is the most recent confirmation in our evidence base.
• It is about access, not selection. Fixing a rendering gap puts your content into the candidate pool. It does not make an engine more likely to choose or cite you. We explain that distinction in whether switching to server-side rendering increases AI citations.

That last point is the one most often overstated. Rendering is hygiene. It removes a block. Nobody controls what the model then retrieves or says.

Treat this as a check you run, not a rewrite you commission. The goal is to confirm that the facts you care about exist in the HTML before any JavaScript executes.

1. List your load-bearing facts. Prices, plan names, product names, what you do in one line, location, key numbers.
2. Fetch the raw HTML. Request the page with JavaScript off, the way a non-rendering crawler does. A quick test from a terminal:

curl -s https://example.com/pricing | grep -i "per month"

3. Render the same URL in a headless browser and extract the visible text.
4. Diff the two. Anything present only after rendering is invisible to the AI crawlers listed above. Our raw HTML vs rendered DOM playbook walks through this step by step.
5. Check interactive content. Tabs, accordions, "load more" buttons and infinite scroll often inject text by fetch. Use the hidden-facts playbook to find them.
6. Fix the gap at the source. Server-side rendering, static generation or prerendering for the affected pages. The guide to JavaScript rendering and AI crawlers compares the options.

Rendering is one gate. A crawler also has to be allowed in by robots.txt and let through by your CDN or firewall. Check all three before drawing conclusions about why an engine doesn't mention you.