What code-level issues stop AI search engines from indexing a page?

Most code-level failures come down to one fact. Many AI crawlers read the raw HTML your server returns and stop there. If a fact is not in that response, or a directive tells the engine not to show it, the engine cannot use it.

Four issues cause nearly all of it:

• Client-side rendering. The page ships an empty shell and JavaScript builds the content in the browser.
• Content behind interaction. Tabs, accordions, 'load more' and infinite scroll that fetch content only after a click or scroll.
• Stray suppression directives. A noindex, nosnippet, max-snippet:0 or data-nosnippet that someone added for one reason and that now applies far wider.
• Oversized HTML. A page heavy enough to pass Googlebot's per-URL byte limit. This is rare.

These sit in the access stage, before retrieval and generation. They are the part you fully control. Two other access gates sit outside your page code: your per-bot robots.txt rules and whether your CDN or firewall turns AI crawlers away. Check all three together.

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of requests) but don't execute them, so "they can't read client-side rendered content." Independent re-testing in June 2026 found the same for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider. Googlebot and AppleBot are the exceptions: they render.

So a single-page app can look complete to Google and blank to ChatGPT search. The usual casualties are the facts buyers ask about: prices, plan limits, product names, locations.

Interactive content follows the same rule:

• If a tab's text is in the initial HTML and only hidden with CSS, a non-rendering crawler still receives it.
• If the tab fetches its text when clicked, the crawler never sees it.
• 'Load more' and infinite scroll usually fetch, so items past the first batch are missing.

The fix is server-side rendering or prerendering for the content that matters. Vercel notes that JSON or React Server Component payloads included in the initial HTML response may still be indexed, but content streamed in later is at risk. Text baked into images is a related gap, covered in whether AI can read text inside images.

Google and Bing document noindex, nosnippet, max-snippet and data-nosnippet as the on-page controls over what appears in search and AI answers. Google states that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet. Bing says content marked data-nosnippet is still indexed but "excluded from snippets and AI summaries."

These are useful when intended. The failure is inheritance:

• A data-nosnippet wrapper meant for a cookie banner ends up around the main content block.
• A staging noindex survives a deploy or a site migration.
• An X-Robots-Tag: noindex or nosnippet HTTP header is set at the server or CDN level, where nobody looking at the HTML sees it.
• A global max-snippet:0 in the base layout applies to every page.

None of these throws an error. The page simply stops appearing.

Google's March 2026 post on Googlebot crawling states that Googlebot fetches up to 2MB per URL (PDFs up to 64MB), and bytes past that are not fetched, rendered or indexed. The same post gives 15MB as the default for other crawlers that don't set a limit. The counter is per URL, and sub-resources count separately.

Google also says most of the web never hits this. Treat it as a check, not a project. Pages most at risk carry huge inline JSON blobs, inlined images or thousands of product variants in one document. If your key facts sit near the bottom of such a page, they may sit past the cut.

You can test the first three issues in minutes without special tools. Fetch the page the way a non-rendering crawler does, then look for a fact you care about.

curl -s https://example.com/pricing -o raw.html
grep -i "per month" raw.html
grep -io 'noindex\|nosnippet\|max-snippet\|data-nosnippet' raw.html
curl -sI https://example.com/pricing | grep -i x-robots-tag
wc -c raw.html

Read the results like this:

1. If your price or product name is missing from raw.html but visible in the browser, it depends on JavaScript.
2. Open a tab or press 'load more' in the browser, then grep raw.html for that text. Missing means it is fetched on interaction.
3. Any directive match needs an owner and a reason. If nobody can give one, it is a bug.
4. The byte count from wc -c flags only pages near 2MB.

This tests your code, not your network. A server can return different responses to real crawler user agents, so pair it with the AI Crawler Checker, which requests the page as each AI crawler and shows which ones get turned away. For shops, the ecommerce product data check applies the same test to prices and stock.

These fixes are subtractive. They remove a block so the content exists for the engine at all. They do not make the engine more likely to choose you, and retrieval still happens per query at each engine's discretion.

That is still the strongest evidence-backed work available. Content rewrites tested in published studies were measured on pages already in the model's context, and the best-designed of them found little or negative effect, as covered in rewrites versus access fixes.

Next move: run the check on your five highest-value pages, then turn each failure into a specific ticket. The developer tickets playbook shows how to write them so they get shipped.