Can AI crawlers see robots meta tags added by JavaScript?

A robots meta tag only works if the crawler reads it. Most AI crawlers read the HTML your server sends and stop there. If your noindex, nosnippet or max-snippet tag is written into the page by JavaScript after load, those crawlers never see it.

Vercel's crawler study found that ChatGPT and Claude crawlers do fetch JavaScript files (11.50% and 23.84% of their requests) but "don't execute them." Independent re-testing in June 2026 found the same for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider. Googlebot and AppleBot are the exceptions: they render. Our guide to JavaScript rendering and AI crawlers covers the wider pattern.

So a JavaScript-injected directive splits your audience. Google may honour it. The non-rendering crawlers see a page with no directive at all. That is rarely what the site owner intended.

JavaScript adds a tag the raw HTML lacks

Say a tag manager adds noindex to staging-like pages or to thin filter pages. Googlebot renders the page, may pick up the tag, and may drop the page. A non-rendering crawler such as OAI-SearchBot gets raw HTML with no tag. Whatever that engine does with robots meta tags, it has nothing to act on. Your intended exclusion only half happened.

JavaScript removes a tag the raw HTML has

The reverse is worse. Some single-page apps ship noindex in the HTML shell and rely on a script to remove it once content loads. Google's JavaScript SEO basics page warns that when Google finds noindex it may skip rendering, so the script that removes it never runs for Google. Non-rendering AI crawlers also see the noindex. The page can end up excluded everywhere.

The second case matters for AI answers on Google. Google states that a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. A stray noindex in the shell removes that eligibility.

The on-page controls with first-party documentation are Google's and Bing's. Google documents noindex, nosnippet, max-snippet and data-nosnippet in its robots meta tag specifications, along with the X-Robots-Tag header. Bing added data-nosnippet support in October 2025 and says marked content "is still indexed normally, but it will be excluded from snippets and AI summaries."

OpenAI, Anthropic and Perplexity document their controls through robots.txt, per bot. OpenAI's bots page says each setting "is independent of the others": GPTBot governs training, OAI-SearchBot governs ChatGPT search. As of October 2026, the evidence we hold does not describe how those crawlers treat a robots meta tag. That is a reason to make the tag readable to every bot, not a reason to assume any particular behaviour.

The Vercel study write-up lists its limits: a one-month sample on one Next.js-heavy site. The no-execution finding has replicated since, so the rendering gap itself is well supported.

• data-nosnippet added by script. It is an HTML attribute, so the same rule applies. If JavaScript adds it, non-rendering crawlers ignore it and Bing may not see it either. Our evidence does not say whether bingbot renders, so do not depend on it.
• Directives in framework head managers. Many frameworks set meta tags from components. Whether the tag appears in raw HTML depends on server rendering. Check the response, not the source code.
• Non-HTML files. PDFs and images cannot carry a meta tag. The X-Robots-Tag header is the only page-level control for them.
• robots.txt blocks. If robots.txt disallows a crawler, it never fetches the page and never reads any meta tag. Blocking and noindexing are different controls with different effects.
• Template inheritance. A nosnippet copied into a shared layout can suppress snippets and AI summaries across the whole site.

1. Fetch the raw HTML. Request the page without JavaScript (curl, or view-source) and search for name="robots" and data-nosnippet. Compare with the rendered DOM using the raw HTML vs rendered DOM check.
2. Move directives server-side. Render the tag into the initial HTML:

<meta name="robots" content="noindex">

3. Or use a header. Set it at the server or CDN, where no script is involved:

X-Robots-Tag: noindex

4. Audit every template. Headers and layouts drift. The X-Robots-Tag header check walks through each template type, and the free noindex and nosnippet check flags directives you did not mean to ship.
5. Confirm crawlers reach the page at all. A directive is irrelevant if a firewall returns 403 first. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.

None of this earns citations. It removes accidental blocks and makes sure your intent reads the same to every bot.