Does an X-Robots-Tag header keep pages out of AI search?

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

Yes for Google, partly elsewhere. Google documents X-Robots-Tag as an HTTP header equivalent of the robots meta tag. A noindex or nosnippet sent that way removes a page's eligibility for AI Overviews and AI Mode. OpenAI, Perplexity and Anthropic document their controls through per-bot robots.txt rules, not this header. To keep files out of those engines, pair the header with robots.txt rules for each bot.

Key takeaways

Google treats X-Robots-Tag directives like robots meta directives, so noindex or nosnippet in the header affects AI Overviews and AI Mode eligibility.
X-Robots-Tag is the practical way to apply noindex to non-HTML files such as PDFs, which cannot carry a meta tag.
OpenAI, Perplexity and Anthropic document per-bot robots.txt rules, not X-Robots-Tag, so the header alone is not a documented control for their search products.
A robots.txt Disallow stops Google from fetching the file, so it never sees the header: do not combine the two on the same URL for Google.
User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, so no single control removes a file from every AI answer.

The short answer, and why

For Google, an X-Robots-Tag header works. For other AI engines, it is not a documented control.

The reasoning runs through Google's eligibility rule. Google's AI features documentation says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. Google's robots meta tag specification lets you send the same directives (noindex, nosnippet, max-snippet) as an HTTP response header called X-Robots-Tag. So a header that says noindex removes index eligibility, and a header that says nosnippet removes snippet eligibility. Either one takes the page out of the pool that AI Overviews and AI Mode draw supporting links from.

OpenAI, Perplexity and Anthropic describe their search controls differently. Their crawler pages, as of October 2026, are built around robots.txt user-agent tokens: OAI-SearchBot, PerplexityBot and Claude-SearchBot. None of the evidence we hold shows those vendors documenting X-Robots-Tag. If you need a file out of those products, a per-bot robots.txt rule is the documented route.

What this means: the header is the right tool for Google and for file types that cannot hold a meta tag. It is not a universal AI off switch.

What the header looks like and when to use it

X-Robots-Tag lives in the HTTP response, not the HTML. Your server or CDN adds it. A response for a PDF you want kept out of Google Search, including its AI features, would include:

X-Robots-Tag: noindex

To keep a file indexed but stop Google from quoting it in snippets and AI answers, send:

X-Robots-Tag: nosnippet

Use the header when:

The file is not HTML. PDFs, spreadsheets and images have no <head> for a meta tag.
You want one rule across a whole directory or file type, set once in server config.
Your CMS makes per-page meta tags hard to control.

One limit: data-nosnippet is an HTML attribute on a page element. It has no header form. For a comparison of the three directives and what each one removes, see noindex vs nosnippet vs data-nosnippet for AI.

Where the header does not reach

Three conditions limit what the header can do.

Google has to fetch the file to read it

A header is only seen when the crawler gets the response. If robots.txt disallows the URL for Googlebot, Google cannot fetch it and never sees your noindex. The URL can still be known from links. For Google, pick one: allow crawling and send noindex, or block crawling and accept that the header is moot.

User-triggered fetchers play by different rules

OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a person initiates the fetch. Perplexity says Perplexity-User "generally ignores robots.txt rules." A file reachable on the open web can still be pulled into a single answer when a user asks for it. If something must stay private, put it behind authentication, not a directive.

Training is a separate decision

Keeping a file out of search answers and keeping it out of model training are different controls.

Google: Google-Extended governs whether content is used for Gemini training and grounding. Google says it does not affect inclusion in Google Search. X-Robots-Tag on Googlebot controls Search and its AI features. There is no robots.txt token that blocks AI Overviews or AI Mode alone.
OpenAI: GPTBot governs training only. OAI-SearchBot governs ChatGPT search. OpenAI states "each setting is independent of the others."
Anthropic: ClaudeBot is the training crawler. Claude-SearchBot and Claude-User are separate agents.

If your goal is to stay visible in AI search while opting out of training, the playbook for blocking AI training but staying in AI search walks through the per-bot rules. Allowing or blocking a training bot is not documented to change search visibility in either direction.

What to do next

Treat X-Robots-Tag as one layer in a per-engine plan.

List what must stay out. Name the files or paths and the engines that matter: Google, ChatGPT, Perplexity, Claude.
For Google, send the header. Use noindex to remove a file from Search and its AI features, or nosnippet to keep it indexed but unquoted. Make sure robots.txt still lets Googlebot fetch it.
For other engines, add per-bot robots.txt rules. Disallow OAI-SearchBot, PerplexityBot or Claude-SearchBot on those paths. Do not block GPTBot expecting it to affect ChatGPT search.
Audit for accidents. The more common failure is the reverse: a staging rule or CDN default that sends noindex on pages you want cited. Run the X-Robots-Tag header check on every template, since headers are invisible in the page source.
Confirm the robots.txt side. The robots.txt Checker for AI Crawlers shows how each AI bot resolves against your rules.

None of these steps earns a citation. They decide eligibility. What engines do with eligible pages is outside your control.

Frequently asked questions

How do I block PDFs from AI search?

For Google, serve the PDFs with an `X-Robots-Tag: noindex` header and leave them crawlable so Googlebot can read it. That removes them from Search, AI Overviews and AI Mode. For ChatGPT search, Perplexity and Claude search, add Disallow rules for OAI-SearchBot, PerplexityBot and Claude-SearchBot on the PDF path. If the files are sensitive, put them behind a login, because user-triggered fetchers may ignore robots.txt.

Is X-Robots-Tag noindex the same as a meta robots noindex for AI Overviews?

Google's robots meta tag specification treats them as equivalent directives delivered in different places. Either one makes a page ineligible for Google's index, and AI Overviews and AI Mode only use indexed pages with snippet eligibility as supporting links. The practical difference is coverage: the header works on any file type and can be set server-wide, while the meta tag only works inside HTML. See [whether noindex or nosnippet stop AI Overviews](/questions/noindex-nosnippet-ai-overviews) for the page-level view.

Can I target only Googlebot with X-Robots-Tag?

Google's specification lets you prefix a directive with a user-agent name, for example `X-Robots-Tag: googlebot: nosnippet`. That scopes the rule to Google's crawler. It does not create a separate control for AI Overviews or AI Mode, because those features run on the same Googlebot index as regular Search. Other AI vendors do not document reading this header, so a scoped header for their bots is not a documented control.

Does nosnippet in the header remove a page from Google entirely?

No. A nosnippet directive keeps the page in Google's index but stops Google from showing text from it in snippets. Because AI Overviews and AI Mode require snippet eligibility for supporting links, the page loses that eligibility too. Use nosnippet when you want the URL findable in classic results but not quoted. Use noindex when you want it gone from Search altogether.

Why does my CDN send an X-Robots-Tag I never set?

Some hosting platforms, staging environments and CDN rules add headers by default, especially on preview domains. If that configuration leaks to production, every response can carry noindex without anything showing in your HTML. Check live response headers on each template, not just page source. Because headers are inherited from server and edge settings, one misconfigured rule can remove an entire section from Google's AI features at once.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.