X-Robots-Tag: definition and when to use it

X-Robots-Tag is an HTTP response header that tells crawlers how to index and display the file it comes with. It accepts the same directives as the robots meta tag: noindex, nosnippet, max-snippet, noarchive and others, as listed in Google's robots meta tag specification.

The difference is where the instruction lives. A meta tag sits inside an HTML page. The header travels with the server's response, so it applies to files with no HTML at all:

• PDFs, Word documents and spreadsheets
• Images and video files
• API or feed responses you don't want indexed

If the file is HTML, use whichever is easier to maintain. If it isn't, the header is your only on-file option.

The server adds the header to its response. A crawler fetches the file, reads the header, and applies the directives. A short example for a PDF price sheet you want kept out of results:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nosnippet

Google's specification also lets you target one crawler by name, for example X-Robots-Tag: googlebot: nosnippet. Directives without a name apply to every crawler that supports them.

Two mechanics matter in practice:

1. The crawler must reach the file. If robots.txt disallows the URL, the crawler never fetches it and never sees the header. A blocked URL can still be known from links elsewhere, so a robots.txt block is not the same as a noindex.
2. You usually set it in server or CDN config, not in the CMS. One rule can cover a whole folder or file type. That is the convenience and the danger.

Google states that a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. A noindex or nosnippet in the header fails that test. Bing documents the same idea for its AI summaries, where content excluded from snippets is excluded from AI answers too.

So X-Robots-Tag is a suppression control. It is useful when you mean it: an outdated rate card, a draft PDF, a gated report. It is costly when you don't.

The common failure is server-wide. Say a staging environment sends X-Robots-Tag: noindex on every response, and that config ships to production. Every page and file drops out of eligibility. Nothing changes in the HTML, so a source-code review won't catch it, and the site looks fine in a browser.

The fix is to check the actual headers on every template, not just the homepage. The X-Robots-Tag header check playbook walks through it, and the free noindex and nosnippet check covers the on-page versions. For how other engines treat the header, see whether an X-Robots-Tag header keeps pages out of AI search.

• "It blocks crawling." It doesn't. It controls indexing and display after the fetch. Crawling is robots.txt's job.
• "It controls AI training." Not for Google. Google-Extended, set in robots.txt, governs whether content is used for Gemini training and grounding. OpenAI separates training (GPTBot) from search (OAI-SearchBot) the same way, as covered in OpenAI's crawler documentation.
• "Some directive in it boosts citations." No. Every directive removes or limits something. There is no additive value to set.
• "Every AI crawler honors it." Google and Bing document these controls. Other operators publish their own rules, so check each one rather than assume.
• "Adding it with JavaScript works." A header is set by the server. For the meta tag version, see whether AI crawlers see robots tags added by JavaScript.

• noindex: the directive most often sent in this header.
• Robots meta tag: the in-page equivalent for HTML.
• data-nosnippet: an HTML attribute that hides one section from snippets and AI summaries while the page stays indexed.
• robots.txt: the crawl-level file that decides whether a crawler fetches the URL at all.

Should I use X-Robots-Tag or the robots meta tag? For HTML pages, either works, so pick the one your team can maintain and audit. For PDFs, images and other non-HTML files, X-Robots-Tag is the only on-file option because those files have no head section. Avoid setting conflicting values in both places, since crawlers apply the most restrictive directive they find.

How do I see a page's X-Robots-Tag header? Inspect the HTTP response, not the page source. Your browser's developer tools show response headers in the network tab, and a command-line request that prints headers works too. Check one URL from each template and file type, including PDFs, because server rules often apply by folder or extension rather than site-wide.

Will nosnippet in the header remove my page from Google entirely? No. nosnippet keeps the page indexed but stops Google showing a text snippet. Because Google's AI Overviews and AI Mode require snippet eligibility, the page also stops being eligible as a supporting link there. If you only want to hide part of a page, data-nosnippet on that section is the narrower tool.

Can a CDN add X-Robots-Tag without my knowing? Yes. CDNs and hosting platforms let you add response headers through rules, and those rules sit outside your CMS. A rule meant for a staging hostname or a file folder can match more than intended. After any CDN or server config change, re-check headers on live production URLs rather than trusting the config file.