Checking X-Robots-Tag Headers on Every Template

Robots directives can live in two places. One is the <meta name="robots"> tag inside the HTML. The other is the X-Robots-Tag field in the HTTP response header, which the server sends before any HTML. Browsers don't display headers, so a page can look perfect and still tell crawlers not to index it. (See the X-Robots-Tag definition for the full syntax.)

The header matters most where there's no HTML at all. A PDF, an image or a feed can't carry a meta tag, so the header is the only way to send it a robots directive. That also makes it the easiest place for a blanket rule to hide.

Google documents noindex, nosnippet and max-snippet as directives that work in either the meta tag or the header. Google's AI features page states that a page must be "indexed and eligible to be shown in Google Search with a snippet" to be a supporting link in AI Overviews or AI Mode. So a header that sets noindex or nosnippet takes that URL out of contention on Google's AI surfaces, as of October 2026. For how each directive behaves, read noindex vs nosnippet vs data-nosnippet for AI.

The next move: stop trusting what the browser shows and read the raw response.

Here is an intended header on an internal price sheet you don't want in search or AI answers:

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex, nosnippet

And here is a directive scoped to one crawler only, which Google's specification supports:

x-robots-tag: googlebot: nosnippet

Three rules keep a header correct:

• Directives are comma-separated in one header, or split across several x-robots-tag lines. Google combines them.
• When rules conflict, Google's specification applies the more restrictive one. A permissive meta tag won't override a restrictive header.
• A crawler has to fetch the URL to see the header. If robots.txt blocks the URL, the crawler never reads the noindex.

On a public page you want found, the usual correct result is no x-robots-tag at all, or one that only sets preview limits you chose deliberately.

You need a terminal with curl and a list of URLs. Pick one representative URL per template, not per page: homepage, product page, category page, blog post, pricing page, docs page, a PDF, an image and any feed.

1. Fetch headers with a normal GET. Some servers answer a HEAD request differently, so use a GET and discard the body:

curl -s -D - -o /dev/null https://example.com/pricing

2. Filter for the directive. Pipe the output through grep -i x-robots-tag. No output means no header was sent.

3. Follow redirects. Add -L and read every hop. A redirect response can carry its own header, and the final URL may differ from the one you typed.

4. Repeat as a crawler user agent. Add -A with Googlebot's user-agent string, then bingbot's. CDN and server rules often branch on user agent. Treat this as a first pass only: a spoofed user agent from your own IP isn't the same as real crawler traffic, and bot management may treat it differently.

5. Confirm with first-party tools. Google Search Console's URL Inspection reports index status for Google's real crawl. Bing Webmaster Tools does the same for Bing, which backs Copilot. Their verdict outranks your curl result.

6. Compare header against HTML. Check the same URL's meta robots tag. A clean header with a noindex meta tag is still blocked. Use the accidental noindex audit for the HTML side.

7. Record the result per template. Note URL, user agent, status code and the exact header value. That log is your baseline for the next run.

Headers are set by rules, and rules apply to whole groups of URLs. One line of config can cover thousands of pages nobody looks at.

PDFs and file downloads

Servers often apply one header to every file of a type. A rule meant to hide internal PDFs can also hide the public whitepaper, the price list or the spec sheet. Test one PDF from each folder you publish from.

CDN and edge rules

CDNs can add or rewrite response headers at the edge. Your origin may send nothing while the edge adds noindex. Always test the public hostname, not the origin. Firewalls and bot management are a separate gate: a crawler blocked with a 403 never sees any header. Bingbot, Copilot and your firewall covers that layer.

Staging configs that reach production

A blanket noindex header on staging is good practice. Copied into a production config during a migration, it silently removes the whole site. Check headers on the day of any launch or move.

Framework and hosting defaults

Some hosting platforms add noindex headers to preview or branch deployments. Confirm your production domain isn't served through one of those paths.

• One URL tested for every page template, including the pricing page.
• At least one PDF, one image and one feed URL tested.
• Each URL fetched with a default user agent, Googlebot and bingbot.
• Redirect chains followed, with headers read at every hop.
• Public hostname tested, not the origin server.
• Every x-robots-tag value recorded exactly as returned.
• Each restrictive value matched to an owner and a reason.
• Meta robots tags on the same URLs checked for conflicts.
• Search Console and Bing Webmaster Tools index status compared with your results.
• Status codes recorded, so a 403 isn't mistaken for a clean header.

If you want a quick single-URL read before running the full list, the free noindex and nosnippet check covers both the header and the HTML.

A good result is boring: every restrictive header has an owner and a reason. A bad result is any restrictive header nobody can explain.

Be plain about the limits. Removing a stray header restores eligibility. It doesn't make Google index the page, and Google states that indexing and serving aren't guaranteed. Which pages AI answers cite is decided by the engines.

Headers change when infrastructure changes, not when content changes. Tie the check to those events:

• After every deploy that touches server, CDN or hosting config. That includes new edge rules, cache settings and security products.
• On launch day of any migration or redesign. Staging headers leak here more than anywhere else.
• When you add a new template or file type. New URL patterns get whatever rule they happen to match.
• Monthly, on a fixed schedule. Rules drift, and vendors change defaults without asking you.

Keep the per-template log from each run and diff it against the last one. A new header that appeared without a ticket is the finding.

Headers are one gate of several. Crawlers also need to reach the page at all, and robots.txt and firewalls can block them first. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.

Does an X-Robots-Tag header affect ChatGPT or Perplexity? Google and Bing document how they handle noindex and nosnippet. OpenAI controls ChatGPT search inclusion through the OAI-SearchBot rules in robots.txt, as documented on its bots page. Whether other AI engines read X-Robots-Tag varies, so check each vendor's own documentation rather than assuming. The page on [whether X-Robots-Tag keeps pages out of AI search](/questions/x-robots-tag-ai-search) goes into the per-engine detail.

Can I use data-nosnippet in an HTTP header? No. data-nosnippet is an HTML attribute you place on a specific element, so it only works inside the page markup. Headers apply to the whole URL. Bing added data-nosnippet support in October 2025 and states that marked content is still indexed but excluded from snippets and AI summaries. Use the header for whole files, and the attribute for sections of an HTML page.

Should I remove X-Robots-Tag headers from all PDFs? Only from PDFs you want found. A header that hides internal documents, drafts or duplicates is doing its job. The problem is a blanket rule that also covers public files like price lists or product sheets. Split your rules by folder or path so public and private files get different headers, then retest one file from each.

Will a browser extension show X-Robots-Tag? Some extensions read response headers, and your browser's developer tools show them under the network tab. Both only show what your browser received, with your user agent and your IP. CDN rules that branch on crawler identity won't appear. Use curl with crawler user agents for a wider read, then confirm with Search Console and Bing Webmaster Tools.

If I remove a stray noindex header, will the page appear in AI Overviews? Removing it restores eligibility, nothing more. Google says a page must be indexed and snippet-eligible to be a supporting link, and that indexing and serving aren't guaranteed. Whether an AI Overview triggers for a query, and which pages it cites, is Google's decision. Read [whether noindex or nosnippet stop pages appearing in AI Overviews](/questions/noindex-nosnippet-ai-overviews) for the eligibility rules.