Robots meta tag: definition and directives

The robots meta tag is a meta element placed in the head of an HTML page. It gives crawlers page-level directives: instructions that apply to that one URL. A typical tag looks like this:

<meta name="robots" content="noindex, nosnippet">

The name value can be robots (all crawlers) or a specific crawler name. The content value holds one or more comma-separated directives. The same directives can also be sent as an HTTP header, which is covered in the X-Robots-Tag entry.

A crawler fetches the page, parses the HTML, reads the tag, and applies its directives when deciding what to store and what to display. The directives that affect AI answers, as documented by Google and Bing as of October 2026:

• noindex: keeps the page out of the index entirely. See the full noindex definition.
• nosnippet: lets the page be indexed but blocks any text snippet from it. More in the nosnippet entry.
• max-snippet:[number]: caps snippet length in characters. max-snippet:0 behaves like nosnippet.
• data-nosnippet: not a meta directive but an HTML attribute on a specific element. It excludes that section from snippets while the rest of the page stays eligible.

Bing states that content marked with data-nosnippet "is still indexed normally, but it will be excluded from snippets and AI summaries," and remains available for ranking. These tags remove display. They do not add anything.

Google says that to be eligible as a supporting link in AI Overviews or AI Mode, a page "must be indexed and eligible to be shown in Google Search with a snippet." That makes noindex, nosnippet and max-snippet:0 direct off switches for Google's AI surfaces. There is no separate robots.txt token for AI Overviews, so these on-page controls plus Googlebot access are the levers.

The risk usually runs the other way. A nosnippet added to hide a cookie banner, or a noindex left on from staging, can quietly remove a page from AI answers. Nothing in a dashboard flags it.

The next move: audit your key pages (pricing, product, about) for robots meta directives and data-nosnippet attributes, and confirm each one is intentional. Remember the scope too. Google and Bing document these controls. For ChatGPT search, OpenAI documents crawl permission for OAI-SearchBot in robots.txt as the control.

"robots.txt and the robots meta tag do the same job"

They don't. Robots.txt controls crawling, per bot. The meta tag controls indexing and display, per page. If robots.txt blocks a URL, the crawler never fetches it and never reads its noindex. Google notes a blocked URL can still be indexed without its content. Read the Disallow directive entry for the crawl side.

"Any meta tag works if it ends up in the page"

Googlebot renders JavaScript. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. A tag injected client-side may be invisible to those bots. See whether AI crawlers see meta robots added by JavaScript.

"There's a tag that opts you out of AI"

No major engine documents a noai directive. Here is what the evidence says about noai tags.

"A permissive robots meta tag helps you get cited"

It doesn't. index, follow is the default. These directives only take eligibility away.

• noindex: the directive that removes a page from the index.
• nosnippet: blocks snippets and Google AI feature display.
• X-Robots-Tag: the same directives delivered as an HTTP header, useful for PDFs and non-HTML files.
• Disallow directive: the robots.txt rule that blocks crawling.

For a wider checklist of markup and rendering problems, see code-level issues that stop AI indexing.