What are content signals in robots.txt, and do AI companies honor them?

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

Content signals are lines in robots.txt, defined by Cloudflare's Content Signals Policy, that state whether your content may be used for search, as input to AI answers, or for AI training. They express a preference and block nothing. None of the AI vendor crawler documentation in our evidence base says its crawlers read them. Per-bot robots.txt rules and network-level controls remain the levers vendors actually document.

Key takeaways

A content signal is a declared preference about use, not a technical block on access.
Cloudflare itself says it is unresolved whether AI crawlers respect robots.txt at all, especially newer or less transparent ones.
OpenAI, Google, Anthropic, Perplexity and Meta document per-bot user-agent rules, not Content-Signal lines, as their controls.
A robots.txt allow means nothing if your CDN or firewall returns a 403 to the crawler, so test the network layer too.
Add content signals if you want your terms on record, but set per-bot rules and check reachability first.

The short answer: a preference, not a control

Cloudflare's Content Signals Policy adds a way to say, inside robots.txt, how you want your content used after it is fetched. It separates three uses: search, AI input (content used to generate answers) and AI training. You set each one to yes or no.

That is useful as a statement of terms. It is not enforcement. robots.txt has always worked on trust: RFC 9309, the Robots Exclusion Protocol, says its rules are not a form of access authorization. A crawler that ignores a Disallow line will ignore a Content-Signal line too.

The bigger gap is adoption. The crawler pages published by OpenAI, Google, Anthropic, Perplexity and Meta describe controls based on user-agent names. None of the vendor documentation in our evidence base says its crawlers parse Content-Signal lines. So, as of October 2026, treat content signals as a record of your wishes, and rely on per-bot rules and network controls for anything you need to actually happen.

What a content signal line looks like

Content signals sit alongside normal robots.txt groups. A site that wants to appear in search but opt out of training might publish this:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

In Cloudflare's policy, the three signals mean:

search: building a search index and showing links or short excerpts.
ai-input: feeding content into an AI model at answer time, such as retrieval for a generated response.
ai-train: using content to train or fine-tune a model.

Leaving a signal out expresses no preference either way. The line only describes use. It does not tell a crawler whether it may fetch the page; the Allow and Disallow rules still do that. For the fetch rules themselves, see our per-bot guide to robots.txt for AI crawlers.

What the evidence says about compliance

The honest answer is that nobody has measured it. Cloudflare, which sees more crawler traffic than most, frames the problem itself: some site owners are unsure whether these bots even respect robots.txt rules, especially newer or less transparent crawlers. That doubt applies to plain Disallow rules, before any new syntax is added.

The vendors' own documentation shows where robots.txt stops working even for well-known bots:

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person triggers those fetches.
Perplexity says Perplexity-User generally ignores robots.txt rules.
Meta documents that Meta-ExternalFetcher may bypass robots.txt.

If user-triggered fetchers can skip Disallow, there is no basis for assuming any of them read an ai-input=no signal. The claim you can make is narrow: you have published your terms. Whether a given crawler acts on them is outside your control and, today, outside anyone's measurement.

The levers vendors actually document

Three controls have a documented consequence stated by the operator of the system.

Per-bot robots.txt rules

Each vendor splits training from search under different user-agents. OpenAI states that each setting is independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to opt out of training. Anthropic separates ClaudeBot from Claude-SearchBot. Google uses Google-Extended for Gemini training and grounding, and says it does not affect inclusion in Google Search. Our comparison of training crawlers vs search crawlers lays out the pairs.

Network reachability

OpenAI's help centre says eligibility for ChatGPT search means allowing OAI-SearchBot in robots.txt and confirming your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Both conditions apply. A firewall that returns 403 to a crawler overrides any allow, and any content signal.

CDN defaults you did not set

From 2026-09-15, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. That is a network rule, not a signal, and it can change your AI access without an edit on your side.

What to do next

Decide per use, then per bot. Choose whether you want search visibility, training use, or both. Our explainer on whether allowing AI search bots means training use covers the trade-off.
Write the user-agent rules first. Start from the robots.txt template for AI search and training bots. Add a Content-Signal line afterwards if you want your terms stated.
Test each bot. Follow how to test robots.txt rules for each AI bot, then confirm the network layer agrees. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
Watch for regressions. CDN and template changes break access silently. Set alerts for AI crawler access so you hear about it.

None of this guarantees a citation or keeps a determined scraper out. It makes sure your stated rules and your actual gates say the same thing.

Frequently asked questions

Can adding a Content-Signal line break my robots.txt?

Nothing in the evidence suggests it causes harm. RFC 9309 lets crawlers interpret records beyond the core protocol if they choose, so a crawler that does not recognize Content-Signal has no obligation to act on it. The risk is elsewhere: editing robots.txt by hand can introduce typos in the rules that matter. Validate the file per bot after every change.

Is ai-train=no the same as disallowing GPTBot?

No. Disallowing GPTBot is the control OpenAI documents: its bots page says disallowing GPTBot indicates content should not be used to train its foundation models. A Content-Signal line is a general statement that OpenAI does not document reading. If you want training opt-out from a specific vendor, use that vendor's named user-agent.

Does ai-input=no keep me out of AI answers?

Not reliably. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, by their operators' own documentation. For Google, the documented on-page controls for limiting AI feature display are noindex, nosnippet, max-snippet and data-nosnippet. There is no robots.txt token that removes a page from AI Overviews or AI Mode.

Will a training signal remove content a model already learned?

Nothing in the evidence says so. Any robots.txt change, signal or rule, applies to future crawling. What a model learned before its knowledge cutoff stays in that model, and any effect from a training opt-out would only appear in a later model generation. Our page on whether [blocking a crawler makes AI forget pages](/questions/blocking-crawler-already-crawled-pages) covers this in more detail.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.