Can I completely stop AI from using my content?

You can't make your content invisible to every AI system. You can, though, control most of the ways AI companies collect and display it.

The reason is that "AI using my content" covers four different things:

• Training: a crawler collects pages to train future models.
• Search indexing: a crawler stores pages so an AI search feature can cite them.
• Answer display: an engine shows your text in an AI answer.
• User-triggered fetching: an agent opens your page because a user asked it to.

Each has its own control, and some have none. A rule that blocks training does nothing to search. A rule that blocks search does nothing to a model trained last year. So the first move is to decide which of the four you actually want to stop. Our comparison of training crawlers and search crawlers lists the names per vendor.

As of October 2026, these are the documented controls.

A training-only opt-out looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

OpenAI states each of its settings "is independent of the others", so you can block GPTBot and still appear in ChatGPT search. Our playbook on how to block AI training but stay in AI search walks through that split. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

On-page snippet directives work on both Google and Bing. Bing documents that content marked data-nosnippet "is still indexed normally, but it will be excluded from snippets and AI summaries." For page-level and section-level recipes, see how to keep specific content out of AI answers.

Be plain with yourself about what these controls can't do.

• User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user starts the action. Perplexity says Perplexity-User "generally ignores robots.txt rules." Meta documents that Meta-ExternalFetcher may bypass them too.
• No robots token for Google's AI surfaces. Google says robots.txt directives for Googlebot are the control, because AI is built into Search. Blocking Googlebot removes you from Google Search entirely. That's usually a worse trade.
• Google's AI features setting is partial. The Search Console control is opt-out, with inclusion as the default, and was still rolling out to a subset of owners as of October 2026. Google also says opting out of display doesn't stop content being used to help Search understand queries. Our guide to Google's AI features control covers who has it and how child properties inherit it.
• Models already trained. A model's knowledge is frozen at its cutoff. A block you add now affects future collection, not what a released model already learned.
• Other sites. If a review site or forum quotes you, engines can cite that page. You control your own domain only.
• Unofficial tags. A "noai" meta tag has no documented support from the major vendors in the evidence we hold.

robots.txt is a request, not a lock. Cloudflare, which sees the network traffic, says it remains unclear whether newer crawlers respect it. A firewall or CDN rule that rejects a crawler by user-agent and verified IP is the harder enforcement layer.

The same mechanism cuts the other way. A firewall can block a crawler your robots.txt allows. From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search stays allowed by default. Some sites lost AI access without editing anything.

What that means: check what your edge actually returns to each bot, not only what robots.txt says.

1. Decide your goal per purpose. Most businesses want to block training but stay findable in AI search. A few want out of everything.
2. Write per-bot robots.txt rules. Name GPTBot, ClaudeBot and Google-Extended for training. Only block OAI-SearchBot, Claude-SearchBot or PerplexityBot if you accept losing those search answers. Read what still happens when you block OAI-SearchBot first.
3. Add snippet directives for pages or passages that must never appear in AI answers.
4. Check Search Console for the generative AI features setting on every property, including inherited ones.
5. Test the edge. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
6. Accept the residue. User-triggered fetchers, trained models and third-party quotes sit outside your control.