If I block AI crawlers, can AI still talk about my brand?
99Visibility Research · Updated October 21, 2026 · 5 min read
Short answer
Yes. Blocking AI crawlers does not erase your brand from AI answers. Models can answer from what they learned before their knowledge cutoff. Engines can retrieve and cite third-party pages that mention you. Some user-triggered fetchers do not reliably follow robots.txt. What blocking mostly removes is your own site as a source, so answers about you lean on other people's version of the facts.
Key takeaways
- A robots.txt block controls which bots fetch your pages, not what a model already knows or what other sites say about you.
- Blocking search crawlers like OAI-SearchBot or PerplexityBot removes your own pages as a source, while third-party pages stay available to the engine.
- Training bots and search bots are separate: blocking GPTBot does not remove you from ChatGPT search, and blocking Google-Extended does not affect AI Overviews.
- User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, so a block is not a guarantee.
- If you want to limit how your content appears, snippet controls are more precise than blocking a crawler outright.
The short answer, and why
Yes, AI can still talk about your brand after you block its crawlers. A robots.txt rule controls whether a given bot may fetch your pages. It does not control what a model already knows, or what other sites publish about you.
Answers about a brand come from three places:
- Parametric knowledge. This is what the model learned in training, frozen at a knowledge cutoff. A robots.txt change today cannot reach it.
- Third-party pages. Reviews, directories, news and forums. When retrieval is on, the engine can fetch and cite those pages. Your block only applies to your site.
- User-triggered fetchers. When someone asks an assistant to open a page, a different agent fetches it. Some of those agents don't reliably follow robots.txt.
So the trade-off is not visible versus invisible. It's your version of the facts versus everyone else's. Block the search crawlers and engines lose the one source you control: your pricing page, your product names, your current positioning. They still answer. They answer from older training data and from pages you didn't write. Our guide to third-party mentions and AI answers covers that second channel in depth.
What each bot block actually removes
Each vendor runs several bots with different jobs. Blocking one does not block the others. As of October 2026, the vendors' own crawler pages document these behaviors:
| Bot | What blocking it does | What still reaches answers |
|---|---|---|
| GPTBot | Signals your content should not train OpenAI models | ChatGPT search, which uses OAI-SearchBot |
| OAI-SearchBot | Your site is not shown in ChatGPT search answers | Navigational links, third-party pages, training data |
| ChatGPT-User | Unreliable: OpenAI says robots.txt "may not apply" to user-initiated actions | User-requested fetches of your pages |
| ClaudeBot | Training use by Anthropic | Claude-SearchBot and Claude-User are separate agents |
| PerplexityBot | Removes you from Perplexity's search index | Perplexity-User, which "generally ignores robots.txt rules" |
| Google-Extended | Cuts Gemini Apps and Vertex training and grounding | Google Search, including AI Overviews and AI Mode |
| Googlebot | Removes you from all Google Search surfaces | Nothing from your site on Google; a very costly block |
| Meta-WebIndexer | Removes you from Meta AI search | Third-party pages; Meta says allowing it "helps us cite and link to your content" |
OpenAI states that "each setting is independent of the others." That's why blocking GPTBot while staying in ChatGPT search is a common setup. There's no robots.txt token that blocks Google AI Overviews alone. Googlebot is the control, so blocking Google-Extended doesn't change AI Overviews.
Vendors change these pages. Recheck them before you ship a robots.txt change.
What the evidence shows, and what nobody has measured
The access side is well documented. The outcome side is not.
Training cutoffs are real and dated. OpenAI's model docs list a Feb 16, 2026 knowledge cutoff for the GPT-5.6 family and Aug 31, 2025 for the Instant path ChatGPT also uses. Anything you published after a model's cutoff is absent from its answers when it doesn't search. A knowledge cutoff check shows which of your announcements fall past each line.
The effect of a training block is unmeasured. Nothing published tests whether excluding GPTBot or ClaudeBot weakens what a future model remembers about your brand. OpenAI doesn't claim it either way. The honest label is unknown, not harmless. See whether training blocks affect brand recall.
Retrieval is per query. Engines decide whether to search for each question. You can't tell from outside whether a given answer came from training data or from live pages. Any single score blends both.
Edge cases worth checking
Your firewall may block more than robots.txt does
A CDN or bot-management rule can turn away a crawler that robots.txt allows. OpenAI's help docs say eligibility needs both robots.txt permission and a host that accepts its published IP ranges. The reverse also happens: you think you've blocked a bot, but only robots.txt says so.
Some blocks are partial by design
OpenAI says sites opted out of OAI-SearchBot "can still appear as navigational links." Read what ChatGPT can still link to before assuming a block hides you.
Other people's pages can misrepresent you
When your site is gone from the engine's sources, third-party pages carry more weight. Some contain outdated facts. A few carry injected instructions aimed at AI agents. That's a real reputational risk, and scanning third-party pages for injected instructions is the defensive response. Never use the same tactic yourself.
What to do next
- Decide per bot, not per vendor. Training and search are separate choices. Many brands block training bots and allow search bots so their own facts stay in live answers.
- Read your robots.txt per agent. Confirm what OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer and Googlebot each get. Blanket AI blocks often remove search eligibility by accident.
- Test reachability, not just rules. Request a page as each crawler and look for 403s or challenge pages.
- Use snippet controls for precision. If you want specific content kept out of AI answers,
nosnippet,max-snippet:0anddata-nosnippetdo that without leaving the index. - See what engines say now. The free checker on this site shows whether AI names you for up to three buyer questions. Compare what it says with your own pages, then fix the facts on third-party sources you can influence.
None of this guarantees a mention. It decides whether your own version of the facts is available when an engine looks.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot governs training use only. ChatGPT search uses OAI-SearchBot, and OpenAI states that each setting is independent of the others. You can disallow GPTBot to keep your content out of training while allowing OAI-SearchBot so your pages can appear in ChatGPT search answers. Blocking GPTBot doesn't stop ChatGPT from answering about you from older training data or third-party pages either.
Can I stop AI Overviews from using my content without leaving Google?
There's no robots.txt token for AI Overviews or AI Mode. Googlebot controls all Google Search surfaces, and Google-Extended doesn't affect Search inclusion. To limit what appears, Google documents nosnippet, max-snippet and data-nosnippet as the on-page controls. Google says a page must be indexed and eligible for a snippet to appear as a supporting link, so snippet controls narrow that eligibility.
Will blocking AI crawlers stop AI from getting my facts wrong?
Usually the opposite risk applies. Blocking search crawlers removes your own pages from the sources an engine can retrieve. The model then relies on training data frozen at a cutoff and on third-party pages you don't control. If those are outdated or wrong, the answer will be too. Keeping your key facts fetchable gives the engine a correct sentence it can quote.
Do user-triggered AI fetchers respect robots.txt?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiates the request. Perplexity says Perplexity-User generally ignores robots.txt rules. Meta documents that Meta-ExternalFetcher may bypass it. If you need to stop these fetches, robots.txt alone isn't enough, and blocking at the firewall has its own trade-offs for search crawlers you may want to keep.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.