If I block an AI crawler now, does AI forget pages it already crawled?

A robots.txt block is an instruction about future crawling. It does not reach into a model that has already been trained, and it does not rewrite a dataset that was collected before you added the rule.

It helps to split AI answers into two channels:

• Parametric answers come from what the model learned in training, with no live web access. That knowledge is frozen at the model's cutoff date. Nothing you change on your site this quarter reaches it.
• Search answers come from pages the engine fetches or holds in a search index at answer time. Here a crawler block has a documented effect.

So the honest answer has two halves. For training, blocking stops new collection by that bot, and no vendor documents removing past training data through robots.txt. For search, OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." That is the only part of "remove site from ChatGPT" that a crawler rule clearly controls.

Which bot you block decides what changes. Vendors run separate crawlers for training and for search, and the rules are independent. Our comparison of training crawlers vs search crawlers covers every vendor. The main ones, as documented in October 2026:

OpenAI puts it plainly: "Each setting is independent of the others." A site can block GPTBot for training and still allow OAI-SearchBot for search. If that is your goal, read whether to block GPTBot and still appear in ChatGPT search. A minimal version looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The same split exists at Anthropic, covered in Claude-SearchBot vs ClaudeBot.

A released model is a fixed snapshot. OpenAI's model documentation, checked in August 2026, listed a February 16, 2026 knowledge cutoff for the GPT-5.6 family and August 31, 2025 for the Instant path used in ChatGPT. Anything the model absorbed before its cutoff stays in that version until the vendor ships a new one.

Three things follow from that:

• Your block has no effect on current models. GPTBot's documented role is to signal that content "should not be used in training." It describes future use, not deletion.
• Any effect lands a model generation later. If your block changes anything in parametric knowledge, it shows up only when a model trained after your block is released.
• Nobody can measure the effect. No vendor exposes what is in a training set, and nothing published tests whether training exclusion changes what a model recalls about a brand. We cover that gap in whether blocking training bots makes ChatGPT forget your brand.

Be plain with yourself about what is out of your control here. You can set the directive. You cannot verify or force removal.

Even a correct block leaves some paths open.

User-triggered fetchers

When a person asks an assistant to open a link, a different agent fetches it. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because the action is user-initiated. Perplexity says Perplexity-User "generally ignores robots.txt rules."

Google's AI features

There is no robots.txt token for AI Overviews or AI Mode. Google says Google-Extended "does not impact a site's inclusion in Google Search." To keep pages out of Google's AI answers, the documented controls are noindex, nosnippet, max-snippet and data-nosnippet. See Gemini crawler controls.

Other sites that mention you

Blocking your own crawler access does nothing to third-party pages that describe your brand. Engines can still retrieve and cite those.

Firewall blocks

A firewall or CDN can block a crawler that robots.txt allows. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file." Use robots.txt as the stated signal, so the crawler can read your intent.

Decide what you actually want, then set the matching control.

1. Write down your goal per channel. Opt out of training, opt out of search, or both. They need different bots.
2. Set per-bot rules. Our per-bot robots.txt guide lists the tokens and their documented consequences.
3. Check for accidental blocks. From September 15, 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers. You may already be blocking bots you never chose to block.
4. Test what crawlers actually receive. robots.txt and your firewall can disagree. The free AI crawler checker requests a page as each AI crawler and shows which ones get turned away.
5. Expect no timeline. No vendor publishes how quickly search answers stop showing a page after a block, or when a new model drops old data. Recheck periodically rather than assuming.

Can I get my content removed from ChatGPT's existing training data? Not through robots.txt. Disallowing GPTBot tells OpenAI your content should not be used in training, which governs future collection. OpenAI's crawler documentation does not describe robots.txt as a way to remove data from models already trained. A current model keeps what it learned until a newer model replaces it, and no one outside OpenAI can verify what any training set contains.

Does blocking GPTBot remove my site from ChatGPT search? No. GPTBot governs training only. ChatGPT search uses OAI-SearchBot, and OpenAI states each crawler setting is independent of the others. If you block GPTBot but allow OAI-SearchBot, you stay eligible for ChatGPT search answers. If you want out of search, you need to disallow OAI-SearchBot, and OpenAI notes your site can still appear as navigational links.

How quickly does ChatGPT stop showing my site after I block OAI-SearchBot? OpenAI does not publish a timeline. Its documentation says opted-out sites will not be shown in ChatGPT search answers, without saying how fast that takes effect for pages already in its index. The practical move is to set the rule, confirm the crawler can actually read your robots.txt, and check your search appearances again over the following weeks.

Will blocking Google-Extended keep my pages out of AI Overviews? No. Google says Google-Extended controls training and grounding for Gemini Apps and Vertex AI, and does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search, and no robots.txt token targets them separately. The documented controls are Googlebot rules plus noindex, nosnippet, max-snippet and data-nosnippet on the page.

Does robots.txt stop an assistant from opening my page when a user shares the link? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiated the request. Perplexity says Perplexity-User generally ignores robots.txt rules. These user-triggered fetchers are separate from the crawlers that build search indexes or training sets, so a robots.txt block does not cover every way an assistant can read your page.