Index-Time Crawlers vs User-Triggered AI Fetchers: Which Control Works Where

AI platforms send two different kinds of bot to your site, and they do different jobs.

An index-time crawler visits on the engine's own schedule. It fetches pages, stores them, and the engine searches that stored copy later when someone asks a question. This is the access stage: whether your page can be in the candidate pool at all. OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer and Googlebot do this job.

An answer-time fetcher, also called a user-triggered fetcher, loads a single URL because a person in a chat asked the assistant to open it or look something up. The fetch belongs to one conversation. ChatGPT-User, Perplexity-User, Claude-User and Meta-ExternalFetcher are in this group.

A third group, training crawlers such as GPTBot, ClaudeBot and Google-Extended, feeds future models rather than search. They matter here only because people confuse them with the other two.

What it means: one robots.txt file is talking to three audiences, and the vendors do not treat them the same way.

As of October 2026, the first-party documentation draws a clear line between the two jobs.

OpenAI

• OAI-SearchBot: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• ChatGPT-User: OpenAI says it is "not used to determine whether content may appear in Search" and that "Because these actions are initiated by a user, robots.txt rules may not apply."
• On the settings overall: "Each setting is independent of the others."

The OpenAI crawler guide covers the firewall side in more detail.

Perplexity

Perplexity describes PerplexityBot as a search indexer that is "not used to crawl content for AI foundation models." It says Perplexity-User "generally ignores robots.txt rules." The PerplexityBot vs Perplexity-User breakdown goes through both.

Anthropic and Meta

Anthropic lists ClaudeBot (training), Claude-SearchBot (search quality) and Claude-User (user-triggered) as separate agents. Meta documents that Meta-ExternalFetcher may bypass robots.txt. Meta also gives the only first-party statement tying a crawler to citation: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

The stated reasoning is the same across vendors: a user started the fetch, so the vendor treats it as acting for that user rather than as crawling.

Reality: robots.txt is a reliable control for the index column and an unreliable one for the answer-time column. Plan accordingly.

If you need to stop, or confirm, answer-time fetches, these are the controls that sit in your hands.

Your firewall or CDN

A firewall rule applies whether or not a bot reads robots.txt. OpenAI publishes IP ranges for its bots, including a separate file for ChatGPT-User, but also warns against relying on IP-only allowlisting. Match on user agent plus your CDN's verified-bot programme instead.

The same mechanism cuts both ways. A CDN can block a crawler that robots.txt allows, and nothing in your dashboards will say so. From September 15, 2026, Cloudflare blocks its Training and Agent bot categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. Check which category your target fetchers fall into on your own zone. Anthropic also warns that IP blocking "impedes our ability to read your robots.txt file."

What the fetcher can read

Vercel's crawler study found ChatGPT and Claude crawlers download JavaScript files but "don't execute them." An independent retest in June 2026 found ChatGPT-User, OAI-SearchBot, PerplexityBot and others still do not render JavaScript. So a user who asks ChatGPT about your pricing page gets whatever is in the raw HTML. If the price loads client-side, the assistant may not see it. More on this in what happens when ChatGPT opens your URL.

Snippet controls

noindex, nosnippet, max-snippet:0 and data-nosnippet are documented by Google and Bing for their own search and AI surfaces. The evidence does not show whether third-party user-triggered fetchers honour them, so treat them as Google and Bing controls.

Every block is per bot and has its own consequence. Get the bot wrong and you lose the wrong thing.

• Block OAI-SearchBot: you drop out of ChatGPT search answers, except as navigational links. This is the costly one.
• Block GPTBot: OpenAI says this signals your content should not be used for training. It does not affect ChatGPT search.
• Block ChatGPT-User at the firewall: ChatGPT cannot read your page for the person who asked. Your search index status is unchanged, because OpenAI says ChatGPT-User does not decide Search inclusion.
• Block ClaudeBot: training only. Claude search uses Claude-SearchBot. See whether blocking ClaudeBot removes you from Claude.
• Block Google-Extended: cuts Gemini Apps training and grounding, and Google says it does not affect Search inclusion or ranking. The Google common crawlers digest explains the split.

The most common real-world error is a blanket AI block that catches retrieval bots along with training bots. A minimal split looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

None of this promises citations. Allowing a retrieval bot buys eligibility. Whether a page is retrieved and cited for a given question is the engine's call.

Pick the control by the outcome you want.

1. You want to be eligible for AI search answers. Allow the index-time bots in robots.txt: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer and Googlebot. Then confirm your CDN returns a 200, not a 403 or a challenge page, to each one. OpenAI's own wording is that eligibility needs both.
2. You want out of training but in search. Disallow GPTBot and ClaudeBot, keep the search bots allowed. Remember Google-Extended also cuts Gemini Apps grounding.
3. You want to stop assistants reading pages on request. Use firewall rules on verified user agents. robots.txt alone will not reliably do it. Accept that people asking about you will get answers without your page.
4. You want specific text kept out of Google and Bing AI answers. Use data-nosnippet or the robots meta snippet controls on those sections.
5. You want answer-time reads to be accurate. Put the facts that matter (prices, plan names, specs) in server-rendered HTML.
6. You want to know what is actually happening. Check server logs for each bot's status codes, or use the free AI Crawler Checker, which requests a page as each AI crawler and shows which ones your firewall turns away. Revisit the list when vendors add agents; this guide on new AI crawlers covers how often.

Does ChatGPT-User obey robots.txt? Not reliably. OpenAI's crawler page says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to decide whether content appears in ChatGPT search. If you need to stop it, use a firewall or CDN rule that matches the verified user agent. If you want it to succeed, make sure your CDN does not challenge it.

Can I block Perplexity-User? Perplexity says Perplexity-User generally ignores robots.txt rules, so a Disallow line is not a dependable block. A firewall rule is the control that applies regardless. Blocking it does not remove you from Perplexity's index, which PerplexityBot builds. Blocking PerplexityBot in robots.txt is the documented way to leave the index, and Perplexity says PerplexityBot is not used for model training.

If I block OAI-SearchBot, can ChatGPT still open my page when someone pastes the link? Possibly. OpenAI says each bot setting is independent, and that robots.txt may not apply to ChatGPT-User. Blocking OAI-SearchBot removes you from ChatGPT search answers, apart from navigational links, but it does not by itself stop a user-triggered fetch. To stop both, you need the robots.txt rule for OAI-SearchBot and a network rule for ChatGPT-User.

Is there a robots.txt token for Google AI Overviews? No. Google says AI is built into Search, so Googlebot rules in robots.txt are the control. To appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible for a snippet. Google-Extended governs Gemini Apps training and grounding, not Search. Search Console also has a generative AI inclusion control, which is on by default and can inherit from a parent property.

Do user-triggered fetches mean an engine indexed my page? No. An answer-time fetch serves one conversation. It is not the index-time crawl that decides whether your page can appear in other people's answers. Seeing ChatGPT-User in your logs tells you someone asked about the page. It does not tell you OAI-SearchBot can reach it, so check the index-time bots separately.