User-triggered fetcher: definition

A user-triggered fetcher is an AI agent that requests a URL because a user, in that conversation, asked for it. The user pastes a link, asks the assistant to check a page, or asks a question the assistant answers by opening a specific URL.

It differs from two other kinds of AI bot:

• Training crawlers such as GPTBot collect pages that may be used to train models.
• Search or index crawlers such as OAI-SearchBot and PerplexityBot fetch and store pages ahead of time so they can surface in AI search answers.
• User-triggered fetchers fetch one page, once, on request.

Each announces itself with its own user agent string. As of October 2026 the named examples are ChatGPT-User, Perplexity-User, Claude-User and Meta-ExternalFetcher.

The sequence is simple. A user asks the assistant something that needs a live page. The assistant sends an HTTP request to that URL with the fetcher's user agent. Your server responds, and the assistant reads what comes back and uses it in the reply.

Three details shape what the assistant actually sees:

• robots.txt may be skipped. OpenAI's bots page says that because these actions are initiated by a user, robots.txt rules may not apply to ChatGPT-User. Perplexity's crawler docs say Perplexity-User generally ignores robots.txt rules. Meta documents that Meta-ExternalFetcher may bypass it too.
• The network layer still applies. Your firewall, CDN or bot management can return a 403 or a challenge page to the fetcher, whatever robots.txt says.
• JavaScript may not run. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Content that only appears after hydration may be missing from what the fetcher reads.

For a walk-through of a single request, see what happens when someone asks ChatGPT to read your page.

Index-time and answer-time controls are separate. robots.txt is the main tool for index-time crawlers, and those decide whether you can appear in AI search results at all. User-triggered fetchers sit outside that system: OpenAI states ChatGPT-User is not used to determine whether content may appear in ChatGPT search.

That has two consequences:

• Disallowing a fetcher in robots.txt is not a reliable block. If you need to stop answer-time fetches, the operators' own wording means robots.txt alone won't guarantee it.
• A fetch request does not mean you are indexed. Seeing ChatGPT-User in your logs tells you a user pointed ChatGPT at your page. It says nothing about OAI-SearchBot access.

The next move is to check what a fetch actually returns: status code, any challenge page, and whether your key facts (pricing, product names) are in the raw HTML. The free ChatGPT page fetch test shows this for one URL. The full comparison of the two control layers is in index-time crawlers vs answer-time fetchers.

• "Blocking GPTBot stops ChatGPT from reading my page." GPTBot governs training only. OpenAI says each bot setting is independent of the others.
• "Allowing ChatGPT-User gets me into ChatGPT search." Search inclusion runs through OAI-SearchBot plus network reachability, not through the fetcher.
• "A Disallow line keeps all AI agents out." Operators say user-triggered fetchers may not follow it. Treat robots.txt as necessary for index crawlers, not sufficient for every agent.
• "If the fetcher reached my page, it read my content." A 200 response can still carry a near-empty HTML shell if your content depends on client-side JavaScript.

For tactics and depth on serving these agents well, read the guide on SEO for AI agents.

• User agent: the name string every bot sends with its request.
• ChatGPT-User, Perplexity-User, Claude-User: the main user-triggered fetchers as of October 2026.
• GPTBot: OpenAI's training crawler, controlled separately.
• Index-time crawler: a bot that fetches and stores pages in advance, such as OAI-SearchBot or PerplexityBot.

To see which of these agents your firewall turns away, the AI Crawler Checker requests a page as each AI crawler and reports the responses.

Can I block a user-triggered fetcher? Not reliably through robots.txt, because OpenAI and Perplexity say their user-triggered fetchers may not follow it. Blocking at the firewall or CDN by user agent or verified IP is the stronger control. Weigh the trade-off first: a blocked fetcher means the assistant cannot read your page when a user explicitly asks it to, which usually means the user gets an answer built from other sources.

Does a ChatGPT-User hit in my logs mean I'm cited in ChatGPT? No. It means a user asked ChatGPT to open that URL. OpenAI says ChatGPT-User is not used to decide whether content appears in ChatGPT search. Search appearance depends on OAI-SearchBot being allowed in robots.txt and on your host or CDN letting OpenAI's search crawler through.

Do user-triggered fetchers run JavaScript? Vercel's study found ChatGPT and Claude crawlers download JavaScript files but don't execute them, and independent retesting in June 2026 found the same for ChatGPT-User and other AI agents. Googlebot is an exception. Put the facts you need an assistant to read, such as prices and product names, in the server-rendered HTML.

How is a user-triggered fetcher different from an index-time crawler? An index-time crawler fetches pages ahead of time so an engine can retrieve them later for any user. A user-triggered fetcher fetches one page, now, for one request. robots.txt is the documented control for the first. For the second, operators say robots.txt may not apply, so network-level rules and page rendering matter more.