SEO for AI Agents: How User-Triggered AI Fetchers Read Your Site

A user-triggered fetcher is an AI agent that requests a web page because a person asked it to, in that moment. Someone pastes your URL into ChatGPT, or asks Claude or Perplexity to check a page, and the assistant sends a request to your server.

The three you'll see most in logs are:

• ChatGPT-User from OpenAI
• Claude-User from Anthropic
• Perplexity-User from Perplexity

Each vendor also runs separate bots that crawl ahead of time. OpenAI runs OAI-SearchBot for ChatGPT search and GPTBot for training. Anthropic lists ClaudeBot (training) and Claude-SearchBot (search quality) as distinct from Claude-User. Perplexity runs PerplexityBot for its search index.

That split matters. Index-time crawlers decide what an engine has stored before anyone asks. Answer-time fetchers decide what the assistant can read right now. They are governed by different rules, and the difference between index-time crawlers and answer-time fetchers explains most of the confusion site owners run into.

The sequence is simple, and every step can fail silently.

1. A user asks the assistant something that needs a live page, or gives it a URL directly.
2. The assistant decides to fetch. This is discretionary. The model or the product decides, not you.
3. The fetcher sends an HTTP request with its own user-agent string.
4. Your CDN, firewall and server respond. That might be a 200 with your page, a 403, a rate-limit 429, or a JavaScript challenge page.
5. The assistant reads whatever text came back and uses it in the answer.

Step 5 is where rendering bites. If your key facts only appear after JavaScript runs, the fetcher may get a near-empty shell. Step 4 is where network rules bite. If the firewall turns the request away, the assistant reads an error page or nothing at all.

For a walk-through of this from the user's side, see what happens when someone asks ChatGPT to read your page.

One thing this fetch does not do: it doesn't add you to a search index. OpenAI states that ChatGPT-User is "not used to determine whether content may appear in Search." A successful fetch serves one answer for one user.

robots.txt is a set of per-bot requests. Index-time crawlers from the major vendors document that they follow it. User-triggered fetchers are a different case, and the vendors say so in their own docs (verified August 2026).

The reasoning the vendors give is that the fetch stands in for a person visiting your page, so it is treated more like a browser than a crawler.

What it means for you: a Disallow line for ChatGPT-User or Perplexity-User is a stated preference, not an enforced block. If you need a page to be unreachable, robots.txt is the wrong tool. If you want a page to be readable, robots.txt is not your main problem either. Rendering and network access are.

Be plain with yourself about the boundary.

In your control

• What the raw HTML contains. Whether your prices, product names and key claims are in the first server response.
• Whether the request gets through. Your CDN, WAF and bot-management rules decide whether a fetcher receives your page or a 403.
• Whether robots.txt itself is reachable. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file".
• Whether your facts agree with each other. If your pricing page and your footer state different numbers, the assistant may quote either.

Not in your control

• Whether a user asks about you.
• Whether the assistant decides to fetch your page or answers from memory.
• What the model writes after reading your page.
• Whether a given fetcher honours robots.txt.

Nothing you do here guarantees a mention or a citation. The access work removes blocks. It makes sure that when an assistant does come, it can read what you meant it to read.

The strongest evidence in AI access work concerns JavaScript. Vercel's crawler study found that ChatGPT and Claude crawlers download JavaScript files (11.50% and 23.84% of their requests) but "don't execute them. They can't read client-side rendered content." Independent re-testing in June 2026 found that none of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent or Bytespider rendered JavaScript. Googlebot and AppleBot are the exceptions.

Claude-User and Perplexity-User weren't in that tested set. Treat them as non-rendering until you've checked, since that's the safe assumption.

What to do:

• Fetch your key pages with JavaScript disabled and read the raw HTML.
• Compare it with the rendered page. List the facts that only appear after hydration.
• Move those facts into server-rendered or prerendered HTML. Prices, plan names, product specs, opening hours and one-line definitions matter most.
• Check content behind tabs, accordions and "load more" buttons that fetch text after page load. Same problem, different fix.

A specific fact stated in fetchable HTML gives the assistant a sentence it can quote. That's the only on-page content signal with any experimental support. You can see your site the way an AI crawler sees it with a raw fetch and a text diff.

robots.txt is a request. Your firewall is enforcement. A CDN or WAF can turn away an AI agent that robots.txt allows, and the agent's operator never tells you.

OpenAI's own help page describes ChatGPT search eligibility as a conjunction: allow OAI-SearchBot "and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses." The same logic applies to ChatGPT-User and other fetchers. Bot-management systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors," and JavaScript challenges or CAPTCHAs block automated clients by design.

Points to check, as of October 2026:

• Status codes per agent. Request a real page with each fetcher's documented user-agent. A 200 with full content is a pass. A 403, 429 or challenge page is a fail.
• CDN defaults you didn't set. From 15 September 2026, Cloudflare blocks its Training and Agent bot categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search stays allowed. Check which category your dashboard puts each fetcher in.
• IPv6-only edges. OpenAI's published IP files listed IPv4 ranges only when verified in August 2026. An AAAA-only origin can't be reached from them.
• Verification, not guesswork. OpenAI publishes IP ranges but warns against IP-only allowlisting. Cloudflare's verified-bot criteria combine a stable user-agent with a published IP list, reverse DNS or a Web Bot Auth signature.

The OpenAI crawlers, robots.txt and firewalls page covers the OpenAI side in detail.

Work from the gate inward. There's no point fixing rendering on a page the firewall never serves.

1. List the pages that matter. Pricing, product, about and the pages that answer your buyers' main questions. Ten to twenty is enough to start.
2. Test reachability per agent. Request each page as ChatGPT-User, Claude-User, Perplexity-User and the index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Record the status code and whether the body is your page or a challenge. The AI Crawler Checker does this per crawler and shows which ones your firewall turns away.
3. Fix network blocks first. Adjust WAF and bot-management rules so verified AI agents get a 200. Confirm robots.txt itself returns 200 to them.
4. Diff raw HTML against the rendered page. Note every key fact that appears only after JavaScript runs.
5. Server-render the load-bearing facts. Start with prices and product names.
6. Reconcile contradictions. Make sure the same fact reads the same everywhere on your site.
7. Set your robots.txt policy for index bots deliberately. Remember the split: OAI-SearchBot controls ChatGPT search, GPTBot controls training, and each setting is independent.
8. Re-test after every CDN or framework change. These failures come back quietly.

If an engineering team owns steps 3 and 5, developer tickets for AI access fixes shows how to write them so they get done.

• Relying on robots.txt to block user fetchers. Vendors say it may not apply. If content must stay private, put it behind authentication.
• Blocking ChatGPT-User to stay out of ChatGPT search. It doesn't do that. ChatGPT-User isn't used for search inclusion. OAI-SearchBot is.
• Allowing GPTBot to get found. GPTBot governs training only. OpenAI asserts no retrieval benefit from it.
• A blanket "block all AI" rule. The most common real error is a single rule that also removes OAI-SearchBot and PerplexityBot, which silently drops you from those search indexes.
• Assuming a robots.txt allow means access. The firewall can still return 403.
• Testing only from a browser. Your browser runs JavaScript and passes challenges. The fetcher does neither.
• Serving fetchers different content. Cloaking for AI agents, hidden text and embedded instructions aimed at the model are manipulation. Research on LLM search agents (arXiv:2606.16821) shows injected content can sway answers, which is exactly why it's a trust and reputation risk. Don't do it.
• Chasing markup instead of access. Schema, llms.txt and "AI-friendly" rewrites have no credible evidence of getting a page read or cited. Access and rendering do the work here.