Claude AI SEO: How Claude Searches and Cites the Web

Claude can produce an answer about your brand two ways. It can answer from what the model learned in training, or it can search the web first and ground the answer in pages it retrieves.

Those two paths need different thinking:

• Training data (parametric). Frozen at the model's knowledge cutoff. Nothing you publish this quarter reaches it.
• Access. Whether Anthropic's crawlers may fetch your pages at all. This is yours to control.
• Retrieval. Whether your page enters Claude's context for a given question. Anthropic controls this, and you only influence it partly.
• Citation. Which retrieved text Claude quotes and credits. The model decides.

The July 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found that no reviewed technique shows a stable, cross-platform causal effect on discoverability. That is why this guide spends most of its time on access. It is the stage where your changes have a documented effect.

Anthropic's crawler help page (updated April 2026) separates three user agents. Treat them as three separate decisions in robots.txt. Our digest of Anthropic's crawler documentation covers the page in more detail.

The most common error is a blanket "block all AI" rule. It is usually written to stop training, and it quietly removes search eligibility too. If you want to keep training out but stay findable, write the rules per agent:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Allowing Claude-SearchBot buys eligibility. It does not buy retrieval or a citation.

Retrieval is not automatic. On claude.ai, web search is a user toggle, and Anthropic publishes no default for it. On the Claude API, web search is a tool that defaults on at the organization level, but that setting does not describe the consumer app.

Even with search available, Claude decides per question whether to use it. Developers who build on Claude also steer that choice through their system prompt. So the same question about your brand can get a grounded answer in one product and a training-data answer in another.

The queries Claude sends are also its own. Like other engines, it writes its search queries rather than passing the user's words through, which is why AI search rarely runs the exact question a user types. As for whose index sits behind those queries, Simon Willison reported in March 2025 that Anthropic listed Brave Search as a subprocessor. Our page on what search engine Claude uses tracks what is and is not confirmed.

One practical consequence: Claude is not gated by Google's index. In OtterlyAI's 2026 experiment, Google deindexed two AI-generated test sites, and Claude was among the engines that still cited them. Healthy Google indexing tells you nothing about Claude, and the reverse is also true.

Claude's web search tool returns each result with a URL, a title and a page_age field (when the site was last updated). When Claude cites, it attaches a cited_text span of up to 150 characters to the claim. Perplexity, by contrast, cites whole sources by number. Our explainer on why Claude and Perplexity cite differently compares the two.

What this means for you:

• A citation points at a sentence. If your price, founding year or one-line definition is buried in a paragraph of hedges, there is no clean span to lift.
• State specific facts plainly, on a page Claude can fetch. This is the only on-page practice with a positive experimental signal in our evidence base, and it only helps after your page is retrieved.
• page_age is exposed, not proven to be weighted. Anthropic does not say it affects selection. Bumping a date is not a lever.
• Citation is not the same as use. Research on attribution (arXiv:2606.28358) shows citation output can be changed without changing the sources. A missing citation does not prove Claude ignored you.

The Claude web search tool documentation shows every field a result carries.

In your control

• Per-agent robots.txt rules for ClaudeBot, Claude-SearchBot and Claude-User.
• Whether your firewall, CDN or bot manager lets those agents through.
• Whether key content exists in the raw HTML. Vercel's crawler study found Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them. Independent testing in June 2026 found Claude-SearchBot and ClaudeBot still do not render JavaScript.
• Whether your own site states each fact once, consistently, in a sentence worth quoting.

Not in your control

• Whether a user turns search on, or whether Claude chooses to search.
• The queries Claude writes, and where your page lands in its context.
• What the model learned before its cutoff.
• Which retrieved source gets the citation.

Be wary of anyone who promises a Claude ranking or citation. No position data exists for Claude answers, and Anthropic does not publish ranking factors.

Run these in order. Each one can silently zero out everything after it.

1. Read your robots.txt per agent. Confirm Claude-SearchBot and Claude-User are allowed, and that a wildcard or blanket AI rule is not overriding them. Decide on ClaudeBot separately.
2. Request a real page as each agent. A CDN or firewall can return 403 or a challenge page to a crawler that robots.txt allows. Anthropic also warns that IP blocking stops it from reading your robots.txt at all. The free AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.
3. Check your CDN's AI settings. As of 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. Confirm which category your setup puts each Anthropic agent in.
4. Diff raw HTML against the rendered page. Fetch the page with JavaScript off and compare it with what a browser shows. Any price, product name or claim that only appears after hydration is invisible to Claude's crawlers.
5. Fix contradictions on your own site. If your pricing page, footer and about page disagree, there is no single correct sentence to quote.
6. Then look at the answers. Ask Claude your buyer questions with search on and off, and compare. Our guide to tracking Claude's answers about your brand covers doing this repeatably.

• Blocking ClaudeBot and thinking you've blocked Claude search. ClaudeBot is the training agent. Search runs through Claude-SearchBot.
• Allowing everything in robots.txt and stopping there. robots.txt is necessary, not sufficient. A firewall 403 removes you just as completely.
• Adding schema or an llms.txt for Claude. Anthropic does not document reading llms.txt. Ahrefs' matched test found adding schema produced no major uplift in AI citations on any platform it measured. Existing markup does no harm, but it is not a lever.
• Rewriting pages in an "AI-friendly" style. C-SEO Bench (arXiv:2506.11097) found most citation-oriented rewriting methods were largely ineffective and often harmful.
• Reading one Claude answer as the truth. A logged-in answer with search off reflects training data, not your current site. Note the retrieval state every time you test.
• Planting instructions for AI in hidden text. Prompt injection and cloaking are illegitimate. Research on search agents (arXiv:2606.16821) shows manipulated content can sway answers, which is exactly why engines treat it as an attack.

The same access-first order applies on other engines. Our Perplexity SEO guide walks through its crawlers.