Getting Cited by Claude: What You Control and What You Don't

Anthropic publishes three user agents, and each does a different job. As of October 2026, its help center describes them this way:

• ClaudeBot collects content that may be used to train Anthropic's models.
• Claude-SearchBot crawls to improve the quality of Claude's search results.
• Claude-User fetches pages when a person using Claude asks it to.

The practical point: these are independent controls. A blanket "block all AI bots" rule hits all three at once. If you want to stay out of training but remain reachable for search, you set them separately. The trade-offs are covered in whether to allow Claude-SearchBot if you block ClaudeBot and in the wider playbook on how to block AI training but stay in AI search.

A minimal robots.txt that opts out of training and keeps search crawling open looks like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

What Anthropic does not document is just as important. It publishes no ranking factors for Claude's citations. It does not state that allowing Claude-SearchBot leads to citation. Allowing the bot buys eligibility, nothing more.

robots.txt is only half the gate. Your CDN, web application firewall or bot-management layer can return a 403 or a challenge page to a crawler that robots.txt allows. Nothing in your analytics will flag it. The crawler simply gets nothing.

Anthropic adds a specific warning: IP blocking "impedes our ability to read your robots.txt file." So an aggressive network rule doesn't just block pages. It can stop Claude's crawlers learning what you allow in the first place.

This risk is not static. From 2026-09-15, Cloudflare began blocking Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can lose AI access without anyone on your team changing a setting.

What to do:

1. Read your robots.txt and confirm the rule for ClaudeBot, Claude-SearchBot and Claude-User is what you intend.
2. Request a real page with each Claude user agent and record the status code. A 200 passes. A 403, 429 or CAPTCHA page fails.
3. Check your CDN's AI crawler settings, especially if you're on Cloudflare.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Vercel's crawler study found that Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them, so it could not read client-side rendered content. Independent re-testing in June 2026 found the same for ClaudeBot and Claude-SearchBot. Googlebot renders JavaScript. Claude's crawlers, as tested, do not.

If your pricing, product names or one-line description load only after a script runs, Claude's crawlers see an empty shell. That's the strongest evidence-backed fix in this whole area, and it is entirely yours to make.

How to check: fetch the page with JavaScript disabled, then render it in a browser, and compare the text. Anything present only in the rendered version is missing for Claude.

Once the content is in the HTML, state specific facts plainly. A sentence with a real price, number or definition gives an engine something it can quote. That is the only on-page content signal with positive experimental support. It is not a formatting trick, and no markup, schema or llms.txt file has evidence of helping Claude cite you.

Three decisions sit outside your reach, and they explain why the same question can cite you one day and not the next.

The search toggle

In Claude.ai, web search is a user toggle, and Anthropic publishes no default for consumers as of October 2026. When search is off, Claude answers from what it learned in training, frozen at a knowledge cutoff. Nothing you publish this quarter reaches that answer.

The integrator's system prompt

Many people meet Claude inside another company's product. On the API, the integrator decides whether to enable web search and writes the system prompt that steers when Claude uses it. The API's org-level default says nothing about Claude.ai consumer behavior.

The model's choice

Even with search on, Claude decides whether to search for a given question and which retrieved sources to cite. Research on generation-stage citation found that topical relevance and a document's position in the model's context matter most, and no publisher can set context position.

The critical survey of 45 GEO studies (arXiv:2607.14035, July 2026) put the limit plainly: no reviewed technique showed a stable, cross-platform causal effect on discoverability. Be wary of anyone promising Claude citations.

Claude's API cites text spans, with a cited_text field of up to 150 characters, rather than whole pages. Its web search results also expose page_age, described as when the site was last updated. Exposure is not weighting: Anthropic doesn't say page_age affects which source gets picked, so bumping a date field is not a strategy.

Two consequences for measurement:

• Citation is not the same as use. Mechanistic work on retrieval-augmented models (arXiv:2606.28358) showed citations can be added or removed without changing the sources. A missing citation doesn't prove Claude ignored you.
• Retrieval state is usually invisible. If an answer shows no citations, you often can't tell whether Claude searched. Any Claude visibility number mixes searched and unsearched answers.

Where does Claude's search get its candidate pages? Simon Willison reported in March 2025 that Anthropic listed Brave Search as a subprocessor. Anthropic doesn't document how its retrieval pipeline ranks sources, so treat the index question as open. Compare this with how ChatGPT handles citations and Perplexity's controls, where the crawler names and rules differ.

When Claude cites other sites for your category instead of yours, being named on those sites is the route into the answer. The evidence shows a strong association between brand prominence and citation, with no promised effect size and no lever to pull directly.

Say you sell scheduling software and Claude keeps citing two comparison articles. Getting accurately listed on those pages puts your name inside sources Claude already uses. That's slower than a robots.txt edit, but it works with the model's choices rather than against them. The guide to third-party mentions covers how to find and approach those pages.

One warning: hidden text and prompt injection aimed at AI agents exist, and research shows they can sway some agents once the content is retrieved. They are illegitimate, results vary by model and change with safety updates, and they put your reputation at risk. Don't use them. Do watch third-party pages about your brand for them.