Claude Visibility Factors, Graded by the Evidence

Claude can answer from two places: what the model learned in training, and what it fetches through web search. Those split into four stages, and only one of them is mostly yours.

• Parametric: what the model knows with no web access, frozen at its knowledge cutoff. Nothing you publish this quarter reaches it.
• Access: whether Anthropic's crawlers may fetch your page at all. This is in your control.
• Retrieval: whether your page enters Claude's context for a given query. Anthropic controls this, and the query Claude actually runs is not visible to you.
• Generation: which retrieved sources Claude uses and cites. The model controls this.

Retrieval is also discretionary. Claude decides whether to search, and an integrator's system prompt steers that choice. As of October 2026, web search in Claude.ai is a user toggle with no published default. That is one reason Claude answers differently in different apps. The practical move: fix the access stage first, because it is the only stage with documented, checkable causes.

Anthropic's support page (updated April 2026) names three agents with different jobs. ClaudeBot collects training data. Claude-SearchBot supports search quality. Claude-User fetches pages when a person asks. Each one reads its own robots.txt rules, so a blanket AI block can remove you from search when you only meant to opt out of training.

A hypothetical setup that allows search while opting out of training looks like this:

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

Two more gates sit beyond robots.txt.

JavaScript rendering

Vercel's crawler study found that Claude's crawlers fetched JavaScript files on 23.84% of requests but did not execute them, so they can't read client-side rendered content. Independent re-testing in June 2026 found the same for ClaudeBot and Claude-SearchBot. If your pricing, product names or definitions appear only after JavaScript runs, Claude's crawlers see a page without them. Server-side rendering or prerendering of that content is the fix.

Network reachability

A CDN, WAF or bot-management rule can return a 403 to a crawler that robots.txt allows. Anthropic warns that IP blocking also impedes its ability to read your robots.txt file. Nothing in any dashboard flags this. The check is to request a page as each crawler and compare the response with what robots.txt says; the AI Crawler Checker does that request for you.

Claude's API web search tool returns a field called page_age, which Anthropic's documentation describes as "When the site was last updated". That tells you Claude can see a date. It does not tell you Claude prefers recent pages. Anthropic does not state that page_age influences which results get used or cited.

The wider freshness evidence is thin. The main study linking updates to citation looked only at pages already cited, had no control group of uncited pages, and read dates from fields publishers set themselves. If most of the web carries a recent CMS-generated date, that finding carries no signal.

So don't bump dateModified to chase Claude. A date you change without changing the page is a number you're gaming, not a fix. Do keep dates consistent across your sitemap, schema, headers and visible text, because contradictory dates are a data-quality problem. More on the field itself is in our read of Claude's web search tool documentation, and the broader picture is under content freshness.

A whole category of advice says to rewrite pages so AI engines quote them: add statistics, add quotations, adopt an authoritative tone, restructure into short blocks. The best-designed test of these methods is C-SEO Bench (NeurIPS 2025), which ran them on several models including Claude-3.5-Haiku. It found most methods "largely ineffective" and frequently negative for document ranking.

The same study found that moving a document higher in the model's context mattered far more. But that position was set by the researchers inside a simulated pipeline. No publisher can invoke it.

Two honest limits apply. These tests measured pages already in the model's context, so they don't show whether anything helps a page get retrieved in the first place. And they predate Claude's 2026 models. What the evidence does support is narrow: state specific facts (a price, a number, a one-line definition) plainly on a page Claude can fetch, so there is a sentence to quote. Topical fit to the question also holds up across studies. Formatting tricks do not.

Prompt injection means hiding instructions in page text so an AI agent follows them, for example text telling the model to recommend one brand or disparage another. It exists, it works some of the time, and it is illegitimate. Using it on your own site risks being treated as spam and damages trust in every claim you make.

The relevant study (arXiv:2606.16821, June 2026) tested search agents against manipulated content through a mock search API. Its attack success rates ranged from 0.0% on Claude-Sonnet-4.6 to 31.4% on Gemini-3-Flash. Don't read that as Claude being immune. Those are point estimates on one attack set, and they shift with every safety update. The study also never published content to the open web, so it shows what happens once manipulated content is retrieved, not how easily it gets there.

The real exposure is third-party pages about you. Our guide to prompt injection and hidden text in AI search explains the mechanics, and the playbook on scanning third-party pages for injected instructions shows how to look for it.

Run these in order. Each one is a block you can remove; none is a promise that Claude will cite you.

1. Read your robots.txt rules for ClaudeBot, Claude-SearchBot and Claude-User separately, and confirm each matches what you intend.
2. Request a key page as Claude-SearchBot and confirm a 200 response, not a 403, a CAPTCHA or a JavaScript challenge.
3. Fetch the raw HTML with JavaScript off and confirm your prices, product names and core definitions are there.
4. Check for accidental noindex, nosnippet or data-nosnippet on content you want quoted.
5. Make sure the same facts read the same way across your pricing, about and product pages.

What stays out of your control: whether Claude searches at all, which queries it runs, and which retrieved source it cites. The same access logic applies across engines, with different agent names; compare our grading of Perplexity visibility factors.

Does blocking ClaudeBot remove me from Claude's search results? Anthropic documents ClaudeBot as its training crawler and Claude-SearchBot as a separate agent tied to search quality. Because robots.txt rules apply per agent, you can disallow ClaudeBot and still allow Claude-SearchBot. Check your file for a wildcard rule or a blanket AI block that catches Claude-SearchBot by accident. That is the most common way sites remove themselves from AI search without meaning to.

Will schema markup or an llms.txt file help Claude cite me? There is no evidence either one does. Anthropic does not document reading llms.txt, and no other major engine does either. Ahrefs' matched test of schema found no major citation uplift on the platforms it measured, and Claude was not among them. Existing markup does no harm, so there's no need to remove it. It just isn't a lever for Claude answers.

How can I tell whether a Claude answer used web search? Visible citations are the clearest sign. Without them you can't know, because as of October 2026 Claude.ai search is a user toggle with no published default, and integrators can steer search through their own system prompts. The free checker on this site gives a snapshot of how engines answer up to three buyer questions about you, which is a starting point rather than continuous tracking.

Can a competitor plant text that changes what Claude says about my brand? Research shows manipulated content can sway search agents once it is retrieved, though results vary by model and shift with safety updates. The realistic defence is monitoring third-party pages that mention you for hidden instructions and reporting them to the host. Our answer on whether [a competitor can manipulate AI answers](/questions/competitor-manipulate-ai-answers) covers the options in more detail.