Claude Visibility Checklist: Five Checks From Access to Accuracy

Anthropic's support article on crawling (updated 7 April 2026) documents three agents. Each one does a different job:

• ClaudeBot collects content that may be used for training.
• Claude-SearchBot crawls to improve the quality of Claude's search results.
• Claude-User fetches pages when a user's request triggers it.

The most common mistake is a blanket AI block. Someone writes it to stop training and it also shuts out Claude-SearchBot. Say you want Claude's search to reach your pages but don't want them used for training. The rules would look like this:

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

Be clear about what allowing an agent buys you: eligibility. It doesn't buy retrieval, a citation or a position. Blocking ClaudeBot is a training decision. Nothing published measures whether it changes what Claude says about you when search is off.

Next move: open your robots.txt and find every group that matches these three names or *. Write down the effective result for each agent. The AI crawler access audit walks through the same check for every major engine.

robots.txt states a preference. Your CDN, WAF or bot manager decides who actually gets a response. A firewall can block a crawler that robots.txt allows, and the block is silent. Anthropic also warns that IP blocking "impedes our ability to read your robots.txt file". That means a firewall rule can stop Claude from seeing your crawl rules at all.

Run this test for each Claude agent:

1. Request a representative content URL using that agent's documented user-agent string.
2. Request /robots.txt the same way.
3. Record the response. A 200 passes. A 403 means bot mitigation blocked the agent. A 429 is rate limiting. A JavaScript challenge or CAPTCHA page is a fail.

Your settings can also change when you didn't touch them. From 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads. That applies to new domains, new sites on existing accounts and all existing free-tier customers. The Search category stays allowed by default. Check your CDN's bot settings even if nobody on your team has changed them.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. If you're unsure whether to do this before measuring anything, see why crawler access comes before tracking.

Vercel's crawler study found that Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them. Independent re-testing in June 2026 found that neither ClaudeBot nor Claude-SearchBot renders JavaScript. If your pricing, product names or key claims load only after the page hydrates, Claude may never see them.

How to check:

1. Fetch the page with JavaScript disabled and save the raw HTML.
2. Render the same URL in a headless browser and save the visible text.
3. Diff the two versions. List every price, product name, heading and claim that appears only after rendering.
4. Repeat for content behind tabs, accordions, "load more" buttons or infinite scroll that loads through a fetch call.

The fix is to server-render or prerender the content that matters. This is hygiene, not optimization. It decides whether your content can enter Claude's candidate set at all. It doesn't make Claude more likely to pick you from that set.

Claude's API cites text spans rather than whole pages. The cited_text field holds up to 150 characters. A specific fact stated plainly in the HTML gives the engine a sentence it can quote. Examples include a price, a founding year or a one-line definition of what you sell. This is the only on-page practice with a positive experimental signal in our evidence base.

Contradictions undo it. Your pricing page may say one number, the footer another and an old blog post a third. Claude can pick up any of them.

• List the 10 facts buyers ask about most: prices, plan names, locations, integrations, founding year, ownership.
• Find every place your site states each one, including structured data and PDFs.
• Settle on one current value and make sure it appears in the raw HTML.

Don't rewrite pages in an "AI-friendly" style. A critical survey of 45 GEO studies (arXiv, July 2026) found no technique with a stable, cross-platform effect on discoverability. It also warned that citation-oriented rewrites can impair retrieval. Nobody documents a writing style that Claude prefers.

As of October 2026, web search in Claude.ai is a user toggle and Anthropic publishes no default. In the API, the integrator's system prompt steers whether Claude searches, and Claude decides per query. With search off, Claude answers from training data frozen at a cutoff. Nothing you publish this quarter reaches that answer.

A test plan that holds up:

1. Write 5 to 10 questions a buyer would actually ask, such as "best payroll software for a 20-person agency".
2. Run each question with web search on. Record whether Claude names you, which URLs it cites and whether each fact it states is correct.
3. Run the same questions with search off. The difference shows what Claude knows about you without retrieval.
4. Log the date, the model and the retrieval state next to every answer.

Answers vary between runs, so treat each run as a sample. The free checker on this site gives a snapshot for three buyer questions. For an ongoing routine, fold these prompts into a quarterly AI visibility review.

So don't buy promises built on the right-hand column. No engine documents reading llms.txt. No Claude rank or position exists to track. Exposing page_age shows the signal is visible to Claude, not that Claude weights it.

One risk sits outside your site. Pages carrying hidden instructions aimed at AI agents can try to steer what an agent says about a brand. Researchers measure this as an attack. It is never a tactic to use.

The same order of work applies elsewhere, with different agent names. See the ChatGPT visibility checklist and the Perplexity visibility audit checklist.

Does blocking ClaudeBot remove me from Claude's search answers? Anthropic documents ClaudeBot as its training crawler and Claude-SearchBot as a separate agent for search quality. Blocking ClaudeBot is therefore a training decision. To keep search eligibility, allow Claude-SearchBot explicitly. Then check that no wildcard rule or firewall setting blocks it anyway. Nobody has measured whether blocking ClaudeBot changes what Claude recalls about you with search off.

Does Claude read llms.txt or schema markup? Anthropic doesn't document reading llms.txt, and no other engine does either. Google says its Search ignores the file. On schema, Ahrefs' matched-control study found that adding it produced no major uplift in citations on any platform it tracked. Existing markup does no harm, so there's no reason to remove it. Just don't expect it to get you cited.

Why does Claude describe my brand differently each time? Several things change between runs. Web search may be on or off. The model may differ. In API products, the integrator's system prompt steers whether Claude searches at all. With search off, Claude answers from training data with a fixed cutoff. With search on, it answers from pages it retrieves for that query. Record all of these whenever you test.

How often should I rerun this checklist? Rerun the access checks after any deploy that touches rendering, robots.txt, your CDN or your firewall. Rerun them after a domain or platform move too, and the [site migration checklist for AI crawler access](/playbooks/site-migration-ai-crawler-checklist) covers that case. Run the fact and prompt checks at least quarterly, or sooner after a pricing or product change.