ClaudeBot, Claude-SearchBot and Claude-User Explained: Anthropic Crawlers and robots.txt

Anthropic's help page on crawling (updated 7 April 2026) describes three agents. They do different things, and treating them as one is the most common mistake.

The split matches how other vendors work. OpenAI separates GPTBot (training) from OAI-SearchBot (ChatGPT search) and states that each setting is independent of the others. The training versus search crawler comparison lines up the equivalents across vendors.

What this means for you: decide on training and search separately. Many sites want to opt out of training but still be eligible as a source in search answers. Those are two rules, not one. For the definitions in short form, see what ClaudeBot is and what Claude-SearchBot does.

robots.txt works per user agent, following the Robots Exclusion Protocol (RFC 9309). A crawler follows the most specific group that names it. If no group names it, it falls back to the * group.

Say you want to keep your content out of training but stay reachable for Claude's search. The rules look like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Three practical points:

• Name the agents exactly. ClaudeBot, Claude-SearchBot and Claude-User are the tokens Anthropic documents. A rule for a token Anthropic does not use does nothing.
• Watch blanket AI blocks. A rule copied from a list of "AI bots" often blocks the search agent along with the training one. The result is a silent loss of search eligibility.
• Check your * group. If it disallows everything, every Anthropic agent you did not name inherits that block. The page on bots robots.txt does not mention covers the fallback in detail.

Allowing Claude-SearchBot makes your pages eligible. It does not guarantee Claude retrieves them for any question, or cites them.

Some teams block AI crawlers at the firewall instead of in robots.txt. Anthropic warns that IP blocking impedes its ability to read your robots.txt file.

That has a consequence people miss. If the crawler cannot fetch /robots.txt, it cannot see the rules you wrote. Your carefully separated allow and block lines stop mattering, because the network layer has already decided.

The same gap runs the other way. A firewall, CDN or bot-management tool can block a crawler that robots.txt allows. The operators document this mechanism. How often it happens across the web is not measured. The page then looks allowed on paper and is unreachable in practice, and no dashboard tells you.

This risk changed recently. From 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts and all existing free-tier customers. Search stays allowed by default. A site can lose AI crawler access without anyone editing a file.

The fix: express your policy in robots.txt, and keep the network open enough for crawlers to read it. The question on whether a blocked IP can still read robots.txt walks through the trade-off.

Vercel's crawler study (published December 2024, updated February 2025) found that Claude's crawler fetched JavaScript files on 23.84% of its requests but did not execute them. In Vercel's words, these crawlers "can't read client-side rendered content." Independent re-testing in June 2026 found the same for ClaudeBot and Claude-SearchBot.

What that means: if your pricing, product names or key claims load only after JavaScript runs, Claude's crawlers may see an empty shell. This is the strongest evidence in the whole AI visibility field, and it is fully in your control.

The next move is a simple test. Fetch a key page with JavaScript disabled and compare it with the rendered page. Anything that appears only in the rendered version is at risk. Server-side rendering or prerendering of that content fixes it. This removes a block. It does not make Claude more likely to pick your page over others.

Be clear about the limits. Anthropic's documentation covers which agents exist and how to block them. As of October 2026, it does not cover several things people assume:

• Ranking factors. Anthropic publishes nothing on how Claude chooses which retrieved pages to use or cite.
• When Claude searches. On Claude.ai, web search is a user toggle with no published default. On the API, the integrator's system prompt steers whether Claude searches. Many Claude answers never touch your site at all.
• Training effects. Nothing published measures whether allowing or blocking ClaudeBot changes what Claude knows about your brand. The honest label is unmeasured.
• Citation as use. Anthropic's API cites text spans, with a cited_text field of up to 150 characters. A citation shows which text was quoted, not how much a page shaped the answer.

Answers given without search come from what the model learned before its training cutoff. Nothing you change this quarter reaches those. For the full reading of Anthropic's pages, see Anthropic's crawler documentation, digested.

Work through the access gates in order. Each one can silently remove you.

1. Read your robots.txt. Confirm what ClaudeBot, Claude-SearchBot and Claude-User each resolve to, including any * fallback.
2. Test the network layer. Request a real content page with each Anthropic user agent and record the status code. A 403 or a challenge page means the firewall disagrees with your robots.txt. The free AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.
3. Confirm robots.txt itself is reachable. If /robots.txt returns an error to the crawler, your rules are invisible.
4. Diff raw HTML against rendered HTML. Move load-bearing facts into the server response.
5. Repeat after CDN or security changes. Defaults change without notice.

The same checks apply to other vendors, with different tokens. See OpenAI crawlers, robots.txt and firewalls for the equivalent setup.

Does blocking ClaudeBot stop Claude from citing my site? Anthropic describes ClaudeBot as the agent that collects content that may be used for training. Claude-SearchBot is the separate agent tied to search quality. So a ClaudeBot block is a training decision by its documented role. If you want to remain eligible as a source in Claude's search, allow Claude-SearchBot explicitly and make sure your firewall does not block it. Eligibility still does not guarantee a citation.

Does Claude search the web for every answer? No. As of October 2026, web search on Claude.ai is a user toggle with no published default, and on the API the developer's system prompt steers whether Claude searches. When Claude answers without search, it relies on what it learned before its training cutoff. Your crawler settings only matter for answers where retrieval actually happens.

What happens if my robots.txt does not mention any Anthropic agent? Under RFC 9309, a crawler with no group naming it follows the `*` group. If your `*` group allows crawling, all three Anthropic agents are allowed. If it disallows everything, they are all blocked, including Claude-SearchBot. Naming each agent explicitly removes the guesswork and makes your intent readable to anyone auditing the file later.

Can my CDN block Claude crawlers even when robots.txt allows them? Yes. A firewall, CDN or bot-management rule can turn away a crawler that robots.txt allows, and Anthropic warns that IP blocking also stops it reading robots.txt. Cloudflare changed its defaults on 15 September 2026 to block Training and Agent crawlers on ad-displaying pages for some customers. Test with each user agent rather than trusting the file alone.

Does Claude's crawler run JavaScript? Vercel's study found Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them, and re-testing in June 2026 found the same for ClaudeBot and Claude-SearchBot. Content that appears only after JavaScript runs may be invisible to them. Put prices, product names and key facts in the server-rendered HTML.