Claude-SearchBot: definition and role

Claude-SearchBot is one of three user agents Anthropic documents in its help centre article on web crawling (updated 2026-04-07). Anthropic describes the split like this:

• ClaudeBot collects content that may be used for training.
• Claude-SearchBot crawls to improve the quality of search results in Claude.
• Claude-User fetches a page when a person using Claude triggers it.

The names look similar, but the jobs differ. That matters because each one reads its own section of your robots.txt. For the full breakdown of all three, see ClaudeBot, Claude-SearchBot and Claude-User explained.

Claude-SearchBot is an index-time crawler. It fetches pages ahead of time, which is the access stage of AI visibility. It does not decide what Claude says in an answer. Retrieval happens later, per query, and Claude decides when to search. As of October 2026, web search in Claude.ai is a user toggle with no published default.

Three mechanics are worth knowing:

• It follows per-bot robots.txt rules. A rule for ClaudeBot says nothing about Claude-SearchBot.
• It reads raw HTML. Vercel's crawler study found Claude's crawlers fetch JavaScript files but don't execute them, and independent re-testing in June 2026 confirmed Claude-SearchBot does not render JavaScript. Content that appears only after hydration may be invisible to it.
• It has to get past your network. Anthropic warns that IP blocking impedes its ability to read your robots.txt file. A WAF or bot challenge can turn it away before any rule is read.

Here is a minimal robots.txt that opts out of training and keeps search access:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

A ready-made version covering other vendors sits in our robots.txt template for AI search and training bots.

If Claude-SearchBot can't fetch your pages, Claude's search has less of your content to work with. Blocking it is a choice to step out of that channel. Allowing it removes a block, nothing more. No vendor promises that an allowed page gets retrieved or cited.

The most common real-world error is a blanket AI block. Someone adds a rule for every AI user agent to stop training, and search access goes with it. The fix is to separate training bots from search bots, a pattern covered across vendors in training crawlers vs search crawlers.

The next move is to check three things:

1. Your robots.txt has the rule you intend for Claude-SearchBot.
2. Your CDN or firewall returns a 200, not a 403 or a challenge page, to that user agent. The firewall testing playbook walks through this.
3. Your prices, product names and key facts are in the HTML before JavaScript runs.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

"Blocking ClaudeBot removes me from Claude"

Not by itself. ClaudeBot governs training collection. Claude-SearchBot is a separate agent with a separate rule. Our answer on whether blocking ClaudeBot removes your site from Claude's answers covers the edge cases.

"Allowing Claude-SearchBot gets me cited"

Allowing it makes you eligible. What Claude retrieves and cites for a given question is decided at answer time, and nobody outside Anthropic controls it.

"robots.txt is the whole story"

It is necessary, not sufficient. A network block overrides an Allow rule silently. And Claude-User, which acts on a person's request, is documented separately from the search crawler.

"Claude knows my site from training anyway"

What a model knows with search off is frozen at its knowledge cutoff. Pages published after it can only reach Claude through retrieval, which depends on search access.

• ClaudeBot: Anthropic's training crawler. Deciding between the two is covered in should I allow Claude-SearchBot if I block ClaudeBot.
• Claude-User: fetches pages when a person asks Claude to.
• OAI-SearchBot and GPTBot: OpenAI's equivalent search and training split.
• Googlebot: governs Google Search and its AI features. There is no separate robots.txt token for AI Overviews.

Can I block ClaudeBot but allow Claude-SearchBot? Yes. robots.txt rules apply per user agent, so you can disallow ClaudeBot to opt out of training collection and allow Claude-SearchBot to stay reachable for Claude's search. Write a separate User-agent group for each. Then confirm your firewall or CDN lets Claude-SearchBot through, because a network block overrides an Allow rule without telling you.

Does Claude-SearchBot run JavaScript? Independent re-testing in June 2026 found it does not, matching Vercel's earlier finding that Claude's crawlers download JavaScript files without executing them. If your prices, product names or key claims load only after JavaScript runs, Claude-SearchBot may not see them. Server-side rendering or prerendering that content puts it in the raw HTML the crawler reads.

Will allowing Claude-SearchBot get my content used for training? Anthropic documents training collection under ClaudeBot, not Claude-SearchBot. Keeping a Disallow rule for ClaudeBot is the documented way to signal that your content should not be collected for training, while a separate rule handles the search crawler. Review Anthropic's help centre article periodically, because crawler definitions can change.

How do I know whether Claude-SearchBot is visiting my site? Check your server or CDN logs for the Claude-SearchBot user agent and look at the status codes it received. A run of 403 or 429 responses, or challenge pages, means your network is turning it away. Logs are the ground truth, since robots.txt only states intent and cannot show what actually happened at the edge.