Should I allow Claude-SearchBot if I block ClaudeBot?

Allow Claude-SearchBot if you want to stay reachable for Claude's search features and you only object to training use. Anthropic's help center (updated April 2026) lists three agents with different purposes:

• ClaudeBot crawls content that may be used to train Anthropic's models.
• Claude-SearchBot crawls to improve the quality of Claude's search results.
• Claude-User fetches a page when a person using Claude asks for it.

Each agent matches its own User-agent group in robots.txt. So a Disallow for ClaudeBot says nothing to Claude-SearchBot. That separation is the whole point: it lets you refuse one use without refusing the others. OpenAI works the same way, stating that "each setting is independent of the others" for GPTBot and OAI-SearchBot, which is covered in blocking GPTBot while staying in ChatGPT search.

For a fuller walkthrough of each Anthropic agent, see ClaudeBot, Claude-SearchBot and Claude-User explained. The same training-versus-search split exists at most vendors, laid out in training crawlers vs search crawlers.

Give each agent its own group. This example blocks training and allows search and user-triggered fetches:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Three details decide whether this works:

1. Named groups beat the wildcard. Under RFC 9309, a crawler follows the group that names it. If Claude-SearchBot has no group, it falls back to User-agent: *. A blanket Disallow: / under the wildcard then blocks it too.
2. Blanket AI blocks are the common mistake. Many sites copy a block list that includes every Anthropic token. That quietly removes search eligibility along with training.
3. Order inside the file does not matter. Matching is by user-agent name, not position.

If you want the same pattern across all vendors at once, the block training, keep AI search playbook applies it to OpenAI, Google, Perplexity and Meta as well.

robots.txt is a request, not a gate. The real gate is whether your server or CDN answers the crawler at all. Anthropic warns that blocking its IP addresses "impedes our ability to read your robots.txt file." If the crawler cannot fetch robots.txt, your careful per-bot rules never get read.

This happens without anyone editing robots.txt. Bot-management and WAF products can return a 403 or a challenge page to legitimate crawlers. Cloudflare changed its defaults on 15 September 2026: Training and Agent crawler categories became blocked by default on ad-displaying pages for new domains, new sites on existing accounts and all existing free-tier customers, while Search stayed allowed. How your zone classifies each Anthropic agent is worth checking in your own dashboard.

What to do: request a page and /robots.txt using each Anthropic user-agent and confirm you get a 200, not a 403, a 429 or a CAPTCHA. The free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. If you see a mismatch between the file and the response, fix the firewall rule first.

Allowing Claude-SearchBot removes a block. It does not earn a citation. Three stages sit between access and an answer, and you control only the first:

• Access (yours): the crawler can fetch and read the page.
• Retrieval (Anthropic's): Claude decides per query whether to search, and Claude.ai search is a user toggle with no published default.
• Generation (the model's): Claude picks which retrieved sources to cite.

The 2026 critical survey of 45 GEO studies (arXiv 2607.14035) found no technique with a stable, cross-platform causal effect on discoverability. Access is still the lever with the clearest mechanism.

Access also depends on what the crawler can read. Vercel's crawler study found Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them. Independent re-testing in June 2026 found the same for ClaudeBot and Claude-SearchBot. Prices, product names and key facts that render only in the browser are missing from what Claude-SearchBot sees. Serve them in the initial HTML.

• Blocking ClaudeBot now does not rewrite past models. A model trained before your block keeps what it learned until a later model replaces it. See whether AI forgets pages it already crawled.
• The training effect is unmeasured. Nothing published tests whether excluding ClaudeBot changes what Claude says about your brand in answers without search. Treat the training decision as a policy choice, not a visibility tactic.
• Blocking ClaudeBot alone does not remove you from Claude. Search and user fetches run on other agents. The details are in does blocking ClaudeBot remove my site from Claude's answers.
• Claude-User is a judgement call. It fires because a person asked Claude about a page. Blocking it means Claude cannot open your page even when a user points straight at it. Check Anthropic's help article for its current statement on how Claude-User treats robots.txt, since vendors differ here.

Next step: write the three groups, then verify with a live request as each agent. The per-bot robots.txt guide covers testing and the other vendors' tokens.