ClaudeBot: definition and what blocking it actually does

ClaudeBot is the user agent name Anthropic gives the crawler that gathers web content that may be used to train its models. It is a training crawler, not a search crawler.

As of October 2026, Anthropic's help article lists three agents with three jobs:

• ClaudeBot: collects content for training.
• Claude-SearchBot: crawls to improve the quality of Claude's search results.
• Claude-User: fetches a page when a person using Claude asks it to. This is a user-triggered fetcher.

Each name gets its own robots.txt rule. For the full breakdown of all three, read ClaudeBot, Claude-SearchBot and Claude-User explained.

ClaudeBot requests pages over HTTP and reads the raw HTML it gets back. Vercel's crawler study found Claude's crawlers fetched JavaScript files on 23.84% of requests but did not execute them. Content that only appears after JavaScript runs is likely invisible to it.

Robots.txt rules are matched per user agent under the Robots Exclusion Protocol (RFC 9309). This rule blocks training collection while leaving Claude's search crawler alone:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Use robots.txt, not an IP block. Anthropic warns that IP blocking impedes its ability to read your robots.txt file. If the crawler can't read the file, it can't see the rules you wrote. A firewall or CDN can also turn away a crawler your robots.txt allows, so check both layers. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.

AI answers come from two places. Some come from what the model learned in training, frozen at a knowledge cutoff. Others come from pages retrieved live at answer time.

ClaudeBot sits on the training side. Its effect lands, if at all, in a future model generation. Nobody has published a measurement of whether allowing it improves how a model recalls a brand, or whether blocking it hurts. The honest label is unmeasured.

Claude-SearchBot sits on the retrieval side, which is where a publisher's access decisions have a documented path into answers. So the practical question is rarely about ClaudeBot alone. It's whether a blanket AI block also caught Claude-SearchBot. If you're weighing that split, see whether to allow Claude-SearchBot if you block ClaudeBot.

One dated change to watch: from 2026-09-15, Cloudflare set Training crawlers to blocked by default on ad-displaying pages for new domains and existing free-tier customers. A site can lose training-crawler access without anyone editing robots.txt.

• "Blocking ClaudeBot removes me from Claude." It addresses training, not search. The detailed answer is in does blocking ClaudeBot remove my site from Claude's answers.
• "Allowing ClaudeBot gets me cited." No evidence supports that. Citation in a live answer depends on retrieval, which ClaudeBot does not govern.
• "A firewall block is stronger than robots.txt." Per Anthropic, an IP block can stop the crawler reading robots.txt at all. That makes your stated policy invisible.
• "Blocking ClaudeBot blocks Claude-User." They are separate agents with separate rules.

• GPTBot: OpenAI's training crawler, which works the same way. OpenAI states each of its bot settings is independent.
• Training crawlers vs search crawlers: the split across vendors in one comparison.
• Training crawler, user agent and user-triggered fetcher: the building-block definitions.

What is the ClaudeBot user agent? ClaudeBot is the name Anthropic's training crawler uses to identify itself, and the token you put in a robots.txt User-agent line. Check Anthropic's help article for the current full string, since vendors update these. Match rules on the token, and remember Claude-SearchBot and Claude-User are different tokens that need their own lines if you want to control them.

Does blocking ClaudeBot affect what Claude knows about my brand today? No. A model's trained knowledge is frozen at its cutoff, so a block you add now cannot change an existing model. Any effect would show up in a later model generation, and no one has measured how large that effect is, in either direction. Today's live answers depend on retrieval agents, not on ClaudeBot.

Should I block ClaudeBot? It depends on whether you want your content used for model training. That is a business and licensing choice, not a visibility tactic. If you block it, do it in robots.txt, not with an IP block, and confirm you did not block Claude-SearchBot by accident with a blanket rule.

Does ClaudeBot render JavaScript? Evidence says no. Vercel found Claude's crawlers downloaded JavaScript files but did not execute them, and an independent re-test in June 2026 found the same for ClaudeBot and Claude-SearchBot. Put the facts that matter, like pricing and product names, in the server-rendered HTML.