Anthropic's Crawler Documentation, Digested

This page digests one vendor document: Anthropic's help article "Does Anthropic crawl data from the web, and how can site owners block the crawler?" on support.claude.com. The version in our evidence base is dated 7 April 2026.

Anthropic wrote it, and Anthropic runs the crawlers it describes. That makes it the best available source on what each agent is for. It also makes it an assertion, not a measurement. Nobody outside Anthropic can verify how the agents behave internally.

The article matters because Claude's agents are split by purpose, the same way OpenAI splits GPTBot from OAI-SearchBot. If you have read our digest of OpenAI's crawler documentation, the structure will look familiar. The tokens are different, and they do not transfer between vendors.

The article is not a study. It has no sample, no experiment and no effect size. It is a list of user agents, a statement of what each one does, and instructions for blocking them.

We read it alongside two kinds of outside evidence:

• Crawler behaviour studies. Vercel's crawler analysis (published December 2024, modified February 2025) looked at what AI crawlers actually fetch. An independent re-test in June 2026 checked JavaScript rendering across the same crawler set.
• Anthropic's own API docs. The Claude web search tool documentation shows what a search result looks like once retrieval has happened.

The help article tells you what each agent is for. The outside evidence tells you what the agents can see. You need both to make a sound robots.txt decision.

As of October 2026, the article describes three agents. Our plain-English explainer on ClaudeBot, Claude-SearchBot and Claude-User covers each in more depth.

The second finding is the one most site owners miss. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file". If your firewall or CDN turns away Anthropic's traffic, the crawler may never see your Allow or Disallow rules at all. The file you edited carefully stops being the control.

Two findings from outside the article complete the picture:

• No JavaScript execution. Vercel found that Claude's crawler fetched JavaScript files on 23.84% of requests but did not execute them. The June 2026 re-test found that neither ClaudeBot nor Claude-SearchBot renders JavaScript.
• Citations are text spans. Anthropic's web search tool cites passages of up to 150 characters in a cited_text field, and exposes a page_age field for when a page was last updated. Exposing a field is not the same as weighting it. Anthropic does not say page age affects selection.

The article tells you which agent does what. It leaves several questions open, and you should not fill them with guesses.

• No effect of training exclusion. Nothing in the evidence base measures whether blocking ClaudeBot changes what Claude recalls about your brand in answers given without web search. The honest label is unmeasured, not harmless and not harmful.
• No promise from allowing search. Allowing Claude-SearchBot makes you eligible to be indexed. It does not mean Claude searches for a given question, retrieves your page, or cites it.
• No published search default for Claude.ai. Web search in the Claude app is a user toggle, and the evidence base records no published default. Many Claude answers may come from the model's training snapshot with no fetch at all.
• No stated rule for Claude-User and robots.txt. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules". Our record of Anthropic's article holds no equivalent statement either way. Read the live page before relying on a Claude-User block.
• No measure of how often firewalls block Anthropic. The mechanism is documented. The frequency is not.

The cross-vendor pattern is summarised in our comparison of training crawlers vs search crawlers.

Decide each agent separately

Work through three questions:

1. Training (ClaudeBot). Do you want your content used to train Anthropic's models? This is a licensing and policy call. The evidence gives no visibility reason to choose either way.
2. Search (Claude-SearchBot). Do you want to be eligible for Claude's search? If yes, allow it. Blocking it while allowing ClaudeBot is rarely what anyone intends. Our answer on whether to allow Claude-SearchBot if you block ClaudeBot walks through the trade-off.
3. User fetches (Claude-User). Do you want Claude to read a page when a user pastes your URL or asks about it? For most commercial sites the answer is yes.

A site that opts out of training but stays eligible for search would write:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Under the robots exclusion standard (RFC 9309), a crawler follows the most specific group that matches its name. A blanket User-agent: * block does not apply to an agent that has its own group, and an agent without its own group falls back to *.

Check that robots.txt is reachable

Anthropic's IP warning turns this into a two-part test:

1. Request /robots.txt with each Anthropic user-agent string. Record the status code.
2. Request a key content page the same way. A 200 is a pass. A 403, a 429 or a page holding a challenge or CAPTCHA is a fail.
3. Compare the raw HTML with the rendered page. Prices, product names and definitions that appear only after JavaScript runs are invisible to ClaudeBot and Claude-SearchBot.
4. If you use Cloudflare, review its AI crawler categories. Since 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and all existing free-tier customers. You can lose access without editing anything.

The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. If ChatGPT shows the same symptom, see why robots.txt can allow AI bots and still leave you out.