Brave Search Crawler and robots.txt: What Is Known and How to Check Your Logs

The short version: nothing in our evidence base comes from Brave itself. As of October 2026, we hold no Brave page that names a crawler user agent, publishes IP ranges, or states which robots.txt token Brave honours.

That gap matters. For other vendors, the per-bot consequence of a robots.txt rule comes straight from the operator. Without that, any claim about how Brave reacts to a given rule is a guess.

So this page does three things:

• It states the one Brave-related fact the evidence supports.
• It shows what complete crawler documentation looks like, so you can judge whatever Brave publishes.
• It explains how to identify and verify Brave traffic in your own logs.

If you find a Brave crawler page, read it for four things: the user agent token, the purpose of each bot, published IP ranges, and whether user-triggered fetches follow robots.txt.

Simon Willison reported in March 2025 that Anthropic's Trust Center had added Brave Search as a subprocessor, in connection with Claude's web search. That is the link between Brave and AI answers on record.

What it means: an index you do not control directly may sit upstream of some Claude answers that search the web. What it does not tell you is how Claude's search and Brave's index interact today, or how much weight either carries.

Anthropic documents its own crawlers separately. ClaudeBot is for training, Claude-SearchBot is for search quality, and Claude-User is user-triggered, per Anthropic's support article. Our page on ClaudeBot, Claude-SearchBot and Claude-User covers what each one controls.

The next move: treat Brave as one more index to stay reachable in, not a lever with a known effect. Retrieval is per query and discretionary on every platform, so being reachable buys eligibility, not citations.

Other operators show the standard to hold Brave's documentation against. OpenAI is the clearest case, digested in OpenAI's crawler documentation.

If Brave's documentation covers fewer of these rows, you know less about what your rules do. That is a reason to rely more on log evidence, not less.

Your access logs are the one place you can see Brave traffic without relying on anyone's claims. Work through these steps.

1. Export a week or more of raw access logs from your origin and, if you use one, your CDN. CDN logs show requests the edge blocked before they reached you.
2. List every distinct user agent string and sort by request count. Look for any string that identifies itself as Brave or as a search crawler you do not recognise.
3. Check whether it fetched /robots.txt. A crawler that reads your rules requests that file. One that never does is either new to you or not following the protocol.
4. Verify the identity. User agent strings are easy to fake. Cloudflare's verified-bot criteria are a useful test: "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS." If Brave publishes IP ranges or a reverse DNS pattern, match the requesting IPs against it.
5. Record the status codes. A 200 means the page was served. A 403, a 429 or a challenge page means your firewall or CDN turned the bot away, whatever robots.txt says.
6. Compare the paths it requested with what you want indexed. If it only hits your homepage, or only hits pages you meant to keep private, that shapes your next rule.

If a request claims to be Brave but fails verification, treat it as an unidentified bot. Our guide on whether to block AI crawlers you don't recognise covers that decision.

robots.txt works per bot. Under RFC 9309, a crawler follows the group whose User-agent line matches its token, and falls back to the User-agent: * group if none does.

That gives you a safe default. If you have not confirmed Brave's token, Brave falls under your * group. Read what happens when robots.txt doesn't mention an AI bot before you assume it is blocked or allowed.

To block Brave specifically, you need the exact token from Brave's own documentation. A guessed token blocks nothing. Matching details also matter, covered in whether AI bot names in robots.txt are case-sensitive.

Think before a blanket block. A rule that disallows every bot except Googlebot can remove you from AI search indexes you never meant to leave. The trade-offs are in does blocking all AI bots hide me from AI search.

For crawlers that are documented, the robots.txt Checker and AI Crawler Checker at 99Visibility's free AI crawler tool show which ones your rules and firewall let through.

You control three things: what your robots.txt says, what your firewall and CDN return to each bot, and whether your key content is in the HTML before JavaScript runs. Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Brave's rendering behaviour is not in our evidence base, so server-rendering what matters is the cautious choice.

You do not control whether Brave indexes you, how its index feeds any AI product, or whether an AI answer cites your page. Nobody outside the operator can see that.

The next move is simple. Check your logs, verify what you find, keep your * group deliberate, and revisit when Brave publishes documentation you can read for yourself.

What is the Brave Search user agent? Our evidence base does not contain a Brave-published user agent string, so we don't quote one. Strings circulate on third-party sites, but a token you can't trace to Brave's own documentation is a weak basis for robots.txt rules. Check your access logs for requests that identify as Brave, then confirm the token on Brave's pages before you write a rule that names it.

How do I block the Brave bot? You need two things: the exact token Brave documents for robots.txt, and a way to verify requests that claim to be Brave. Add a Disallow rule under that token once you have it. If verified Brave requests keep arriving after the rule, a firewall rule matched on the verified identity is the stronger control. Blocking on user agent alone is unreliable because strings are easy to fake.

Does blocking Brave remove my site from Claude's answers? Nobody can say from the evidence. Simon Willison reported in March 2025 that Anthropic added Brave Search as a subprocessor for Claude's web search, but how Claude's search uses Brave today is not documented in our sources. Anthropic's own crawlers, including Claude-SearchBot, have separate controls. Blocking one index is not the same as being removed from an answer engine.

If my robots.txt doesn't mention Brave, is it allowed? Under RFC 9309, a crawler with no matching group follows your User-agent: * group. If that group allows a path, an unlisted crawler that honours the protocol may fetch it. If it disallows everything, the crawler should stay out. Whether a specific Brave crawler honours robots.txt is something to confirm in Brave's documentation and your logs.

Can my firewall block Brave even if robots.txt allows it? Yes. Bot management, WAF rules and CDN defaults act before robots.txt matters, and they can return a 403 or a challenge page to a crawler your rules allow. OpenAI documents this conjunction for its own search bot: allow it in robots.txt and confirm the host or CDN lets its IP addresses through. Check status codes in your logs to see what actually happened.