Mistral Crawlers and robots.txt: What Is Known

As of October 2026, the evidence base behind this library contains no first-party Mistral page on crawlers. So we can't tell you, from verified sources:

• which user agent tokens Mistral uses
• whether it separates training crawling from Le Chat search or user-triggered fetching
• whether its fetchers honour robots.txt
• whether it publishes IP ranges for verification
• which search index, if any, Le Chat's web answers draw on

That gap matters because other vendors document these details, and the details change what a robots.txt rule does. OpenAI, Anthropic and Perplexity each publish named bots with separate jobs. You can see how that looks in OpenAI's crawler documentation, digested and Perplexity's crawler documentation, digested.

The next move: find Mistral's own crawler page and read it directly. Copy the token exactly as written there. Don't copy a token from a blog post or a third-party bot list, because those go stale.

Robots.txt works per bot. Under RFC 9309, a crawler looks for the group whose User-agent line matches its product token. If no group matches, it follows the User-agent: * group. If there is no * group, nothing is disallowed.

What that means for Mistral today:

• If your * group allows everything, any Mistral bot that honours robots.txt may crawl.
• If your * group disallows everything, a compliant Mistral bot stays out, along with every other unnamed crawler.
• If you added a Mistral line from an unverified list and the token is wrong, the rule does nothing and the bot falls back to *.

We cover the fallback in more depth in whether an unlisted AI bot is allowed, and token matching in whether AI bot names are case-sensitive.

Once you have a token from Mistral's own documentation, a group looks like this:

User-agent: [token exactly as Mistral publishes it]
Disallow: /private/

Robots.txt is a request, not a lock. A crawler has to choose to follow it.

Your access logs show what actually reaches your site. That beats any list. Here's a short routine:

1. Export raw access logs from your origin server or CDN for the last 30 days.
2. Filter the user agent field for the string mistral, case-insensitive. Note every distinct full user agent string you find.
3. For each string, record the request count, the paths requested, and the status codes returned.
4. Check whether each agent fetched /robots.txt before other pages. A crawler that never requests it isn't reading your rules.
5. Look at the status codes. A run of 403 or 429 responses, or challenge pages, means your firewall or bot manager is turning the agent away.
6. Compare the paths it requested against your robots.txt rules. Requests to disallowed paths are worth flagging.

If you find nothing, that is a result too. It can mean Mistral hasn't visited, or that its fetcher uses a generic user agent you can't attribute. Logs can't tell those apart.

Anyone can send a request that claims to be any bot. A user agent string is a label, not proof. Cloudflare's bot documentation describes verified bots as ones that can be confirmed by a cryptographic signature, a published IP list with a stable user agent, or reverse DNS.

Apply the same tests to anything labelled Mistral:

• Published IP ranges. If Mistral publishes them, match request IPs against the list. OpenAI does this for its search bot, for example, in searchbot.json.
• Reverse DNS. Look up the hostname behind the request IP and check it resolves back to the same IP on a domain the vendor controls.
• Consistency. A real crawler tends to hit robots.txt, crawl at a steady rate, and use a stable string.

If you can't verify an agent, decide whether it's worth serving. We weigh that choice in whether to block AI crawlers you don't recognize.

Other vendors show why a single "block Mistral" rule can misfire. OpenAI states that its bot settings are independent: GPTBot governs training use, while OAI-SearchBot controls whether a site shows in ChatGPT search. Blocking one does not block the other. OpenAI also says robots.txt "may not apply" to ChatGPT-User, because a person triggers those fetches. Perplexity says Perplexity-User "generally ignores robots.txt rules." Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User, as covered in ClaudeBot, Claude-SearchBot and Claude-User explained.

We don't know whether Mistral follows a similar split. So decide what you want first:

• Keep content out of model training. You need the token Mistral documents for training, if it has one.
• Stay eligible for Le Chat answers that cite the web. You need to know which agent or index serves those answers. That isn't documented in our evidence base.
• Block everything from Mistral. You need every token, plus a firewall rule if some fetchers ignore robots.txt.

Nobody outside Mistral controls what Le Chat says from its trained knowledge. A robots.txt change today can't reach an answer the model gives with no web access. For a per-bot template across vendors, use our robots.txt guide for AI crawlers.

Robots.txt is one gate of several. Two more decide whether any AI crawler, Mistral's included, can use your pages.

Firewall and CDN rules

A firewall or bot manager can return 403 to a crawler that robots.txt allows. OpenAI's help documentation makes eligibility for ChatGPT search a two-part condition: allow OAI-SearchBot, and make sure the host or CDN lets its IP addresses through. Anthropic warns that IP blocking can stop it reading your robots.txt at all. The status codes from the log routine above tell you which gate is closed.

JavaScript rendering

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. Nothing in our evidence base says how Mistral's fetchers behave. The safe move is to put prices, product names and key facts in the HTML your server sends, before any script runs. That helps every crawler that doesn't render.

Check both gates after any CDN or hosting change, even one you didn't make yourself.