DeepSeek Crawlers and robots.txt: What Is Known

The research base this library is built on covers first-party crawler pages from OpenAI, Google, Anthropic, Perplexity, Meta and Microsoft. It contains nothing from DeepSeek. As of October 2026 that means we can't confirm any of the following:

• A crawler user agent name or full user agent string.
• Whether DeepSeek separates a training crawler from a search or retrieval crawler.
• Whether any DeepSeek fetcher reads or honours robots.txt.
• A published list of IP ranges or a reverse DNS pattern for verification.
• How DeepSeek's answers pull live web content, if they do.

So this page won't hand you a token to paste. A guessed token is worse than none: it looks like a control while doing nothing. If DeepSeek publishes crawler documentation, the right move is to read that page directly and match its exact token.

What you can control today is the same for every undocumented bot. You can set a default rule for unlisted crawlers, watch your logs, and enforce at the network layer if a bot ignores you.

The vendors that do document their crawlers give you a useful benchmark. They split bots by purpose, and each robots.txt setting works on its own.

OpenAI's bots page says "Each setting is independent of the others", so blocking GPTBot stops training use without removing you from ChatGPT search. It also says robots.txt rules "may not apply" to ChatGPT-User because a user starts those requests. Perplexity says Perplexity-User "generally ignores robots.txt rules."

That split is the question to ask of any new crawler: is it collecting training data, building a search index, or fetching a page for a user right now? Each has a different consequence for you. Our digests of OpenAI's crawler documentation, Anthropic's crawler setup and Perplexity's crawler documentation cover each one in full. For DeepSeek, as of October 2026, we can't answer that question at all.

Your access logs show what actually reaches your origin. Here is a short routine for any bot you can't find documented.

1. Search the user agent field. Filter your access logs for strings containing "deepseek", matching case-insensitively. Also list every user agent that mentions "bot", "crawler" or "spider" and that you don't recognise.
2. Check whether it reads robots.txt. Look for requests to /robots.txt from the same user agent and IP range. A crawler that never fetches it can't be following it.
3. Check whether it obeys. Compare the paths it requests with your Disallow rules. Requests to disallowed paths after a robots.txt fetch mean the rule isn't being honoured.
4. Look up the IP owner. Run a WHOIS or ASN lookup on the source IPs. Traffic claiming to be a known company but coming from an unrelated hosting provider is a warning sign.
5. Try reverse DNS. Resolve the IP to a hostname, then resolve that hostname forward again. A genuine bot from a vendor that supports this returns the vendor's domain both ways.
6. Measure the load. Note request rate and bandwidth. A bot that hammers your server is an operational problem whatever its owner.

If you find nothing, that's a result too. It means no traffic in your logs identifies itself as DeepSeek. It does not prove no DeepSeek system has read your pages through another route.

Any client can send any user agent string. A scraper can call itself GPTBot, ClaudeBot or a DeepSeek name it made up. Treating the string as identity lets bad traffic through and can make you block the wrong thing.

Cloudflare's verified-bot criteria show what real verification needs: "a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS", plus robots.txt compliance. OpenAI, for example, publishes its OAI-SearchBot IP ranges as a JSON file, and still advises against relying on IP allowlisting alone.

For a bot with no published IP list and no documented reverse DNS pattern, you can't fully verify it. Treat it as unverified, and base your decision on its behaviour in your logs rather than the name it gives.

robots.txt works by matching a crawler's product token to a group. Under the Robots Exclusion Protocol (RFC 9309), a crawler that finds a group naming it follows that group. If none names it, it falls back to the * group. That fallback is your only robots.txt control over a bot whose token you don't know.

This example allows two documented search crawlers and disallows everything else:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /

Be careful with this pattern. A blanket * disallow also blocks every crawler you forgot to list, including ones you want, such as Googlebot and bingbot. Many sites do the opposite: allow by default and add named blocks as vendors document their bots.

Two limits apply either way. A robots.txt rule is a request, and a crawler that ignores the file ignores your rule. And a block only affects future fetches: it can't remove content from a model trained before you added it. See whether unlisted AI bots count as allowed and our per-bot robots.txt guide for the full rule set.

If your logs show a bot ignoring robots.txt, or loading your server too heavily, the enforcement point is your firewall, WAF or CDN. You can block by user agent, by IP range or ASN, or by rate.

The same layer can hurt you. A firewall or CDN can block an AI crawler that robots.txt allows, and nothing in robots.txt will tell you. OpenAI's help centre says eligibility for ChatGPT search needs OAI-SearchBot allowed in robots.txt and the host or CDN allowing OpenAI's published IPs. A broad rule aimed at an unknown bot can catch documented ones you meant to keep.

The next move: after any new block, request a page as each documented AI crawler you care about and confirm it still gets a 200. Our answer on blocking AI crawlers you don't recognise covers the trade-offs, and how often to revisit robots.txt explains how to keep up as vendors publish new bots.

What is the DeepSeek crawler user agent? As of October 2026, we have no first-party DeepSeek documentation naming one, so we can't confirm a user agent string or robots.txt token. Strings attributed to DeepSeek on other sites may be guesses or spoofed traffic. Check your own logs for anything claiming the name, verify its source IPs, and wait for vendor documentation before relying on an exact token.

Can I block DeepSeek with robots.txt? Without a documented token, you can only cover it through the `User-agent: *` group, which applies to every crawler not named elsewhere in the file. That also affects bots you might want. And robots.txt only works if the crawler reads and honours it, which we can't confirm for DeepSeek. For a firm block, use a firewall or CDN rule based on what your logs show.

Does blocking a crawler remove my content from an AI model? No. A robots.txt block or firewall rule stops future fetches by that crawler. It does not remove anything a model already learned in training, which is frozen at the model's knowledge cutoff. For documented vendors, training and search crawlers are separate, so a block on one may leave the other untouched.

How do I tell if a bot claiming to be DeepSeek is real? You often can't, fully. Verification normally relies on a published IP list, reverse DNS that resolves to the vendor's domain, or a cryptographic signature. Without those from DeepSeek, look up who owns the source IPs and whether the bot fetches and follows robots.txt. Judge it by behaviour, not by the name in the user agent.

Should I block every AI crawler I don't recognise? It depends on what you want from AI search. Blocking unknown bots protects server load and content, but a broad rule can catch documented search crawlers like OAI-SearchBot or PerplexityBot that make you eligible for answers on those platforms. Block specifically, then test that the crawlers you want still receive a normal response.