Grok Crawlers and robots.txt: What Is Known

Nothing, and that is the finding. Our evidence base was built from primary documents fetched and checked by hand. It covers crawler pages from OpenAI, Google, Anthropic, Perplexity, Meta and Microsoft. It contains no xAI crawler page.

So this page cannot tell you:

• the exact user agent string any Grok or xAI crawler sends
• the robots.txt token that would match it
• whether xAI separates training crawls from search or answer-time fetches
• whether any xAI fetcher honours robots.txt
• which IP ranges xAI uses

If a blog post gives you a Grok robots.txt line, treat it as unverified until xAI publishes it. We take the same approach on our pages for DeepSeek crawlers and Mistral crawlers. The next move is not to guess a token. It is to look at what actually reaches your server.

The vendors that do publish crawler pages give you a useful template. When xAI documents its bots, this is what to look for.

The pattern matters. OpenAI states "Each setting is independent of the others." Blocking a training bot does not remove you from search, and allowing it does not get you cited. Our digest of OpenAI's crawler documentation walks through one vendor's page in detail.

Without an xAI equivalent, you cannot know which of these roles a Grok-related bot plays. That is why a blanket rule based on a guessed name is risky in both directions.

robots.txt rules are grouped by user agent token, as set out in the Robots Exclusion Protocol (RFC 9309). A crawler follows the group that matches its token. A crawler with no matching group falls back to the wildcard group, User-agent: *.

That gives you one lever for undocumented bots: your default.

User-agent: *
Allow: /

If your wildcard group disallows everything, every unlisted crawler that honours robots.txt is shut out, including ones you might want. If it allows everything, every unlisted crawler is let in. Neither is wrong. It should be a choice, not an accident. Our answer on whether an unlisted AI bot is allowed covers the fallback rules in more depth.

Two limits apply. First, robots.txt is a request, not an enforcement layer. Vendors document user-triggered fetchers that may not follow it. Second, you can only add a named group once a vendor publishes the token. When that happens, add it, and check the rules on token case as you do. Our per-bot robots.txt guide shows how to structure the file.

Your server or CDN logs are the only place you can see what really fetches your pages. Work through them in this order.

1. Filter by user agent. Pull requests whose user agent mentions a vendor, a product name, or the word bot. Note the exact string. Do not shorten it.
2. Check the claim. A user agent string is just text, and anyone can copy it. Cloudflare's bot documentation describes verified bots as those identified by a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. OpenAI, for example, publishes its OAI-SearchBot IP ranges at a JSON file.
3. Look for a robots.txt fetch. A crawler that requests /robots.txt before other pages is at least reading your rules. One that never does is telling you something.
4. Check status codes. Note whether the bot got 200, 403, 429 or a challenge page. A 403 means your firewall decided, not your robots.txt.
5. Look at the pattern. Volume, timing and which URLs it requests help separate a bulk crawler from a fetch triggered by one user's question.
6. Record it. Keep the string, the IP ranges seen and the dates. When a vendor publishes documentation, you can match it.

If a string claims to be from xAI but you cannot verify it through any of these routes, treat it as unverified traffic. Our answer on blocking AI crawlers you don't recognize covers the trade-offs.

robots.txt is one gate. Your firewall, CDN and bot-management settings are another. OpenAI's help centre says eligibility for ChatGPT search needs both: allow OAI-SearchBot and confirm your host or CDN allows OpenAI's published IP addresses. A firewall can block a crawler that robots.txt allows, and nothing in your robots.txt will show it.

This layer changes without you. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search stayed allowed by default. If you are on Cloudflare, your AI crawler access may have changed with no edit on your side.

For documented crawlers, you can test this directly. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. For an undocumented bot, your logs are the evidence.

You cannot write a verified Grok rule today. You can still make good decisions.

• Set your wildcard default deliberately. Decide whether unknown crawlers should be allowed or not, and write that into User-agent: *.
• Get the documented bots right first. Check OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer, because their consequences are stated by the vendors.
• Review logs monthly. Look for new strings and verify them before acting.
• Check your CDN settings. Confirm which bot categories are blocked by default on your plan.
• Update when vendors publish. When xAI documents a crawler, add a named group. See whether you need to update robots.txt for every new crawler.

What you control is access. What you do not control is whether any engine, Grok included, retrieves or cites your pages once it can reach them.