How Grok Chooses Sources: What Is Documented

Short version: nothing we can verify. Our evidence base holds 140 claims checked against primary sources from Google, OpenAI, Anthropic, Perplexity, Meta, Microsoft, academic papers and crawler infrastructure providers. None of them is an xAI document about Grok.

That means we can't tell you:

• which user agent Grok's crawler sends, if it runs one
• whether Grok searches a third-party index, its own index, or both
• when Grok decides to search instead of answering from memory
• how Grok formats or chooses citations
• which robots.txt token, if any, controls inclusion in Grok answers

We won't fill those gaps with guesses. Plenty of pages online name a Grok bot or claim Grok favours certain sources. If a claim doesn't point to a first-party xAI page, treat it as unconfirmed. If xAI publishes documentation later, it should replace anything on this page.

What we can give you is the general model of how AI answers get built, and the checks that hold regardless of vendor.

Every AI answer about your brand comes from one of four channels. Separating them shows what you can influence and what you can't. Our guide to how AI search picks sources covers them in depth.

Two facts hold across every platform in our evidence. First, retrieval is per query and discretionary: engines search for some questions and answer others from memory. Second, citation isn't the same as use. OpenAI, for example, exposes retrieved and cited URLs as separate fields, and the retrieved set is larger. Expect any Grok answer to be a mix of both, with no reliable way to tell which you got unless sources are shown.

The engines that do publish crawler docs show how vendor-specific these rules are. OpenAI lists separate bots for separate jobs: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-triggered fetches. OpenAI states that "each setting is independent of the others." Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot from Perplexity-User, and says the latter "generally ignores robots.txt rules." Meta uses Meta-WebIndexer for Meta AI search.

Google works differently again. To appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible for a snippet in Google Search, and there's no separate robots.txt token for those features.

The lesson for Grok: allowing or blocking any of these named bots doesn't tell you anything about Grok access. Each vendor documents its own tokens, and none of them covers another vendor. For comparison, see how Claude picks sources and how Perplexity chooses sources, where first-party documentation does exist.

The strongest evidence on AI visibility concerns access, and access checks work on your side of the connection. They don't need xAI to publish anything.

1. Server-render the facts that matter

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. We don't know whether Grok's fetcher renders JavaScript. Assume it might not. Fetch your key pages with JavaScript disabled and confirm your pricing, product names and core claims appear in the raw HTML.

2. Look for blanket AI blocks in robots.txt

Rules like a wildcard Disallow or a long list of AI agents copied from a template can shut out crawlers you never meant to block. Read your robots.txt line by line and know why each rule is there.

3. Check your firewall and CDN

A firewall or CDN can block an AI crawler that robots.txt allows, often with a 403. Review your server or CDN logs for user agents that identify themselves as xAI or Grok. If they appear, see what status codes they get. If they don't appear, that's information too, though it proves nothing about how Grok sources answers.

4. Audit snippet and index controls

Accidental noindex, nosnippet, max-snippet:0 or data-nosnippet tags keep content out of AI answers on Google and Bing. Template inheritance is a common cause.

5. State facts once, consistently

A specific fact on a fetchable page (a price, a number, a one-line definition) gives any engine a sentence it can quote. Contradictions between your pricing page, about page and footer give it room to get things wrong.

Without vendor docs, observation is the only direct evidence. Make it repeatable.

1. Write the buyer questions a prospect would really ask, not just your brand name. Engines rewrite queries, so the wording you type isn't necessarily what gets searched (more on that in whether AI searches the exact question).
2. Ask each question in a clean session and record the full answer, the date, and any sources shown.
3. Note whether the answer shows sources at all. No sources usually means you can't tell whether it searched.
4. Flag every factual error about your brand: wrong price, old product name, competitor confusion.
5. Repeat on a schedule, since model versions and behaviour change without notice.

If Grok cites third-party pages instead of yours, being named accurately on those pages is the route into the answer. That's an association in the evidence, not a guaranteed effect. This playbook on finding the pages AI engines cite shows how to build that list. Keep Grok results in their own column. Blending engines into one number hides the differences that matter.

Some of this is out of everyone's hands. You can't change what a model learned before its training cutoff. You can't see the queries an engine actually runs, or where your page sits in its candidate list.

Be sceptical of Grok-specific optimization tips. A July 2026 critical survey of 45 GEO studies (arXiv 2607.14035) concluded that already-retrieved content can change its citation, but no reviewed technique shows a stable, cross-platform causal effect on discoverability. The C-SEO Bench study found most conversational SEO methods largely ineffective and often harmful. Schema, llms.txt, "AI-friendly" phrasing and content chunking have no credible evidence of getting pages cited, on Grok or anywhere else.

What survives is topical fit to the question being asked, covered in whether staying on topic helps AI choose your page, plus the access checks above. Fix what blocks you. Then measure what Grok actually says.