How Meta AI Picks Sources to Cite

Meta's web crawler documentation (dated 2026-05-21 when our evidence base verified it) contains one sentence that matters most here: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

That sentence is unusual. Across the first-party crawler pages we reviewed from OpenAI, Anthropic, Perplexity, Google and Meta, it is the only one that ties allowing a crawler directly to being cited. OpenAI, for example, says blocking OAI-SearchBot keeps you out of ChatGPT search answers, but it frames that as exclusion, not as help.

What the sentence tells you:

• Meta AI draws on an index built by the Meta-WebIndexer crawler.
• If that crawler can't fetch your pages, Meta AI has a harder time citing and linking to them.
• Meta considers robots.txt permission for that agent a meaningful input.

What it does not tell you: how much it helps, how Meta AI chooses between two pages it has indexed, or how often it cites anything at all. "Helps" is a direction, not a measured effect. For a fuller read of the page itself, see our digest of Meta's crawler documentation.

Meta runs more than one crawler, and they do different jobs. Mixing them up is the most common way to block the wrong thing.

This split mirrors how OpenAI separates GPTBot (training) from OAI-SearchBot (search). It means you can allow search indexing while declining training. A minimal robots.txt that does that:

User-agent: Meta-WebIndexer
Allow: /

User-agent: Meta-ExternalAgent
Disallow: /

Whether blocking training has any effect on how Meta AI talks about you is unmeasured by anyone. The evidence base labels training-corpus effects as unknown, not null. The question is covered in more depth in whether you can block Meta's training crawler and still appear in Meta AI.

One caution on Meta-ExternalFetcher: because it may not honour robots.txt, a Disallow rule is not a reliable way to stop it.

Beyond the access statement, Meta's selection logic is a blank. As of October 2026, our evidence base contains no first-party Meta statement on:

• which signals rank one indexed page above another
• whether Meta AI searches the web on every question or only some
• how it rewrites a user's question into search queries
• whether it cites whole pages or quoted passages
• whether it honours snippet controls such as nosnippet or data-nosnippet (Google and Bing document these for their own AI answers; Meta's page, as digested, does not)

The evidence base also records that Meta AI is effectively unmeasured: no published study we found counts Meta AI citations or tests what changes them. So any claim that a specific tactic gets you cited in Meta AI is a guess.

The wider research doesn't fill the gap either. A July 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found that no reviewed technique shows a stable, cross-platform causal effect on discoverability. The general mechanics of retrieval and citation are in how AI search picks sources, but treat them as background, not as Meta's rules.

It helps to split any AI answer into four stages and ask which ones Meta has spoken about.

1. Parametric. What the model knows with no web access. Frozen at a training cutoff. Nothing you do this quarter changes it.
2. Access. Whether a crawler can fetch and store your page. This is where Meta's Meta-WebIndexer statement lives, and it's the stage you control.
3. Retrieval. Whether your page enters the model's context for a given question. Meta says nothing about this.
4. Generation and citation. Which retrieved pages get used and cited. Meta says nothing about this either.

So Meta has documented the one stage that site owners can actually fix. Stages 3 and 4 belong to Meta. A page that passes stage 2 is eligible, and that is all.

Because the documented lever is access, the work is mechanical and checkable.

Check robots.txt per agent

Confirm Meta-WebIndexer is not caught by a blanket User-agent: * disallow or a broad AI-bot block copied from a template. A blanket AI block is the most common real-world error with search crawlers.

Check the firewall and CDN

A firewall, bot-management rule or CDN can return 403 to a crawler that robots.txt allows. OpenAI documents this conjunction for its own search bot, and the mechanism applies to any crawler. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains and existing free-tier customers, with Search left allowed. Check which category your CDN assigns to each Meta crawler rather than assuming. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. The free Meta AI crawler check on this site covers the same question for Meta specifically.

Put key facts in the raw HTML

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Independent testing in June 2026 found the same for Meta-ExternalAgent, among others. We have no published test for Meta-WebIndexer, so the safe assumption is that it reads what's in the HTML before JavaScript runs. If your pricing or product facts only appear after hydration, server-render them.

State specific facts plainly

A page that states a price, a number or a one-line definition gives any engine a sentence it can quote. That is the only on-page signal with positive experimental support, and it doesn't depend on Meta's undocumented logic.

Some advice gets applied to every engine, Meta AI included. The evidence doesn't support it.

• Schema or FAQ schema. Ahrefs' matched test of 1,885 pages found no major citation lift on any platform it tracked, and a statistically significant 4.6% drop on AI Overviews. Nothing suggests Meta behaves differently, and nothing tests it.
• llms.txt. No engine, Meta included, documents reading it.
• Rewriting for "AI style" or chunking. The best-designed studies found such rewrites ineffective or harmful, and those were measured on pages already in context.

Injected content (hidden text or instructions aimed at AI agents) is a different category. Research on LLM search agents (arXiv:2606.16821) shows manipulated pages can sway answers once retrieved. That is illegitimate, it risks your reputation, and third parties can do it to you. If Meta AI describes your brand from weak pages, the cause is often elsewhere; see why AI cites weak or outdated sources about your brand.

Does allowing Meta-WebIndexer guarantee Meta AI will cite me? No. Meta says allowing the crawler "helps" it cite and link to your content. That makes you eligible. It doesn't promise selection, and Meta publishes no figure for how much it helps. Whether a given answer cites you depends on retrieval and generation steps Meta doesn't document. Allow the crawler, confirm your firewall lets it through, and treat any citation as an outcome you influence, not one you control.

Is Meta-WebIndexer the same as Meta-ExternalAgent? No. In the evidence base, Meta-WebIndexer is the indexing crawler tied to Meta AI citations, and Meta-ExternalAgent is a training crawler. They are separate robots.txt user agents, so you can allow one and block the other. Blocking training is a policy decision. Nobody has measured whether it changes how Meta AI describes you, so don't expect it to help or hurt your citations.

Can I keep a section of a page out of Meta AI answers? Not with a documented Meta control, as of October 2026. Google and Bing document `nosnippet`, `max-snippet` and `data-nosnippet` for limiting what their AI answers show. Meta's crawler documentation, as summarised in our evidence base, does not say Meta AI honours these. If content must stay out, the reliable options are not publishing it or blocking Meta-WebIndexer from that URL.

How can I tell if Meta AI is citing my site? Meta offers no publisher report for Meta AI citations in our evidence base, unlike Bing Webmaster Tools' AI Performance report. The practical route is to ask Meta AI the questions your buyers ask and record which sources appear, noting the date. Any such sample is a snapshot. Results vary by question and over time, so track the same questions repeatedly rather than trusting one run.

Does ranking well in Google help with Meta AI? There's no evidence either way. Meta doesn't say which index or signals sit behind Meta AI beyond its own Meta-WebIndexer crawl. Google rank data applies to Google surfaces, and even there Ahrefs found only 38% of AI Overview citations came from top-10 pages. For Meta, focus on the documented step: let Meta-WebIndexer fetch readable HTML. Compare how other engines work in [where ChatGPT citations come from](/platforms/chatgpt-citation-sources).