Meta's Web Crawler Documentation, Digested

99Visibility Research · Updated October 21, 2026 · 5 min read

Short answer

Meta's web crawler documentation, dated May 21, 2026, lists several crawlers with separate jobs. Its key line is about Meta-WebIndexer: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." No other vendor page in our evidence base ties crawler access to citation. Meta-ExternalAgent and Meta-ExternalFetcher serve other roles, and the fetcher may bypass robots.txt.

Key takeaways

Meta-WebIndexer is the crawler Meta ties to citations and links in Meta AI's responses.
That sentence is a first-party assertion, not a measured effect, so it describes a route in, not a guarantee.
Meta-ExternalFetcher is documented as potentially bypassing robots.txt, so a Disallow rule is not a reliable block for it.
A robots.txt Allow does nothing if your firewall or CDN turns Meta-WebIndexer away, so check both layers.
Decide on training and search access separately: they are different crawlers with different consequences.

What the page is and who published it

The source is Meta's own developer page on its web crawlers, published under its webmaster documentation and dated May 21, 2026. It is the operator describing its own bots. It is not a study, and it reports no data.

The page matters because Meta is one of the few AI operators that publishes per-crawler roles at all. It names separate user agents for separate jobs, which lets a site owner make separate decisions. For a fuller list of the agents, see our reference on Meta's crawler user agents.

What this means for you: treat the page as the rulebook for Meta's bots, written by the party that runs them. It tells you what each crawler is for. It does not tell you how Meta AI picks sources.

How we read it

We read the Meta page as one of seven source classes in a verification-first evidence base, checked on 2026-08-22. A claim only entered that base if the primary document was fetched and its wording recorded. A second reviewer then tried to prove each claim wrong.

We compared Meta's wording with the equivalent pages from OpenAI, Anthropic and Perplexity. The question for each vendor was simple: what happens, in the operator's own words, when you allow or block a given bot?

One caveat applies to all operator documentation. These pages are assertions, not measurements. They are the best evidence available for each platform's own behavior, but nobody outside the company can test them directly.

What the documentation says

The page separates Meta's crawlers by job. Three matter for AI visibility.

Meta-WebIndexer

This is the crawler behind Meta AI search. Meta's exact words: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." Our glossary entry on Meta-WebIndexer covers the term in more depth.

Meta-ExternalAgent

Our evidence base groups Meta-ExternalAgent with training-corpus crawlers such as GPTBot, ClaudeBot and Google-Extended. That is a separate channel from Meta AI search. Independent re-testing in June 2026 also found that Meta-ExternalAgent does not execute JavaScript, so content that only appears after scripts run is invisible to it.

Meta-ExternalFetcher

Meta documents this fetcher as potentially bypassing robots.txt. A Disallow line is therefore not a dependable way to stop it.

Why the WebIndexer line stands out

Every other first-party statement in our evidence base is negative or neutral. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", and that GPTBot governs training only (see our digest of OpenAI's crawler documentation). Perplexity says Perplexity-User "generally ignores robots.txt rules" (see our Perplexity crawler digest). Meta's sentence is the only one that says allowing a crawler helps citation.

What it does not show

The word is "helps". It is not "guarantees", and Meta attaches no number to it. Here is what the page leaves open:

No effect size. Nobody has published a before and after on allowing Meta-WebIndexer. The size of the effect is unmeasured.
No selection rules. Allowing the crawler makes your pages eligible to be fetched. It says nothing about which pages Meta AI picks for a given question.
No tracking. Our evidence base found that no vendor tracks Meta AI answers at all, so there is no outside data on how often Meta AI cites anyone.
No coverage of the network layer. robots.txt is a request, not a gate. A firewall, CDN or bot-management rule can block a crawler that robots.txt allows. Operators document this mechanism. How often it happens is not measured.
No promise of stability. Crawler names and roles change. Everything here describes the page as of October 2026.

The wider research points the same way. A critical survey of 45 GEO studies (arXiv 2607.14035) found no technique with a stable, cross-platform causal effect on discoverability. Access is a precondition, not a ranking lever.

What a practitioner should do with it

The controllable part is access. Check it in this order.

Read your robots.txt for Meta-WebIndexer. Look for a named rule, and for blanket rules (User-agent: * with Disallow: /) or copied AI blocklists that catch it by accident. A minimal explicit rule looks like this:

User-agent: Meta-WebIndexer
Allow: /

Decide training separately. Blocking Meta-ExternalAgent is a training decision. Our answer on blocking Meta's training crawler while staying in Meta AI walks through it.
Test the firewall and CDN. Request a page with the Meta-WebIndexer user agent and record the status code. A 403, a challenge page or a CAPTCHA means the crawler is turned away whatever robots.txt says. Cloudflare changed its defaults on 2026-09-15 for new domains and free-tier sites, so recheck if you sit behind it. The free Meta AI crawler check on this site, or the AI Crawler Checker, shows what each crawler actually receives.
Put key facts in the raw HTML. Prices, product names and one-line definitions should be readable before JavaScript runs.
Recheck after changes. Template edits, CDN migrations and security rule updates can remove access without anyone touching robots.txt.

None of this promises a Meta AI citation. It removes the blocks you control.

Frequently asked questions

Is Meta-WebIndexer the same as Meta-ExternalAgent?

No. Meta documents them as separate crawlers. Meta-WebIndexer is the one Meta links to citing and linking content in Meta AI's responses. Our evidence base groups Meta-ExternalAgent with training-corpus crawlers like GPTBot and ClaudeBot. Because each has its own user agent, you can write a separate robots.txt rule for each one and make the training decision independently of the search decision.

Will blocking Meta-ExternalFetcher in robots.txt stop it?

Not reliably. Meta's documentation says the fetcher may bypass robots.txt. That matches other vendors: Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not apply to ChatGPT-User. If you need to stop a user-triggered fetcher, robots.txt alone is not the tool. Network-level controls exist, but test that they do not also catch the crawlers you want to allow.

Does allowing Meta-WebIndexer guarantee my site appears in Meta AI?

No. Meta says allowing it helps Meta cite and link your content, which describes a route in. It gives no effect size and no selection rules. Whether a page is used depends on the question, the other candidate sources and the model. Nobody outside Meta measures how often Meta AI cites any given site. Allowing the crawler removes a block. It does not promise an outcome.

If my robots.txt has no rule for Meta-WebIndexer, am I fine?

Probably for robots.txt, since no matching rule usually means allowed. Check for a wildcard block that applies to every agent. Then check the network layer. A WAF, bot-management rule or CDN default can return a 403 to a crawler that robots.txt permits, and nothing in your analytics will flag it. Request a page as Meta-WebIndexer and look at the status code you get back.

Do subdomains need their own rules for Meta's crawlers?

Yes. robots.txt applies per host, so a docs or help subdomain needs its own file. If your help center sits on a separate subdomain with a stricter security setup, it can block Meta-WebIndexer while your main site allows it. Check each host that carries content you want Meta AI to be able to fetch, including pricing and documentation pages.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.