Meta Crawlers: Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher

Meta documents its crawlers on its web crawlers page for webmasters (dated 2026-05-21 in our evidence base). Three agents matter for AI visibility. They do different jobs, and blocking one does nothing to the others.

The pattern mirrors other vendors. OpenAI splits training (GPTBot) from search (OAI-SearchBot) and says "each setting is independent of the others." Anthropic does the same with ClaudeBot and Claude-SearchBot, covered in our page on ClaudeBot, Claude-SearchBot and Claude-User. For a line-by-line reading of Meta's own page, see Meta's web crawler documentation, digested.

The next move: list which of the three your robots.txt names today, and whether that matches what you intend.

Meta-ExternalAgent sits in the same class as GPTBot, ClaudeBot and Google-Extended. It gathers content that may go into model training. That is the parametric stage: what Meta's models know with no web access, frozen at a training cutoff.

Two things follow.

• The effect of allowing or blocking it is unmeasured. Nothing in the published evidence tests whether excluding a site from training changes how a model recalls that brand later. Any effect would land a full model generation later. Treat claims in either direction as unsupported.
• It does not execute JavaScript. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but "don't execute them. They can't read client-side rendered content." An independent re-test in June 2026 extended that finding to a wider set of crawlers, including Meta-ExternalAgent, PerplexityBot and Bytespider. Googlebot and AppleBot were the exceptions.

If your pricing, product names or key facts load only after JavaScript runs, Meta-ExternalAgent likely sees an empty shell. The fix is server-side rendering or prerendering of the content that matters. Compare the raw HTML (JavaScript off) with the rendered page and note what appears only after hydration.

Meta-WebIndexer is the agent behind Meta AI search. Meta's documentation states: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

That sentence is unusual. Across the crawler documentation from Google, OpenAI, Anthropic, Perplexity and Microsoft in our evidence base, it is the only first-party statement tying crawler access to citation. OpenAI, for comparison, says opting out of OAI-SearchBot removes you from ChatGPT search answers, but it never says allowing the bot helps you get cited.

Keep the claim in proportion. Allowing Meta-WebIndexer is a precondition Meta describes, not a guarantee. Meta publishes no ranking factors for Meta AI, and no study has measured what allowing the bot changes. We cover the limits in does allowing Meta-WebIndexer help Meta AI cite my site, and the term itself in the Meta-WebIndexer glossary entry.

The next move: confirm no blanket rule (such as User-agent: * with Disallow: /, or a broad AI-bot block) is catching Meta-WebIndexer by accident.

Meta documents Meta-ExternalFetcher as potentially bypassing robots.txt. That puts it in the same position as OpenAI's ChatGPT-User, where OpenAI says robots.txt "may not apply," and Perplexity-User, which Perplexity says "generally ignores robots.txt rules."

What this means in practice:

• A Disallow rule for Meta-ExternalFetcher is not a reliable block.
• Index-time control (whether a crawler stores your page) and fetch-time control (whether a page gets fetched on demand) are different things.
• If you genuinely need to keep content away from this fetcher, robots.txt is the wrong tool. Access controls at the server, CDN or authentication layer are the controls that actually refuse a request.

For most marketing sites, the practical question is the reverse: make sure your firewall is not refusing it when you want your public pages read.

Because rules apply per user agent, you can opt out of training while staying eligible for Meta AI search. A minimal robots.txt looks like this:

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Allow: /

A few checks before you ship it:

• Keep each agent in its own group. A crawler follows the most specific group that names it.
• Check what happens to agents you don't name. Our page on whether an unlisted AI bot is allowed walks through how a User-agent: * group applies.
• Don't rely on robots.txt for Meta-ExternalFetcher, for the reason above.

The trade-offs are laid out in can I block Meta's AI training crawler and still appear in Meta AI. The short version: the training block has an unmeasured effect, and the search allow is the one Meta ties to citation.

robots.txt states permission. It does not deliver the request. A web application firewall, bot-management rule or CDN policy can return a 403 or a challenge page to a crawler that robots.txt allows. The crawler then sees nothing, and no dashboard tells you.

This is a documented mechanism, not a measured rate. Nobody has published how often it happens for Meta's agents. One dated change raises the odds: from 2026-09-15, Cloudflare blocks Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts and all existing free-tier customers. Search crawlers stay allowed by default. A site can lose access without anyone editing a file.

As of October 2026, the check is simple:

1. Request a public page with each Meta user agent string.
2. Record the status code and whether a challenge page comes back.
3. Confirm /robots.txt itself is reachable.
4. If you use Cloudflare, see how your zone classifies each Meta agent.

The free Meta AI crawler check on this site runs that first test for Meta-WebIndexer.

Be clear about the gaps, because they set the limit on what anyone can promise you.

• No ranking factors. Meta does not say how Meta AI chooses which indexed pages to cite.
• No reporting surface. There is no Meta equivalent of Bing Webmaster Tools' AI Performance report or Google Search Console's generative AI report.
• No measurement. Our evidence base records no vendor that tracks Meta AI answers at all.
• No training effect data. Whether blocking Meta-ExternalAgent changes what Meta's models say about you is untested.

What is in your control: per-agent robots.txt rules, network reachability, and server-rendered content. What is not: how Meta's retrieval and models use your pages once they can read them. Re-check Meta's crawler page periodically, since user agents and their descriptions change.