Meta-ExternalAgent: definition

Meta-ExternalAgent is the user agent Meta's training crawler sends when it fetches public web pages. The content it collects may be used to train Meta's AI models. It sits in the same class as GPTBot, ClaudeBot and Google-Extended: crawlers whose job is building model knowledge, not answering a live question.

Meta runs several crawlers with different jobs. Meta-ExternalAgent handles training. Meta-WebIndexer handles indexing for Meta AI search. Meta-ExternalFetcher fetches pages on demand. Each has its own name, and each needs its own robots.txt rule. For the full list and user-agent strings, see the guide to Meta's crawlers and user agents.

Meta-ExternalAgent requests a URL and reads the HTML the server sends back. It checks robots.txt first, matching rules against its own user-agent token under the Robots Exclusion Protocol (RFC 9309).

Three mechanics decide what it actually gets:

• robots.txt. A Disallow under User-agent: Meta-ExternalAgent asks it not to fetch those paths. A rule for another Meta bot does not apply to it.
• JavaScript. Vercel's 2024 crawler study found that ChatGPT and Claude crawlers download JavaScript files but don't execute them. Independent re-testing in June 2026 found the same for Meta-ExternalAgent. Text that appears only after scripts run is not in what it collects.
• Your firewall or CDN. Bot management can block a crawler that robots.txt allows. That block is silent from your side unless you check.

A typical split looks like this:

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Allow: /

If you want ready-made rules for every major AI bot, start from the robots.txt template for AI search and training bots.

Training crawlers act on the parametric stage: what a model knows with no web access. Anything they collect reaches answers only after a new model is trained, so the delay is a full model generation.

The evidence here is thin, and you should know that before you decide. No published study measures whether blocking a training crawler weakens a model's recall of your brand, or whether allowing one improves it. The honest label is unmeasured.

The Meta crawler with a stated link to visibility is the other one. Meta's documentation says: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." So the practical decision is two decisions:

1. Do you want your content used for Meta's model training? That is your Meta-ExternalAgent rule.
2. Do you want to be eligible for citation in Meta AI? That is your Meta-WebIndexer rule.

Then check that both bots can actually reach you. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

• "Blocking Meta-ExternalAgent removes me from Meta AI." Not on its own. Meta-WebIndexer has a separate token, and Meta ties citations to that one. The longer answer is in whether you can block Meta's training crawler and still appear in Meta AI.
• "Allowing it gets me cited." Meta makes no such statement about Meta-ExternalAgent. Its citation statement covers Meta-WebIndexer only.
• "It sees my site the way a browser does." Re-tests found it does not execute JavaScript. Put the facts that matter (prices, product names, definitions) in the server-rendered HTML.
• "robots.txt is the only gate." A CDN or WAF rule can block it regardless. As of September 2026, Cloudflare blocks training and agent crawler categories by default on ad-displaying pages for new domains and free-tier customers, so access can change without you editing anything.

• Training crawler: the category Meta-ExternalAgent belongs to, and what blocking one does.
• Meta-WebIndexer: Meta's indexing crawler for Meta AI answers.
• Meta's web crawler documentation, digested: what Meta's own page says about each bot.
• Meta-ExternalFetcher: Meta's on-demand fetcher, documented as potentially bypassing robots.txt.

What is the difference between Meta-ExternalAgent and Meta-WebIndexer? Meta-ExternalAgent collects content that may be used to train Meta's AI models. Meta-WebIndexer indexes content for Meta AI search, and Meta says allowing it helps Meta cite and link to your content in Meta AI responses. They use different user-agent tokens, so each needs its own robots.txt rule. Blocking one does not block the other.

Does Meta-ExternalAgent respect robots.txt? It is a named crawler with its own token, and robots.txt is the documented way to tell it what not to fetch. Robots.txt is a request, not an enforcement mechanism. If you need a hard block, apply it at your firewall or CDN as well. Meta-ExternalFetcher is the Meta bot documented as potentially bypassing robots.txt, not Meta-ExternalAgent.

Will blocking Meta-ExternalAgent hurt how Meta AI describes my brand? Nobody knows. No published study measures how excluding a site from training changes a model's later answers, and any effect would arrive only with a new model generation. What you can control today is citation eligibility through Meta-WebIndexer and whether your important facts are in the server-rendered HTML.

How do I check whether Meta's crawlers can reach my site? Read your robots.txt for rules naming Meta-ExternalAgent and Meta-WebIndexer, including any blanket AI-bot blocks. Then request a page with each crawler's user agent and look at the status code. A 403 or a challenge page means your firewall or CDN is blocking it even if robots.txt allows it. The [free Meta AI crawler check](/free/free-meta-ai-crawler-check) walks through this.