Can I block Meta's AI training crawler and still appear in Meta AI?

Meta documents its crawlers by purpose, and robots.txt lets you treat each one differently. That is what makes this possible.

• Meta-ExternalAgent gathers content that may be used to train Meta's AI models. See the Meta-ExternalAgent definition.
• Meta-WebIndexer builds the index Meta AI uses for search answers. Meta's crawler documentation says: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."
• Meta-ExternalFetcher fetches pages on demand and is documented as potentially bypassing robots.txt.

Because these are separate user agents, you can disallow training and allow indexing. Meta's line about Meta-WebIndexer is notable: it is the only first-party statement we know of from any AI vendor that ties allowing a crawler to citation. Even so, it says "helps", not "guarantees".

The full breakdown of each agent lives on our page about Meta's crawlers and user agents.

Add a named group for each Meta crawler you want to control. A crawler follows the group that names it, so these rules sit independently of any User-agent: * group, as the Robots Exclusion Protocol (RFC 9309) specifies.

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Allow: /

Three things to check after you deploy it:

1. No blanket AI block above it. Many sites copy a list that disallows every AI bot. If Meta-WebIndexer appears in that list, it is blocked.
2. No accidental wildcard catch. If you have User-agent: * with Disallow: / and no named Meta-WebIndexer group, Meta-WebIndexer falls under the wildcard and is blocked.
3. The file itself is reachable. A crawler that can't fetch robots.txt can't read your rules.

This mirrors the OpenAI setup, where GPTBot governs training and OAI-SearchBot governs ChatGPT search. OpenAI states that "Each setting is independent of the others." The same logic is covered in training crawlers vs search crawlers across vendors.

Robots.txt is necessary but not sufficient. Two gaps matter here.

Meta-ExternalFetcher may ignore it

Meta documents that Meta-ExternalFetcher may bypass robots.txt. Treat any Disallow for it as a request, not a control. If you need to stop it, that has to happen at the server, CDN or firewall.

Your firewall can overrule your robots.txt

A WAF, CDN bot rule or IP block can return a 403 to Meta-WebIndexer while robots.txt says allow. Nothing in robots.txt tells you this has happened. Bot-management defaults also change without you editing anything: from 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. Check how your own CDN classifies each Meta crawler rather than assuming.

If you block by IP, read whether a blocked crawler can still read your robots.txt. The quickest test is to request a page as each crawler and compare the response with your rules. The AI Crawler Checker on 99Visibility does that for each AI user agent.

Here is what holds up, stage by stage.

• Access (in your control). Allowing Meta-WebIndexer and keeping it reachable makes your pages eligible for Meta's index. That is the lever.
• Retrieval and citation (not in your control). Whether Meta AI pulls your page for a given question, and whether it cites it, is Meta's decision. A critical survey of 45 GEO studies (arXiv, July 2026) found no technique with a stable, cross-platform causal effect on discoverability.
• Training (unmeasured). Nobody has published whether blocking a training crawler changes what a model later says about your brand from memory. The honest label is unknown, not harmless and not harmful.
• Meta AI tracking. No published study or vendor measurement of Meta AI citations exists in our evidence base. Anyone quoting a Meta AI citation rate is not quoting a public number.

One practical detail: tested AI crawlers, including Meta-ExternalAgent, don't execute JavaScript. Assume Meta's other fetchers may not either, and put the facts you want cited in the server-rendered HTML.

For more on the citation question itself, see whether allowing Meta-WebIndexer helps Meta AI cite you.

1. Decide your training stance. Blocking Meta-ExternalAgent is a policy choice about training use, not a visibility tactic.
2. Add the two named groups above and remove Meta-WebIndexer from any blanket AI block.
3. Test reachability from outside, per crawler. A 200 to Meta-WebIndexer is the result you want. The free Meta AI crawler check runs this test.
4. Check that key facts (prices, product names, a one-line description) appear in the raw HTML.
5. Recheck after any CDN, firewall or hosting change.

Blocking training today does not pull content out of models already trained. That is covered in what happens to pages a crawler already fetched.