Meta-WebIndexer: definition and how it works

Meta-WebIndexer is the user agent Meta's crawler sends when it fetches pages to build the index behind Meta AI's search answers. It is one of several Meta crawlers, and each does a different job:

• Meta-WebIndexer: indexes pages so Meta AI can cite and link to them.
• Meta-ExternalAgent: crawls content that may be used for AI training.
• Meta-ExternalFetcher: fetches pages on a user's behalf, and Meta documents that it may bypass robots.txt.

This entry stays short on purpose. For the full picture, read the Meta AI visibility guide, and see Meta's crawlers and user agents compared.

Meta-WebIndexer reads your robots.txt file before it crawls. robots.txt rules are grouped per user agent, so a rule must name the bot (or use a wildcard) to apply to it. A minimal group that lets it in looks like this:

User-agent: Meta-WebIndexer
Allow: /

Three things decide whether it actually gets your content:

1. robots.txt permission. A disallow for Meta-WebIndexer, or a blanket User-agent: * disallow, keeps it out.
2. Network reachability. A firewall, CDN or bot-management rule can return a 403 to a crawler that robots.txt allows. Nothing in robots.txt reveals this.
3. What is in the HTML. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. The evidence base does not say whether Meta-WebIndexer renders JavaScript, so put prices, product names and key facts in the server-rendered HTML.

On-page controls like noindex are documented by Google and Bing. Meta's documentation, as of October 2026, is the place to confirm which controls Meta honours.

Most AI platforms describe what their crawlers do. Meta goes one step further. Its crawler documentation (dated 2026-05-21) says: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."

No other first-party source in our evidence base makes that link between access and citation. OpenAI, for example, says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, which describes exclusion rather than help.

What this means for you: Meta-WebIndexer is one of the few levers with a stated mechanism behind it. Check that your robots.txt allows it and that your firewall does not turn it away. The AI Crawler Checker requests a page as each AI crawler and shows which ones get blocked.

• "Allowing it gets me cited." It makes you eligible. Meta's statement is an assertion with no published study behind it, and a July 2026 critical survey of 45 GEO studies found no technique with a stable, cross-platform causal effect on discoverability. More on this in does allowing Meta-WebIndexer help Meta AI cite my site.
• "Blocking Meta's training crawler blocks Meta AI search." Rules are per bot. A disallow for Meta-ExternalAgent does not name Meta-WebIndexer. See blocking Meta's training crawler while staying in Meta AI.
• "A blanket AI block is harmless." A wildcard disallow removes retrieval bots along with training bots. See what blocking all AI bots does.
• "If robots.txt allows it, it can reach me." A CDN or firewall can still block it.

• robots.txt: the per-bot permission file Meta-WebIndexer reads.
• noindex: an on-page directive that keeps a page out of an index.
• Meta crawlers compared: Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher side by side.
• Meta AI visibility guide: the full guide to this crawler and Meta AI.

Is Meta-WebIndexer the same as Meta-ExternalAgent? No. Meta-WebIndexer indexes pages for Meta AI's search answers. Meta-ExternalAgent crawls content that may be used for AI training. Because robots.txt rules apply per user agent, you can set each one separately. A site that wants to appear in Meta AI answers but keep content out of training would allow Meta-WebIndexer and disallow Meta-ExternalAgent, then confirm both rules in Meta's current documentation.

Does Meta-WebIndexer respect robots.txt? Meta documents robots.txt as the control for Meta-WebIndexer, and its statement about citation refers to allowing it in robots.txt. The exception in Meta's crawler family is Meta-ExternalFetcher, which fetches pages on a user's request and is documented as potentially bypassing robots.txt. Treat robots.txt as necessary but not sufficient, and also check that your firewall lets the crawler through.

How do I know if my firewall blocks Meta-WebIndexer? Request a page from your site using the Meta-WebIndexer user agent and record the response. A 200 means the page came back. A 403, a CAPTCHA or a JavaScript challenge means the crawler is being turned away, even if robots.txt allows it. Your CDN or server logs, filtered to verified bots, show the same thing from the inside. Repeat the check after any CDN or bot-management change.

Can anyone measure how often Meta AI cites my site? Not reliably, as of October 2026. Nothing in our evidence base measures Meta AI citations at scale, and Meta publishes no ranking or citation report for site owners. What you can measure is access: whether robots.txt allows Meta-WebIndexer, whether the crawler gets a 200, and whether your key facts are in the HTML. Those are the parts in your control.