Free Check: Can Meta AI's Crawler Index Your Site?

The check answers two yes-or-no questions for each Meta crawler: does your robots.txt permit it, and does your server actually let it in.

That matters because Meta splits its crawling by purpose, as of October 2026:

• Meta-WebIndexer indexes pages for Meta AI search. See the Meta-WebIndexer definition for how it works.
• Meta-ExternalAgent collects content for training Meta's models.
• Meta-ExternalFetcher fetches pages on a user's behalf, and Meta documents that it may bypass robots.txt.

Meta's crawler documentation (dated 2026-05-21) contains one sentence no other AI vendor has published: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." It is the only first-party statement in our evidence base that ties crawler access to citation.

Read it carefully. "Helps" is not "guarantees". Allowing the bot makes your pages eligible. Whether Meta AI retrieves and cites them for a given question is decided by Meta, not by you. What you control is whether you block yourself. The full breakdown of user agents is on our Meta crawlers reference.

You need a terminal and five minutes. Replace example.com with your domain.

1. Fetch your live robots.txt. Run curl -s https://example.com/robots.txt. Check the file you serve, not the one in your repo. If this returns an error, fix that first: our guide to whether AI crawlers can fetch your robots.txt covers it.
2. Find the groups that apply. Look for User-agent: Meta-WebIndexer, User-agent: Meta-ExternalAgent and User-agent: *. Under RFC 9309, a crawler obeys the most specific group that names it. If a named group exists, the * group does not apply to that bot.
3. Read the rules in each group. Disallow: / blocks the whole site. A blank Disallow: allows everything. Note any path rules that hit your key pages, such as /pricing or /products/.
4. Request a page as each crawler. Run curl -s -o /dev/null -w "%{http_code}\n" -A "Meta-WebIndexer" https://example.com/ and repeat with -A "Meta-ExternalAgent". For the exact full user-agent strings, copy them from Meta's crawler page.
5. Compare against a normal request. Run the same command without -A. If the browser-style request gets 200 and the Meta request gets 403, 429 or an HTML challenge page, your firewall is filtering it.
6. Check your server or CDN logs. Filter for meta-webindexer and meta-externalagent and look at the status codes the real bots received. Logs show what the actual crawler saw, which a test from your laptop cannot.

If you want Meta AI search but not training, the intended setup looks like this:

User-agent: Meta-WebIndexer
Allow: /

User-agent: Meta-ExternalAgent
Disallow: /

Whether that split costs you anything in Meta AI is answered in blocking Meta training while staying in Meta AI.

Put the two gates side by side for Meta-WebIndexer.

The second row is the one to hunt for. Nothing in any dashboard warns you about it, and it removes you as completely as a robots.txt block.

A common mistake is a blanket AI block, such as one group listing every AI bot with Disallow: /. That shuts out Meta-WebIndexer along with the training crawlers. If your goal was only to stop training, move Meta-WebIndexer into its own group.

Also check your CDN settings. From 2026-09-15, Cloudflare sets Training and Agent crawler categories to blocked by default on pages that display ads, for new domains, new sites on existing accounts and all existing free-tier customers. Search stays allowed by default. Confirm which category your CDN assigns each Meta crawler and what the rule does, rather than assuming.

Passing both gates is necessary. It is not sufficient, and it is not a result.

• It does not show citations. The check confirms access only. We know of no report from Meta that shows site owners when Meta AI cited them, so you cannot confirm the outcome from Meta's side.
• Spoofed requests are not the real crawler. Bot-management tools often verify crawlers by IP range or reverse DNS. A request from your laptop with a Meta user agent may be treated differently from the real bot, in either direction. Your logs are the ground truth.
• It does not cover user-triggered fetches. Meta-ExternalFetcher may ignore robots.txt, so your robots.txt rules do not reliably control it. The difference between these two crawler types is explained in index-time crawlers vs answer-time fetchers.
• It does not check rendering. An independent June 2026 retest found Meta-ExternalAgent does not execute JavaScript. We have no equivalent test for Meta-WebIndexer. The safe move is to put your prices, product names and key facts in the HTML before scripts run.
• It does not touch what models already know. Training data from earlier crawls is frozen in the model. Changing robots.txt today affects future crawling only.

The manual steps work, but they cover one site, one moment and one URL. The free robots.txt Checker for AI Crawlers parses your file and resolves the rule for each named AI bot, including Meta's, so you can see which crawlers each group actually allows or blocks.

Pair it with the curl requests or your logs for the firewall half. A clean robots.txt result and a 403 in your logs means the fix is in your CDN or WAF, not in the file.

If you want to know what allowing the indexer can and cannot do for you, read whether allowing Meta-WebIndexer helps Meta AI cite your site.

Does blocking Meta-ExternalAgent remove me from Meta AI? Meta documents Meta-ExternalAgent and Meta-WebIndexer as separate crawlers with separate tokens, so robots.txt treats them independently. Blocking Meta-ExternalAgent targets training use. Meta AI search indexing is governed by Meta-WebIndexer. Nobody outside Meta has measured whether training exclusion changes what the model remembers about your brand, so treat that effect as unknown rather than zero.

Will allowing Meta-WebIndexer get my site cited in Meta AI? No one can promise that. Meta says allowing the crawler helps it cite and link your content, which makes your pages eligible. Whether Meta AI retrieves your page for a specific question, and then cites it, depends on Meta's retrieval and the model. Allowing the bot removes a block. It does not create a citation.

My robots.txt has no Meta groups. Am I allowed? If there is no group naming a Meta crawler, that crawler follows your `User-agent: *` group. If the `*` group allows everything, Meta-WebIndexer is allowed by robots.txt. You still need to test whether your firewall or CDN lets the request through, because those layers can block a bot that robots.txt permits.

How do I tell a real Meta crawler from a fake one? Anyone can set a user-agent string, so the string alone proves nothing. Bot-management systems usually verify crawlers with published IP lists, reverse DNS or cryptographic signatures. Check whether your CDN lists Meta's crawlers as verified bots, and use your server logs to see what requests claiming to be Meta actually received.

Why does my CDN block Meta crawlers when I never changed anything? CDN defaults change without you editing a file. Cloudflare, from 2026-09-15, blocks Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites and existing free-tier accounts. Open your bot or AI crawler settings, find each Meta crawler's category, and set the rule you actually want.