Free Check: Can AI Crawlers Fetch Your robots.txt?

The check answers one narrow question: when an AI crawler asks for https://yoursite.com/robots.txt, does it get the file? Most robots.txt advice assumes the answer is yes. It often isn't tested.

There are two gates between an AI crawler and your content. robots.txt is the rule. Your host, CDN or firewall decides whether the request gets through at all. If the second gate turns a bot away, the first gate never gets read. Our comparison of robots.txt and firewalls covers how the two interact.

The result is a status code per crawler. That tells you three things:

• Whether each bot can read your rules at all.
• Whether your security layer treats AI crawlers differently from browsers.
• Whether a fix belongs in robots.txt or in your firewall settings.

Anthropic's crawler documentation warns site owners that IP blocking "impedes our ability to read your robots.txt file." Put plainly: if you block the IPs, the bot can't see your rules, including any rule meant to let it in. Our answer on what happens when you block an AI crawler's IPs goes deeper.

The Robots Exclusion Protocol (RFC 9309) sets the default behavior. If robots.txt is unreachable because of server or network errors, crawlers must assume complete disallow. So a firewall that times out or returns a 5xx to a bot can work like a blanket block.

OpenAI makes the same point from the other side. Its ChatGPT search help page says eligibility needs two things: allow OAI-SearchBot to crawl, and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. robots.txt alone is not enough behind a blocking firewall.

This can change without you editing anything. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts, and existing free-tier customers. Search crawlers stayed allowed by default. If you use Cloudflare, check your AI crawler settings as of October 2026.

You need a terminal with curl. Replace example.com with your domain.

1. Request robots.txt as one crawler and print the status code.

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/robots.txt

2. Repeat for each AI crawler you care about. A short loop saves time:

for ua in OAI-SearchBot GPTBot ChatGPT-User ClaudeBot Claude-SearchBot PerplexityBot Meta-WebIndexer bingbot Googlebot; do
  printf "%s " "$ua"; curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://example.com/robots.txt
done

3. Use the full user-agent strings. The loop above sends only the token. Some firewalls match on the full string, so copy it from each vendor's crawler page (OpenAI, Anthropic, Perplexity, Meta) and rerun the ones that matter.

4. Look at the body, not just the code. A 200 can still be a challenge page.

curl -s -A "ClaudeBot" https://example.com/robots.txt | head -20

You should see lines like User-agent: and Disallow:. HTML, a CAPTCHA or a JavaScript challenge means the bot is being stopped.

5. Compare with a normal browser request. Run the same command without -A. If the browser gets 200 and a bot gets 403, your security layer is filtering by user agent.

6. Repeat from a server if you can. AI crawlers fetch from datacenters. A request from a cloud machine shows you more of what they see than one from your office.

OpenAI's advertiser guidance notes that bot-protection systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors." On WordPress, a common source is a security plugin, covered in whether WordPress security plugins block AI crawlers.

Once every crawler gets a clean 200, read the rules themselves. The free robots.txt checker for AI crawlers walks through per-bot Allow and Disallow lines. Remember the tokens are independent: OpenAI states that blocking GPTBot controls training use, while OAI-SearchBot controls ChatGPT search.

A hand test has real blind spots. Be clear about them before you act.

• IP-based rules. Your curl request comes from your IP, not the crawler's. A firewall that verifies bots by IP range, reverse DNS or a signed request may block your fake bot and allow the real one, or the reverse. This is a common reason AI crawler checkers give different results.
• Your pages. robots.txt can be reachable while content URLs are blocked. Test a real page next with the free firewall test for AI bots.
• User-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt rules.
• Outcomes. A 200 buys eligibility only. Retrieval is decided per query by each engine, and nobody outside the engine controls whether your page is chosen or cited.

The most reliable source is your own CDN or server logs filtered to verified bots. They show the status codes real crawlers received. For the full process, see the playbook on making sure crawlers can fetch robots.txt itself.

If you'd rather not script it, the free AI Crawler Checker from 99Visibility requests a page as each AI crawler and shows which ones your firewall turns away. It covers the same ground as the steps above in one run.

The same limits apply to any external tool. It can't see your firewall's internal rules or your logs. Treat a block it finds as something to confirm in your CDN or security settings, then retest after you change them.

Why would robots.txt return 403 to a bot but 200 to my browser? Your firewall, CDN or security plugin is filtering by user agent or by traffic pattern. Many bot-protection products treat unknown or automated clients as threats by default. The fix sits in that security layer, not in robots.txt. Look for a bot-management or AI crawler setting, allow the crawlers you want, and rerun the request with each crawler's user agent to confirm a 200 with plain-text rules.

Does a 404 on robots.txt block AI crawlers? No. Under RFC 9309, a robots.txt that is unavailable (a 4xx response) means a crawler may access any resources on the server. A missing file is effectively an open door. That is different from a 5xx or a network error, which tells crawlers to assume complete disallow. If you want specific rules per AI bot, publish a robots.txt file at the root of your domain.

Should I allowlist AI crawlers by IP address? OpenAI publishes IP ranges for its crawlers, including OAI-SearchBot at openai.com/searchbot.json, and its ChatGPT search page asks hosts to allow traffic from them. It also warns against relying on IP-only allowlisting. A sturdier approach is your CDN's verified-bot program, which Cloudflare describes as checking signatures, published IP lists with stable user agents, or reverse DNS.

If robots.txt is reachable, will ChatGPT or Claude cite my site? Not necessarily. Reachability is the first gate. After that, the crawler still needs permission in your rules, your pages must be fetchable, and key content should appear in the HTML before JavaScript runs. Even then, each engine decides per query whether to search and which pages to use. You can remove blocks. You can't control what a model chooses to say.

How often should I run this check? Run it after any change to your CDN, firewall, hosting or security plugins, and after vendor default changes like Cloudflare's September 2026 update. Platform and crawler behavior changes, so a monthly recheck is reasonable for sites that depend on AI search. Your server logs are the best ongoing signal, because they record what real crawlers actually received.