Free AI Crawler Checker: Can AI Bots Reach Your Page?

Most site owners check robots.txt and stop there. That covers only one of the gates an AI crawler has to pass. The other is the network: your host, CDN, web application firewall or bot-management settings can refuse a request that robots.txt permits.

A crawler check answers one narrow question: when a request arrives carrying an AI crawler's user agent, what does your server return? You get a status code and a response body for each bot. That tells you whether a block exists at the network layer, which bots it hits, and whether the block is a hard refusal, a rate limit or a challenge page.

This matters because the failure is silent. Nothing in a dashboard warns you when your firewall returns 403 to a search crawler. If you only want the robots.txt half, the free robots.txt checker for AI crawlers covers it. This page covers the half people skip.

OpenAI states the rule plainly in its help article on ChatGPT search. To make a site eligible, you allow OAI-SearchBot to crawl it and confirm that your host or CDN allows traffic from OpenAI's published searchbot IP addresses. The word is "and". Passing one half is not enough.

Two details trip people up:

• The bots are separate. OpenAI's crawler page says each setting is independent. GPTBot governs training use. OAI-SearchBot governs ChatGPT search. Blocking GPTBot does not remove you from ChatGPT search, and allowing it does not put you there. If training use is your worry, read whether allowing search bots means training use.
• The IP list is public. OpenAI publishes OAI-SearchBot's IP ranges at openai.com/searchbot.json. OpenAI also warns against allowlisting on IP alone, so match on user agent plus your provider's verified-bot settings.

As of October 2026, a CDN can also change your access without you editing anything. From 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that show ads, for new domains and existing free-tier customers, while Search stays allowed. If you are on Cloudflare, check those settings first.

You need a terminal with curl and one URL that matters, such as your pricing or product page.

1. Pick the bots. Test the search bots first: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer and bingbot. Add the training bots (GPTBot, ClaudeBot) if you care about training access.
2. Get the user agent strings. Each operator documents its own: OpenAI on its crawler overview page, Anthropic in its support article, Perplexity in its crawler docs. The bot token alone works for a first pass. Paste the full documented string for a closer match.
3. Request the page as each bot and print the status code.

for ua in OAI-SearchBot GPTBot Claude-SearchBot ClaudeBot PerplexityBot; do
  printf "%s " "$ua"
  curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "$ua" https://example.com/pricing
done

4. Run a baseline. Repeat the request with a normal browser user agent. You now have a status code and a byte size to compare against.
5. Inspect any suspicious body. If a bot gets a 200 but a much smaller response, save it with curl -s -A "OAI-SearchBot" -o bot.html https://example.com/pricing and open the file. Look for a challenge, CAPTCHA or "verify you are human" page.
6. Check robots.txt the same way. Request /robots.txt as each bot. Anthropic notes that IP blocking can stop its crawler reading robots.txt at all.

For a fuller walkthrough of firewall rules, see testing whether your firewall lets AI crawlers in.

Read the status code and the body together.

OpenAI's own guidance notes that bot-protection systems can mistakenly block legitimate crawlers, often with a 403. A 403 to OAI-SearchBot is a complete failure for ChatGPT search on that page.

If every bot gets the same response as your browser, the network gate looks open for that URL. If only some bots fail, you have found a rule that names them. Use the free firewall AI bot test to narrow down which layer is responsible.

A hand test is useful and imperfect. Be clear about its limits.

• It sends the bot's name, not the bot's address. Your request comes from your own IP. A firewall that verifies bots by IP range or reverse DNS may block your spoofed request and allow the real crawler, or the reverse. That is one reason crawler checkers give different results.
• It sees one location. Rules can differ between datacenter and residential traffic, and by region.
• It ignores robots.txt. curl does not read robots.txt, so a 200 says nothing about permission.
• It doesn't render. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. A 200 page whose prices load only after JavaScript still hides them. Run a JavaScript rendering check on the same URL.
• It isn't proof of what real bots received. Your server logs, filtered to verified bots, are the ground truth. The free AI bot log check shows how to read them.

Passing every check means the page is reachable. It does not mean any engine will retrieve or cite it.

Running curl for nine bots across several pages gets tedious. The free AI Crawler Checker from 99Visibility requests a page as each AI crawler and shows which ones your firewall turns away, in one pass.

Use it the same way you would use the manual test. Start with the pages that carry your load-bearing facts. Fix any 403, 429 or challenge result at the firewall or CDN. Then confirm robots.txt and rendering for the same URLs, because all three gates have to be open.

Is it legal or safe to send a request with a bot's user agent? Sending your own site a request with a crawler's user agent is a normal diagnostic. You are testing how your server responds, not impersonating the bot to anyone else. Keep the volume low so you don't trip your own rate limits. Don't use spoofed user agents against sites you don't run, and never serve different content to AI crawlers than to people, which is cloaking and a real risk.

My robots.txt allows every AI bot. Why would a crawler still be blocked? robots.txt is a request to the crawler, and your network sits in front of it. A WAF, CDN bot-management setting or hosting rule can refuse the connection before robots.txt matters. OpenAI's eligibility rule for ChatGPT search names both: allow OAI-SearchBot in robots.txt and let its published IPs through your host or CDN. Test both halves.

Do ChatGPT-User and Perplexity-User follow robots.txt? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user triggers those fetches. Perplexity says Perplexity-User generally ignores robots.txt. If you want to control those agents, the network layer is the place to do it. If you want them to reach you, make sure your firewall doesn't refuse them.

Should I allowlist AI crawlers by IP address? Use IP lists as one input, not the only one. OpenAI publishes its crawler IP ranges as JSON files and also warns against IP-only allowlisting. A safer setup combines the documented user agent with your CDN's verified-bot program or reverse DNS checks. Recheck after any firewall or CDN plan change.