How do I check whether GPTBot, ClaudeBot and PerplexityBot can reach my website?

Crawler access has two gates, and each one can fail silently.

Step 1: resolve robots.txt per bot

Open yoursite.com/robots.txt and find the rule that applies to each AI user agent by name. Rules are per bot. A Disallow for GPTBot does not touch OAI-SearchBot, and a wildcard User-agent: * block catches every bot without its own group. OpenAI's crawler documentation states it plainly: "Each setting is independent of the others." Our guide to testing robots.txt rules for each AI bot walks through the resolution order.

Step 2: request a page as each crawler

robots.txt is a request, not a lock. Your firewall, CDN or bot-management layer decides what actually happens. Send a request to a real content page (your pricing page is a good choice) with each crawler's full user-agent string, for example with curl -I -A "<user agent>" https://yoursite.com/pricing. Copy the exact strings from each vendor's crawler page. Then record the status code.

OpenAI's help center describes eligibility as a conjunction: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published search bot IP addresses. If either half fails, ChatGPT search cannot fetch you. The firewall testing playbook covers this step in depth.

The three bots in the question do different jobs. Two of them are training crawlers.

OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." It makes no such claim for GPTBot. So blocking GPTBot keeps your content out of training, and allowing it does not get you into ChatGPT search.

The most common mistake is a blanket AI block that was meant to stop training and also removed the search bots. If you want training out and search in, write separate groups:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

A fuller version sits in our robots.txt template for AI search and training bots.

The response tells you what the crawler would have received.

• 200 with your real content: the crawler gets through.
• 403 Forbidden: a firewall or bot-management rule blocked it. OpenAI notes that these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."
• 429 Too Many Requests: rate limiting. The bot may get through sometimes and fail at other times.
• 200 with a challenge or CAPTCHA page: a fail dressed as a pass. Automated crawlers cannot solve it, so read the body, not just the code.

Also request /robots.txt itself as each bot. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file," which means your rules may never be seen.

One limit of any test you run yourself: you control the user agent, not the IP address. Bot management often checks both. A spoofed request from your laptop can be blocked where the verified crawler would pass, or pass where the real one would fail. That is the main reason crawler checkers give different results. Your server logs filtered to verified bots are the ground truth, and you can see which AI bots crawl your site from those logs.

CDN defaults you did not set

As of September 15, 2026, Cloudflare blocks its Training and Agent crawler categories by default on pages that display ads, for new domains, new sites on existing accounts, and existing free-tier customers. Search crawlers stay allowed by default. A site can lose some AI access without anyone editing it, so recheck after any CDN change.

User-triggered fetchers

OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Blocking these through robots.txt is not reliable. Test them at the network layer if you care.

JavaScript

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. If your prices or product facts only appear after JavaScript runs, a 200 response still delivers a page without them. Read Vercel's AI crawler study for the detail, then compare your raw HTML with the rendered page.

Google

There is no robots.txt token for AI Overviews or AI Mode. Googlebot access controls them. Google-Extended governs Gemini training and grounding, not Search.

1. Resolve robots.txt for OAI-SearchBot, Claude-SearchBot and PerplexityBot first, then the training bots.
2. Request a content page and /robots.txt as each bot. Log the status code and check the body for challenge pages.
3. Where robots.txt says allow but the request fails, fix the firewall rule. OpenAI advises validating on user agent plus verified-bot programs rather than IP lists alone.
4. Confirm your key facts are in the HTML before JavaScript runs.
5. Confirm in your logs that the real crawlers get 200s.

If you want the request step done for you, 99Visibility's AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Keep expectations straight. Access buys eligibility, not retrieval or citation. Engines decide per query whether to search at all. If crawlers reach you and you still do not appear, the cause is further down the chain, which we cover in why AI crawlers visit but never cite.