Troubleshooting 403 Forbidden Errors for AI Crawlers

99Visibility Research · Updated October 21, 2026 · 8 min read

Short answer

A 403 served to an AI search crawler means that crawler cannot fetch your page, whatever robots.txt says. If OAI-SearchBot or PerplexityBot gets a 403 or a challenge page, that engine has nothing of yours to index. The fix is to find which layer refused the request (WAF rule, bot management, rate limit, geo or IP block) and allow verified crawlers through that layer.

Key takeaways

robots.txt permission is only half the access gate; OpenAI says the host or CDN must also allow its published search bot IP addresses.
A 403, a 429 or a CAPTCHA page served to a search crawler is a silent failure: no dashboard warns you and robots.txt still looks correct.
Test each crawler user agent, then confirm against your own logs filtered to verified bot traffic, because a spoofed test can be treated differently from the real bot.
Fixing a 403 restores eligibility only; it does not guarantee retrieval or citation.
Re-run the check after every CDN, WAF or plan change, since a vendor default can change your access without you editing anything.

Why a 403 beats robots.txt every time

An AI engine has to pass two gates before your page can appear in an answer. robots.txt says whether a crawler may fetch. Your server, CDN and firewall decide whether it actually can. The second gate wins. If it answers with a 403 Forbidden, the crawler never sees your content.

OpenAI states this as a conjunction in its ChatGPT search help: to be eligible, allow OAI-SearchBot to crawl the site and confirm the host or CDN allows traffic from OpenAI's published search bot IP addresses. OpenAI's crawler documentation adds that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." A firewall block has the same practical effect as that opt-out, except nobody chose it.

The failure is silent. Your robots.txt still reads correctly. Your analytics show nothing, because a blocked crawler leaves no referral. The comparison of robots.txt and the firewall as two separate gates covers the model in more detail.

Two limits on what this fixes:

This is the access stage only. Passing it makes you eligible. It does not make an engine retrieve or cite you.
How often sites block crawlers this way is not measured anywhere. Treat it as a thing to check, not an assumption that you are blocked.

The layers that return 403s and challenges

Several systems can refuse an AI crawler, often without anyone deciding to.

Bot management and WAF rules

Security products score traffic and block or challenge anything that looks automated. AI crawlers are automated by definition. A managed rule set, a "block AI bots" toggle, or a custom rule matching user agents can return 403 to every crawler at once.

CDN defaults

From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on pages that display ads, for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers stayed allowed by default. A site can change category without its owner editing anything, so check rather than assume.

Rate limits

A crawler fetching many URLs quickly can trip a rate limit and receive 429 Too Many Requests. Some setups return 403 instead.

Geo and IP blocks

Country blocks, ASN blocks and datacenter IP blocklists catch crawlers because crawlers run from cloud infrastructure. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file", which means it cannot even see your rules.

Challenges and CAPTCHAs

A JavaScript challenge often returns 200 or 503 with a challenge page as the body. The crawler gets HTML, but not your content. The guide on whether CAPTCHAs and bot challenges block AI crawlers explains why automated fetchers fail these.

IPv6-only paths

OpenAI's published IP files listed IPv4 prefixes only when the evidence base checked them in August 2026. An edge that answers crawlers on IPv6 alone can be unreachable to those bots.

Diagnostic steps: find which layer is refusing

Run these in order. Each step narrows the cause.

Pick three test URLs. Use your homepage, a high-value page (pricing or a core product page) and /robots.txt.
Request each URL as each crawler. Copy the user agent strings from each vendor's crawler page (OpenAI, Perplexity, Anthropic, Meta, Bing). Record the status code and the first lines of the body.

curl -sS -o body.html -w "%{http_code}\n" -A "<documented user agent string>" https://example.com/pricing

Read the body, not just the code. A 200 whose body says "checking your browser" or contains a CAPTCHA is a challenge, and it counts as a failure.
Repeat from a second network. Test from a cloud server and from an ordinary connection. If only the cloud request fails, you have an IP, ASN or datacenter rule.
Open your WAF or CDN event log. Filter by the time of your test and by crawler user agent. The log names the rule, the bot score or the rate limit that fired.
Check your origin logs for the real bots. Filter to requests from published crawler IPs, such as OpenAI's searchbot.json ranges. A spoofed curl request from your own IP is not the real crawler. Verified-bot rules may block your test but allow the genuine bot, or the reverse. Origin logs are the ground truth.
Fix the specific rule. Allow verified crawlers in that rule. Do not switch off your whole firewall.
Re-test steps 2 to 6. Confirm a 200 with real content for each crawler you want.

On allowlisting: OpenAI publishes IP ranges and also warns against relying on IP-only allowlisting. Cloudflare's bot documentation describes verified bots as identified by a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. Prefer a verified-bot setting over a hand-maintained IP list. For a 99Visibility tool that runs step 2 for you, the AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

What good and bad results look like

Result for a search crawler	What it means	Next move
200, your real content in the HTML	Access gate passed	Check the content is present before JavaScript runs
200, challenge or CAPTCHA page	Blocked by a challenge	Exempt verified bots from the challenge rule
403 Forbidden	WAF, bot management, geo or IP rule	Find the rule ID in the event log and allow verified crawlers
429 Too Many Requests	Rate limit	Raise the limit for verified bots
503 with challenge body	Challenge mode at the edge	Same as a challenge page
Timeout or connection reset	Network-level block or IPv6-only edge	Check IP blocks and IPv4 reachability
403 on /robots.txt only	Crawler cannot read your rules	Allow robots.txt for all agents

A clean result is boring: the same 200 and the same content for every crawler you intend to allow. Anything that differs between a browser and a crawler deserves a look.

One trap: a 200 is not the end. Most AI crawlers do not execute JavaScript, per Vercel's crawler study. If your pricing only appears after hydration, a 200 still delivers an empty page.

Which bots to test and what blocking each one costs

Blocks are per bot, and the consequences differ. Test the retrieval bots first, because those govern whether an engine can show you in search answers.

OAI-SearchBot: ChatGPT search. Blocking removes you from ChatGPT search answers. See OpenAI's crawlers, robots.txt and firewalls.
GPTBot: OpenAI training only. OpenAI says each setting "is independent of the others", so a GPTBot block does not affect ChatGPT search.
PerplexityBot: Perplexity's search index. Perplexity-User, which fetches on a user's request, "generally ignores robots.txt rules" per Perplexity, but a firewall can still stop it. The details are in PerplexityBot vs Perplexity-User.
Claude-SearchBot and ClaudeBot: Claude search quality and Anthropic training, respectively.
Meta-WebIndexer: Meta AI search. Meta states that allowing it "helps us cite and link to your content in Meta AI's responses."
Googlebot and bingbot: Google's AI Overviews and AI Mode need the page indexed in Google Search, and Bing's index backs Copilot. A firewall that blocks either one blocks its AI surfaces too.

If you want search visibility but not training use, you can split them. The question on whether allowing search bots means training use covers that choice.

How often to repeat the check

Access breaks when something upstream changes, not when you publish. Re-run the diagnostic steps:

after any change to your CDN, WAF, bot management or hosting plan;
after a security vendor announces new defaults, as Cloudflare did for 2026-09-15;
after a site migration, redesign or new subdomain;
on a fixed schedule, such as monthly, so a silent change does not sit for a quarter.

The evidence base has no number for how fast blocks appear or how long engines take to recrawl after a fix. Set the schedule by how often your infrastructure changes. For the wider set of CDN failure modes, read when your CDN or firewall blocks AI crawlers.

Frequently asked questions

Can I fix a 403 for AI crawlers by editing robots.txt?

No. robots.txt only states permission. A 403 comes from your server, CDN or firewall, which refuses the request before robots.txt matters. You have to change the rule that fired, usually in your WAF or bot management settings. Find it in the event log, then allow verified crawlers through that rule. Editing robots.txt alone leaves the block in place, and the crawler keeps receiving the same refusal.

Why does my curl test pass when the real OAI-SearchBot is blocked?

Your test sends the right user agent from the wrong IP. Bot management often checks whether a request claiming to be a crawler comes from that crawler's published ranges. Your spoofed request may be treated as an ordinary visitor and allowed, while the genuine bot hits a separate rule. The reverse also happens. Origin logs filtered to OpenAI's published IP ranges show what the real crawler received.

Should I allowlist AI crawlers by IP address?

Use IP ranges as one signal, not the only one. OpenAI publishes IP ranges for its crawlers and also warns against relying on IP-only allowlisting. Cloudflare describes verified bots as identified by a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. A verified-bot setting in your CDN is easier to maintain than a hand-kept list that drifts as ranges change.

Does a 429 rate limit hurt as much as a 403?

It depends on how often it fires. A 429 tells the crawler to slow down, so occasional ones may only delay fetching. A rate limit tight enough to refuse most crawler requests has the same practical effect as a block. Check your logs for the share of crawler requests getting 429. If it is most of them, raise the limit for verified bots.

If I unblock the crawlers, will AI engines start citing me?

Not necessarily. Removing a 403 makes your pages reachable and eligible. Whether an engine retrieves them for a given question, and whether it cites them, depends on retrieval and generation steps you do not control. What you can control is making sure the block is gone and your key facts appear in the HTML before JavaScript runs.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.