403 Forbidden: definition and why it breaks AI crawler access

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

403 Forbidden is the HTTP status code a server returns when it understands a request but refuses to fulfil it. Served to a person, it is an error page. Served to an AI search crawler, it is a complete and silent visibility failure. The crawler gets nothing, and no analytics or webmaster dashboard tells you it happened. You find it only by requesting pages as each crawler or by reading your server logs.

Key takeaways

A 403 means the server refused the request, not that the page is missing.
Firewalls, CDNs and bot management rules can return 403 to AI crawlers that your robots.txt allows.
OpenAI says ChatGPT search eligibility needs both OAI-SearchBot allowed in robots.txt and your host or CDN allowing its traffic.
No dashboard reports a 403 served to an AI crawler, so you have to test for it directly.
Removing a 403 restores eligibility only; it does not guarantee a citation.

Definition

403 Forbidden is an HTTP response status code. It tells the client that the server received and understood the request, then decided not to serve it. The server is saying "you may not have this", not "this does not exist".

It sits next to three codes people often confuse with it:

401 Unauthorized: the request needs credentials, such as a login.
404 Not Found: the server has nothing at that address.
429 Too Many Requests: the client hit a rate limit and may retry later.

For a web crawler, a 403 means the fetch failed and there is no content to store or quote.

How a crawler ends up with a 403

Most 403s served to bots are not set by anyone on purpose. They come from a security layer that sits in front of the site:

A CDN or web application firewall rule that blocks a user-agent or IP range.
Bot management that classifies an AI crawler as unwanted automation.
A JavaScript challenge or CAPTCHA, which an automated crawler cannot pass. Some return 403; others return a 200 with a challenge page instead of your content.
A CMS security plugin. See whether WordPress security plugins block AI crawlers.

Defaults also change without your input. From 2026-09-15, Cloudflare set Training and Agent crawler categories to blocked by default on ad-displaying pages for new domains and existing free-tier customers, while leaving Search allowed. A site can lose some AI access without anyone editing a setting.

Why it matters for AI visibility

Access comes before everything else. A crawler that gets a 403 cannot index your page, so no retrieval system can put it in an answer. This is the access stage, and it is one of the few stages you fully control.

OpenAI's help page on ChatGPT search states eligibility as a conjunction: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. robots.txt alone is not enough. The two gates work independently, and either one can shut you out.

The failure is silent. Search Console and Bing Webmaster Tools do not report what your firewall told OAI-SearchBot, PerplexityBot or Claude-SearchBot. Anthropic also warns that IP blocking impedes its ability to read your robots.txt file, so a block can hide your crawl rules too.

How to test for it

Request a representative page using each crawler's documented user-agent string. Record the status code and whether the body is your content or a challenge page.
Repeat from a datacenter connection, since real crawlers come from datacenter IPs.
Request /robots.txt the same way. Making sure crawlers can fetch robots.txt itself covers this step.
Filter your server or CDN logs to verified bots and look for 403s. Logs are the ground truth, because some firewalls treat a spoofed user-agent differently from the real crawler.

The AI Crawler Checker runs steps 1 and 3 for each AI crawler. For fixes, follow the playbook for troubleshooting 403 errors for AI crawlers.

Common misconceptions

"robots.txt allows it, so the bot gets in." robots.txt states permission. The firewall decides reachability. They can disagree.
"Blocking every AI bot is one decision." Bots differ by purpose. OpenAI says each of its crawler settings is independent: blocking a training crawler such as GPTBot stops training use, while a 403 to OAI-SearchBot removes you from ChatGPT search answers.
"Allowlisting IPs solves it." OpenAI publishes IP ranges but warns against relying on IP-only allowlisting. Validate on user-agent plus verified-bot programmes.
"Fixing the 403 gets me cited." It makes you eligible. What the engine retrieves and cites is outside your control.

Frequently asked questions

Is a 403 to an AI crawler the same as being deindexed?

The effect can be similar, but the cause differs. Deindexing is a search engine's decision. A 403 is your own server refusing the crawler before any indexing happens. Over time, pages a crawler cannot fetch drop out of what it can use. The difference matters for the fix: a 403 sits in your firewall, CDN or plugin settings, which you control, so you can test and remove it yourself.

Can a 403 hit only some AI crawlers?

Yes, and that is common. A firewall rule may match one user-agent or IP range and not others, so OAI-SearchBot might get a 403 while PerplexityBot gets a 200. Each engine relies on its own crawler, so a block on one removes you from that engine only. Test every crawler you care about separately rather than assuming one result covers all of them.

Does a challenge page count as a 403?

Not always in the status code, but it counts in effect. Some bot defences return a 200 status with a JavaScript challenge or CAPTCHA in the body. An automated crawler cannot solve it, so it receives the challenge instead of your content. When you test, check the response body as well as the status code. A 200 that contains a challenge is still a failed fetch.

Will user-triggered fetchers also get blocked?

They can be. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Neither statement stops a firewall from returning 403 to them. So a user who asks an assistant to read your page may still get nothing if your security layer blocks the fetcher. Check these agents in your logs alongside the search crawlers.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.