Could Cloudflare's AI crawler settings be hiding my site from ChatGPT?
99Visibility Research · Updated October 21, 2026 · 6 min read
Short answer
Possibly, but usually not from ChatGPT search directly. Since 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search crawlers stay allowed. Your own zone rules or bot settings can still block a search crawler. Check your Cloudflare settings, send a test request, then confirm in your server logs.
Key takeaways
- Cloudflare's September 2026 default blocks Training and Agent crawlers on ad-displaying pages, and leaves Search crawlers allowed.
- ChatGPT search depends on OAI-SearchBot, not GPTBot, so blocking training alone does not remove you from ChatGPT search answers.
- OpenAI says eligibility needs both robots.txt permission and a CDN that lets its published searchbot IPs through.
- A user-agent test from your own machine can disagree with what real crawlers get, so your logs are the ground truth.
- Allowing crawlers makes you eligible. It does not guarantee you get retrieved or cited.
The short answer
Cloudflare can hide your site from AI answers, and you would not see it from the outside. The question is which crawler it turns away.
AI platforms run separate bots for separate jobs. OpenAI's crawler documentation lists GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for actions a user triggers. OpenAI states that "each setting is independent of the others." Blocking GPTBot keeps your content out of training. It does not remove you from ChatGPT search.
Cloudflare's September 2026 default targets the Training and Agent categories, not Search. So the default alone should leave search crawlers such as OAI-SearchBot able to reach you. The risk sits elsewhere: a custom firewall rule, a bot-fight setting, a JavaScript challenge, or an IP allowlist can still return a 403 to a search crawler that your robots.txt allows. That is the silent failure covered in robots.txt vs firewall: the two gates for AI crawlers.
The next move is a three-part check: your Cloudflare settings, a live request, and your server logs.
What changed on 15 September 2026
From 15 September 2026, Cloudflare sets Training and Agent crawlers to blocked by default on pages that display ads. Search crawlers remain allowed by default. The change covers three groups:
- All new domains onboarding to Cloudflare
- New sites added by existing customers
- All existing free-tier customers
If you fall into one of those groups and run ads, your AI crawler access may have changed without anyone on your team editing a setting.
Why Search is treated separately
Training crawlers collect content for future models. They send no visitors back. Search crawlers build the index that live AI answers draw on, and that index is where citations and links come from. Cloudflare's split follows the same line the operators draw. OpenAI, for example, says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," while GPTBot governs training only.
What the Agent category can affect
Agent-type fetches happen when a person asks an assistant to open a page. Blocking them will not remove you from a search index. It can stop an assistant from reading your page when a user pastes your URL. Check how Cloudflare categorises each named bot in your own dashboard rather than assuming.
Step 1: Check your zone's bot settings
Log in to the Cloudflare account that fronts your domain and review each layer that can turn a bot away. The full walkthrough is in reviewing CDN bot settings for AI crawlers.
- Open your AI crawler controls and note the setting for each category: Search, Agent, Training.
- Confirm which named bots sit in Search. You want OAI-SearchBot, PerplexityBot and Claude-SearchBot allowed if you want to appear in those engines.
- Review custom firewall (WAF) rules for anything that blocks by user agent, country, ASN or datacenter IP ranges.
- Check bot-fight or challenge settings. Automated crawlers can't solve a JavaScript challenge or CAPTCHA.
- If you use IP allowlisting, compare it with OpenAI's published searchbot IP list. OpenAI also warns against relying on IP-only allowlisting, so pair it with Cloudflare's verified-bot handling.
Write down what you find. You'll compare it against real traffic in the next two steps.
Step 2: Confirm with a real request
Settings tell you intent. A request tells you what happens. OpenAI's help page describes eligibility as a conjunction: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses.
Request a representative content URL using each crawler's documented user-agent string, then read the response:
- 200 with your real page content: pass.
- 403: a firewall or bot rule is blocking the crawler.
- 429: rate limiting.
- 200 with a challenge or CAPTCHA page: a fail, since the crawler sees no content.
Also request /robots.txt itself. Anthropic warns that IP blocking can stop its crawler from reading your robots.txt at all. The free robots.txt reachability check covers that piece, and the firewall testing playbook covers the rest.
The edge case that trips people up
A spoofed user agent from your laptop is not the real crawler. Cloudflare identifies verified bots by signals such as published IP lists and reverse DNS, not the user-agent string alone. Your test can be blocked when the real bot is allowed, or allowed when the real bot is blocked. Treat a test result as a lead, then confirm in logs. The AI Crawler Checker at 99Visibility's free tools requests a page as each AI crawler and shows which ones the firewall turns away.
Step 3: Confirm with your server logs
Logs are the ground truth. They show what real crawlers from real IP ranges actually received.
- Pull Cloudflare's firewall events or your origin access logs for the last 30 days.
- Filter to verified bots, and to user agents including OAI-SearchBot, PerplexityBot, Claude-SearchBot and Meta-WebIndexer.
- Group by status code. A run of 403s or challenge responses for a search crawler is a complete visibility failure for that engine.
- Compare dates. A change that starts on or after 15 September 2026 points at the new defaults.
If you're not sure how to read the logs, the free AI bot log check walks through it.
If access is clean
Access is only the first gate. If every search crawler gets a 200 and you still never appear, look at the other causes in why robots.txt allows AI bots but you never show up in ChatGPT, such as content that only appears after JavaScript runs. For Google's AI features, also check whether a Search Console setting is hiding you from AI Overviews.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the crawler for ChatGPT search, and states each setting is independent of the others. You can block GPTBot to opt out of training and still allow OAI-SearchBot so your pages stay eligible for ChatGPT search answers. The common mistake is a blanket AI block that catches OAI-SearchBot along with the training bots.
Does Cloudflare's default affect Google AI Overviews?
Not through the Training and Agent categories. Google AI Overviews and AI Mode use Googlebot, and there is no separate robots.txt token for them. To appear, a page must be indexed and eligible to show in Google Search with a snippet. If Googlebot can reach you, the Cloudflare AI default is not the gate for Google. Check Search Console and your snippet controls instead.
Is a robots.txt allow rule enough?
No. robots.txt states permission, but a firewall or CDN can still block the crawler at the network layer, often with a 403. OpenAI's own guidance says eligibility needs both the robots.txt permission and a CDN that allows its published searchbot IPs. Some user-triggered fetchers also handle robots.txt differently: Perplexity says Perplexity-User generally ignores robots.txt rules.
If I unblock every AI crawler, will ChatGPT start citing me?
Nobody can promise that. Removing a block makes your pages eligible to be fetched and indexed. Whether an engine retrieves your page for a given question, and whether it cites it, is decided by the engine. Access is the part you control, and fixing it removes a cause of absence rather than guaranteeing presence.
Why do two crawler checkers give me different results?
They send requests from different networks and identify themselves differently. Cloudflare can treat a request from a datacenter differently from a residential connection, and a spoofed user agent differently from a verified bot. That is why a checker result is a starting point and your logs, which record what the real crawlers received, are the final answer.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.