How can I see my website the way an AI crawler sees it?

An AI crawler sees your page as a single HTTP response. It gets a status code, some headers and the HTML your server sent. For most AI crawlers, that is the whole picture.

Three checks reproduce it:

1. View the raw HTML with JavaScript off. Use view-source in your browser, or disable JavaScript and reload. Anything missing here is likely missing for the crawler.
2. Compare it with the rendered page. Open the normal page and note which prices, product names, headings and claims appear only after scripts run. Our raw HTML vs rendered DOM check walks through the diff step by step.
3. Request the page with each crawler's user agent. Record the status code and whether the body is your content or a challenge page.

The first two checks tell you what a crawler can read. The third tells you whether it gets in at all. You need both, because a perfect server-rendered page behind a blocking firewall is still invisible.

Vercel's study of AI crawler traffic found that ChatGPT and Claude crawlers do fetch JavaScript files (11.50% and 23.84% of their requests) but don't execute them. In Vercel's words, "They can't read client-side rendered content." Independent testing in June 2026 found the same for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider.

Googlebot is the exception, and so is Applebot. Both render pages. That's why a site can look healthy in Google Search Console and still be blank to ChatGPT search.

What to look for in the raw HTML:

• Your pricing, plan names and key numbers
• Headings and the main body copy
• Content inside tabs, accordions or "load more" sections
• Robots meta tags. A noindex or nosnippet injected by a script may not be seen at all, which is covered in whether AI crawlers see robots meta tags added by JavaScript

If a load-bearing fact only exists after hydration, the fix is server-side rendering or prerendering for that content. This is the strongest publisher-controlled finding in the evidence. It decides whether your content exists for the crawler. It does not make an engine choose you.

Robots.txt rules are per bot, and so are firewall rules. Test each crawler separately. A minimal test with curl looks like this:

curl -s -o /dev/null -w "%{http_code}" -A "<exact user-agent string from the operator's crawler page>" https://example.com/pricing

Copy the exact strings from each operator's documentation: OpenAI's bots page for GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity's crawler page for PerplexityBot, and Anthropic's help article for ClaudeBot and Claude-SearchBot.

How to read the result:

• 200 with your real content: the crawler gets through.
• 403: a bot-mitigation block. OpenAI's own guidance says these systems "can mistakenly block legitimate crawlers, often returning 403 Forbidden errors."
• 429: rate limiting.
• 200 with a challenge or CAPTCHA page: also a fail. An automated crawler can't solve it.

OpenAI's help centre states the gate as a conjunction: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows OpenAI's published searchbot IP addresses. Robots.txt alone isn't enough. Our guide on checking whether GPTBot, ClaudeBot and PerplexityBot can reach your site covers the robots.txt half in more detail.

A user-agent test from your laptop is not the same request the real crawler makes. Keep these limits in mind.

• IP verification. Many firewalls check the source IP, not just the user agent. Your spoofed request may be blocked while the real bot passes, or the reverse. To confirm what real crawlers received, read your server or CDN logs, filtered to verified bots. The free AI bot log check and the guide to verifying a real AI crawler hit show how.
• Defaults you didn't set. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. A site can lose AI access without anyone editing it.
• User-triggered fetchers. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. These fetchers still hit your firewall, so test them too.
• Page size. Googlebot fetches up to 2MB per URL and ignores the rest. Google says most pages never come close. Check only if your HTML is unusually heavy.

None of this controls what an engine does after it reads your page. Access buys eligibility, not citations.

Work through the checks in order, and fix what's broken before anything else.

1. Diff raw HTML against the rendered page for your most important URLs: home, pricing, product and comparison pages.
2. Move any load-bearing fact that appears only after JavaScript into the server response.
3. Test OAI-SearchBot, PerplexityBot, Claude-SearchBot and the other crawlers you care about, and record each status code.
4. Where robots.txt allows a bot but the response is a 403 or a challenge, change the firewall or CDN rule, then confirm in your logs.
5. Recheck after deploys and after any CDN setting change.

If you'd rather not script the user-agent tests, the free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Is view-source the same as what an AI crawler sees? Close, for most AI crawlers. View-source shows the HTML your server sent before any script ran, which is what non-rendering crawlers read. It won't show whether a firewall blocks a specific bot, and it won't reflect user-agent or IP-based variations. Pair it with a user-agent request and a status code check to get the full picture.

Does Google see my site the same way ChatGPT does? No. Googlebot renders JavaScript, so Google can read content that appears after scripts run. ChatGPT's crawlers, as Vercel found, fetch JavaScript without executing it. Google also has no robots.txt token for AI Overviews or AI Mode. Eligibility there follows normal Google indexing and snippet eligibility, which Google says requires nothing extra.

If my user-agent test returns 200, am I definitely reachable? Not definitely. Your request comes from your IP, not the crawler's. A firewall that verifies bot IPs or uses reverse DNS may treat the real crawler differently. A 200 is a good sign. A confirmed 200 in your server logs, from a verified bot address, is the stronger evidence.

Should I allow GPTBot so ChatGPT can see my site? GPTBot controls training use only. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent of the others. You can allow OAI-SearchBot and block GPTBot, or the reverse. Decide on training separately, and test OAI-SearchBot if ChatGPT search is the goal.