AI Crawler Access Audit: A Step-by-Step Playbook

An AI answer passes through four stages. The model's built-in knowledge (parametric) is frozen at a training cutoff, and nothing you change this quarter reaches it. Access decides whether a crawler can fetch and store your page. Retrieval decides whether that page enters the model's context for a given query. Generation decides which retrieved pages get used and cited.

Access is the only stage where you control the mechanism and the platform documents the consequence. The levers here are subtractive: each one removes a block. A July 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found no technique with a stable, cross-platform causal effect on discoverability. The access findings come from the operators themselves.

Keep the limit in view. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews or AI Mode, and adds that indexing and serving isn't guaranteed. The audit sets a floor. It is also why fixing crawler access before you track visibility saves you from measuring a site no engine can read. For the wider scope beyond access, see what a real AI SEO audit checks.

Pick 5 to 10 URLs that carry your key facts: homepage, pricing, main product pages, about page and your most visited docs. Get access to Google Search Console, Bing Webmaster Tools, your CDN or firewall dashboard and, ideally, your server logs.

Then fix the bot list. Each platform runs separate crawlers with separate jobs, as of October 2026:

Gate 1: robots.txt, bot by bot

1. Fetch yourdomain.com/robots.txt. Confirm it returns 200 and plain text, not an error page.
2. For each bot in the table, find the group that applies. A bot follows the group that names it, so a named group overrides your * defaults.
3. Look for blanket AI blocks. The most common error is a block meant to stop training that also disallows OAI-SearchBot, PerplexityBot or Claude-SearchBot.
4. Decide training and search separately. This is valid and common:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

There is no robots.txt token for AI Overviews or AI Mode. Googlebot rules control all Google Search surfaces at once.

Gate 2: firewall and CDN reachability

5. Request each key URL with each crawler's documented user agent. Record the status: 200, 403, 429, or a challenge or CAPTCHA page.
6. Request /robots.txt the same way. Anthropic warns that IP blocking stops it reading your robots.txt at all.
7. Open your CDN's bot settings. From 15 September 2026, Cloudflare blocks Training and Agent crawler categories by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search stays allowed by default. Your settings can change without anyone on your team editing them.
8. Confirm with logs. A request from your laptop with a spoofed user agent is not the real crawler, and a firewall that verifies bots may treat it differently. Filter server or CDN logs to verified bots and check their status codes. OpenAI publishes OAI-SearchBot's IP ranges at openai.com/searchbot.json.

OpenAI's help center makes ChatGPT search eligibility a conjunction: allow OAI-SearchBot and confirm your host or CDN allows traffic from its published IP addresses. The AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away. For other options, see tools that test whether your firewall blocks AI crawlers.

Gate 3: rendering

9. Fetch the raw HTML with no JavaScript (curl, or view-source in a browser). Search it for your price, product names, key claims and headings.
10. Compare it with the rendered page. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they can't read client-side rendered content. Googlebot renders, so this gate matters most for non-Google engines.
11. Check tabs, accordions and load-more buttons that fetch content after a click. Same problem, different fix.

Gate 4: index and snippet controls

12. Search the HTML and HTTP headers (including X-Robots-Tag) for noindex, nosnippet, max-snippet and data-nosnippet. Bing says content marked data-nosnippet stays indexed but is excluded from snippets and AI summaries.
13. In Search Console, inspect your key URLs for index status. If your property shows the generative AI inclusion control, check it. Inclusion is the default, but child properties inherit the parent's setting.
14. In Bing Webmaster Tools, confirm the same URLs are indexed. In Otterly's 2026 test of two AI-generated sites, Google's AI surfaces stopped citing them once Google deindexed them, while Copilot kept citing them because Bing kept them. Index status is engine by engine. The Copilot visibility checklist and the Google AI Mode checklist cover each side.

Copy this into your ticket tracker and mark each line pass or fail per URL.

• robots.txt returns 200 and plain text
• OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot and bingbot are allowed where you want search visibility
• Training bots (GPTBot, ClaudeBot, Google-Extended) are set by a deliberate decision, not a copied block
• Each retrieval crawler gets a 200 with real HTML, not a 403, 429 or challenge page
• Each crawler can fetch /robots.txt itself
• CDN bot categories match your intent after the September 2026 Cloudflare default change
• Verified-bot entries in your logs show 200s
• Prices, product names and key claims are in the raw HTML before JavaScript runs
• No key content sits behind clicks that load it later
• No noindex, nosnippet or max-snippet:0 on pages you want in AI answers
• data-nosnippet wraps only what you meant to hide
• Key URLs are indexed in Google and in Bing
• The Search Console generative AI control, where visible, is not opted out by inheritance

A bad result is specific and fixable. Route robots.txt and snippet issues to whoever owns the templates. Route 403s and challenges to your CDN or security admin, and ask them to allow verified bots rather than raw IP lists. Route rendering gaps to engineering: server-side rendering or prerendering of the content that carries your facts.

One failure can cancel the rest. A 403 to OAI-SearchBot removes you from ChatGPT search however clean your robots.txt is.

Access breaks quietly, so tie the audit to change rather than to a fixed calendar alone.

• After any robots.txt edit
• After any CDN, firewall or bot-management change, including plan changes
• After framework migrations or redesigns, which often move content into client-side rendering
• After template changes that touch meta robots tags
• When a vendor announces a new default, as Cloudflare did for September 2026

Between those events, rerun the 14 steps on your key URLs monthly. That cadence is a practical choice, not a measured threshold. It catches changes made by third parties before they sit unnoticed for a quarter.

Passing makes you eligible. Retrieval is per query and discretionary on every platform, and generation is the model's call. Nobody outside the engines controls either stage.

Some items don't belong in an access audit. Google says structured data isn't required for generative AI search, and that Google Search ignores llms.txt. No other engine documents reading llms.txt. Existing markup does no harm, so leave it, but don't treat it as an access fix.

Once access is clean, the next question is accuracy: what engines say about you. The ChatGPT brand visibility checklist runs from access to accuracy, and the free GEO audit covers both in order. The free checker on this site then shows whether engines name you for up to three buyer questions. It is a snapshot, not continuous tracking.