Should I fix crawler access before tracking AI visibility?

AI visibility has four stages: what the model already knows (parametric), whether a crawler can fetch your page (access), whether the page enters the answer's context (retrieval), and what the model cites (generation). Access is the only one of these where you hold direct, checkable control.

If access is broken, every later number inherits the break. Say ChatGPT never names you for your core buyer questions. That could mean competitors are stronger, or it could mean your CDN returns a 403 to OAI-SearchBot. A tracking chart cannot tell those apart. You would spend months watching a flat line caused by a config setting.

The evidence also points the same way. The levers that hold up under scrutiny are access levers, and they work by removing blocks. That makes access the cheapest work with the clearest consequence, and the right thing to do before you measure. The practical order of work for GEO follows the same logic.

Run these checks in sequence. Each one depends on the one before it.

1. robots.txt, per bot. Check the rules for each retrieval bot by name: OAI-SearchBot (ChatGPT search), PerplexityBot (Perplexity), Claude-SearchBot (Claude search), Meta-WebIndexer (Meta AI) and Googlebot (all Google surfaces). The common error is a blanket AI block that removes search eligibility along with training.
2. Firewall and CDN reachability. Request a real page with each crawler's user agent and record the response. A 200 passes. A 403, a 429, or a challenge page fails, even when robots.txt says allow.
3. Rendering. Fetch the page with JavaScript off and compare it with the rendered page. Prices, product names and key claims that only appear after hydration are the ones to move into server-rendered HTML.
4. Suppression controls. Look for noindex, nosnippet, max-snippet:0 and data-nosnippet that a template applied by accident. The accidental noindex audit covers this step.
5. Baseline. Only now measure how engines describe you.

A minimal robots.txt that allows search bots while opting out of OpenAI training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The full AI crawler access audit playbook walks through each step. When you hand fixes to engineering, developer tickets for AI access fixes keeps them specific.

The operators document these gates themselves.

• robots.txt per bot. OpenAI's crawler page states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," and that each bot setting is independent of the others.
• Reachability. OpenAI's ChatGPT search help page says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from OpenAI's published searchbot IP addresses. robots.txt alone is not enough.
• Rendering. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them, so they cannot read client-side rendered content. Googlebot renders JavaScript, so this gap matters for non-Google engines.
• Google eligibility. Google states that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet, with no additional technical requirements.

Note what none of these say. Passing every gate buys eligibility, not a citation. Google adds that indexing and serving isn't guaranteed.

Some things sit outside your control, and you should know them before reading any numbers.

• User-triggered fetchers. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt rules. Blocking these is not reliable, and allowing them is not the search control.
• Answers without retrieval. Engines search at their own discretion. When they do not, the answer comes from training data frozen at a cutoff. Access fixes do not reach those answers this quarter.
• Changes you did not make. From 2026-09-15, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stayed allowed. A site can lose access without anyone editing it, so recheck access before each measurement cycle.
• Google AI Overviews. There is no robots.txt token that blocks AI Overviews or AI Mode on its own. Googlebot rules and the on-page snippet controls are the levers there.

Any single visibility score mixes answers given with and without live retrieval. Read it as a model of the situation, not a measurement of one cause.

Start with the first gate. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. Compare its result with what your robots.txt says, and fix every place they disagree.

Once fixes ship, set a baseline using the steps in setting an AI visibility baseline: the same buyer questions, each engine scored separately, logged out, with the date and the fix date recorded. The free checker on this site gives a snapshot for up to three buyer questions, which is a reasonable first data point.

Then decide what kind of tool you need. If your problem is access and wrong facts, a fixing workflow matters more than a trend line. The comparison of tracking tools and fixing tools sets out the difference.

Does allowing GPTBot help me appear in ChatGPT? No. OpenAI documents GPTBot as a training crawler only and claims no search benefit from allowing it. ChatGPT search uses OAI-SearchBot, and OpenAI states each setting is independent. You can allow OAI-SearchBot to stay eligible for ChatGPT search while disallowing GPTBot to opt out of training. Blocking GPTBot does not remove you from ChatGPT search.

How do I know if my firewall is blocking AI crawlers? Request a real content URL using each crawler's documented user agent and record the response. A 403, a 429, or an HTML challenge or CAPTCHA page means the crawler is turned away, whatever robots.txt says. Your own CDN or server logs, filtered to verified bots, show whether OAI-SearchBot, PerplexityBot and Claude-SearchBot are getting 200 responses in practice.

Will fixing crawler access get my brand cited? Nobody can promise that. Access determines whether your content can enter the candidate set at all. Which pages an engine retrieves and cites for a given question is decided by the engine, and topical relevance plays a large part. Fixing access removes a block you control. It does not control what the models say.

How soon after fixing access should I measure again? No published study gives a reliable wait time, so do not trust anyone who quotes one. Record the date each fix shipped, then re-run the same buyer questions on a fixed schedule against your baseline. Keep the questions, engines and logged-out conditions identical, so a change in the result reflects the site rather than the test.

Does JavaScript rendering matter for Google AI Overviews? Less than for other engines. Googlebot renders JavaScript, so client-side content is usually readable for Google surfaces. The rendering gap matters most for crawlers like OAI-SearchBot, ClaudeBot and PerplexityBot, which do not execute it. For AI Overviews, the gates are indexing, snippet eligibility, and avoiding accidental noindex or nosnippet directives.