AI Crawler Fix Tickets: Templates Developers Can Close and Verify

Most AI visibility tickets die because they ask for an outcome nobody can verify. "Make us show up in ChatGPT" has no acceptance criteria. A developer can't test it, so it sits in the backlog.

Access fixes are different. They're binary and mechanically checkable. A crawler either gets a 200 or it doesn't. A price is either in the raw HTML or it isn't. That's why they're the levers with evidence behind them, unlike on-page rewrites (see rewrites vs access fixes for the comparison).

Every ticket below uses the same five fields:

• Problem: the specific block, observed, with the URL and agent.
• Change: the exact edit, in the file or console where it lives.
• Acceptance criteria: pass or fail statements, no judgement calls.
• Verification test: numbered steps anyone can rerun next month.
• Out of scope: what this ticket does not promise. Always write: "This removes a block. It does not guarantee the page is retrieved or cited."

That last line protects the developer. They own the fix, not the model's answer.

Problem: robots.txt blocks a retrieval bot, often through a blanket AI block meant for training crawlers.

Change: add per-agent groups. OpenAI's crawler page says GPTBot is for training, OAI-SearchBot is for ChatGPT search, and "each setting is independent of the others." So you can opt out of training and stay eligible for search. A hypothetical example for a company that made that choice:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The training decision (GPTBot, ClaudeBot, Google-Extended) is a policy call for leadership. Put it in the ticket as a stated decision, not a developer guess. The per-bot robots.txt guide lists each token and what it controls.

Acceptance criteria

• Each named retrieval bot resolves to Allow for the paths that matter.
• No wildcard group overrides a named group by accident.
• /robots.txt returns 200 as plain text.

Verification test

1. Fetch the live file: curl -s https://example.com/robots.txt.
2. For each agent, find its named group. If none exists, it falls back to User-agent: *.
3. Confirm the rule for /, /pricing and one product URL.
4. Paste the output into the ticket.

Note: user-triggered fetchers behave differently. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. More in user-triggered AI fetchers.

Problem: robots.txt allows the bot, but the firewall, CDN or bot manager returns a 403, a rate limit or a challenge page. Nothing in any dashboard flags it.

OpenAI's help centre states eligibility as two conditions: allow OAI-SearchBot in robots.txt and confirm your host or CDN allows traffic from OpenAI's published searchbot IP addresses. Defaults can also change under you. From 15 September 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers, while Search stayed allowed.

Change: allow the verified retrieval bots in the bot-management rules. Prefer the vendor's verified-bot category or user agent plus published IP ranges. OpenAI publishes IP lists but warns against IP-only allowlisting.

Acceptance criteria

• OAI-SearchBot, PerplexityBot, Claude-SearchBot and bingbot receive 200 on sample URLs.
• No JavaScript challenge or CAPTCHA HTML is served to them.
• /robots.txt itself is reachable to each bot.

Verification test

1. Pick three URLs: home, pricing, one deep page.
2. Request each with the crawler's documented user-agent string, for example curl -s -o /dev/null -w "%{http_code}" -A "<OAI-SearchBot UA>" https://example.com/pricing.
3. Record the status code. 200 passes. 403, 429 or a challenge page fails.
4. Confirm in CDN or origin logs, filtered to verified bots, that real crawler requests now get 200.

Step 4 matters. A spoofed user agent from an office IP may be treated differently from the real bot. The AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away, if you want a second reading.

Problem: a fact you want quoted (a price, a plan limit, a one-line definition) only appears after JavaScript runs.

Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them, so they "can't read client-side rendered content." Googlebot renders JavaScript, so Google often looks fine while other engines see an empty shell. This is hygiene, not optimization: it decides whether the fact exists for the crawler at all.

Change: render the named facts in the initial HTML response, through server-side rendering, static generation or prerendering. Scope the ticket to specific strings, not "migrate to SSR." Pair it with JavaScript rendering for AI crawlers if the team needs background, and with fixing AI pricing errors if the fact is a price.

Acceptance criteria

• Each listed fact string appears in the raw HTML with JavaScript disabled.
• The raw value matches the rendered value exactly.
• Accordion, tab and "load more" content holding these facts is in the initial HTML.

Verification test

1. List the exact strings in the ticket, for example "$49/mo" and "25 keywords".
2. Fetch raw HTML: curl -s https://example.com/pricing > raw.html.
3. Search for each string in raw.html. Every one must be present.
4. Load the page in a browser and confirm the rendered values match.

Problem: a template carries a directive that hides content from AI answers. Common culprits: noindex, nosnippet, max-snippet:0, an X-Robots-Tag header, or data-nosnippet wrapped around more than the cookie banner it was meant for.

Google says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode. Bing says content marked data-nosnippet is still indexed but "excluded from snippets and AI summaries." Both are deliberate controls. The ticket only removes the ones nobody intended.

Change: remove or narrow the directive on the affected template. List which URLs should keep it on purpose. For a wider list of culprits, see code issues that stop AI indexing.

Acceptance criteria

• No unintended noindex, nosnippet or max-snippet:0 in meta tags or response headers on in-scope URLs.
• data-nosnippet wraps only the elements listed as intentional.
• Canonical tags point to the indexable version.

Verification test

1. Fetch headers: curl -sI https://example.com/pricing and check for X-Robots-Tag.
2. Search the raw HTML for robots, nosnippet and data-nosnippet.
3. Compare matches against the intentional list in the ticket.
4. In Google Search Console, inspect one URL and confirm it is indexable.

Run through this before a ticket leaves your hands. If an item fails, the ticket isn't ready.

• One block per ticket, with one owner.
• The affected URLs are listed, not described.
• The agent is named exactly (OAI-SearchBot, not "ChatGPT").
• Training versus search is a documented decision, signed off.
• Acceptance criteria are pass or fail statements.
• The verification test has numbered steps and expected output.
• "Before" evidence is attached: status codes, raw HTML grep, robots.txt output.
• Out of scope states that no citation or ranking is promised.
• The test is saved somewhere reusable, such as a script from AI SEO scripts you can run yourself.

Attach the "before" evidence every time. It turns the ticket from a request into a bug report, and bug reports get prioritized.

What good looks like

• Every retrieval bot gets 200 on sample URLs, confirmed in logs.
• Named facts appear in raw HTML, matching the rendered page.
• The only suppression directives left are on the intentional list.
• The ticket closes with pasted test output, not "looks fine."

What bad looks like

• The ticket closes on a config change with no request test.
• robots.txt allows a bot that the CDN still challenges.
• The fact renders server-side on one template but not its siblings.
• Someone measures success by whether an AI answer changed next week. That depends on retrieval and the model, which nobody outside the engine controls.

How often to repeat

1. After any deploy that touches templates, rendering, headers or robots.txt.
2. After any CDN, WAF or bot-management change, including vendor default changes like Cloudflare's September 2026 update.
3. Monthly, as a scheduled sweep of the four tests on your top URLs.

These blocks come back silently. A rerunnable test is the only way to know they haven't.

Should the robots.txt ticket allow GPTBot? That's a business decision, not a visibility fix. GPTBot governs training use only, according to OpenAI's crawler documentation. OAI-SearchBot controls whether your pages can appear in ChatGPT search. Blocking GPTBot does not remove you from ChatGPT search, and allowing it is not documented to help you get cited. Record the decision in the ticket so the developer isn't guessing.

Why can't the developer just check the firewall config? Config review misses rule interactions, inherited defaults and challenge pages. A real request shows what the crawler actually receives. OpenAI's own guidance frames eligibility as robots.txt permission plus CDN reachability, and bot-management systems can block legitimate crawlers with 403 responses. Test with crawler user agents, then confirm with logs filtered to verified bots, because a spoofed request may be treated differently.

Does server-side rendering matter if Google already sees our content? Yes. Googlebot renders JavaScript, so Google can look healthy while other engines miss the same content. Vercel found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. If your price only appears after hydration, those crawlers never see it. The raw HTML test in Ticket 3 shows the difference in seconds.

How do we know a fix worked if AI answers don't change? Judge the ticket by its verification test, not by an answer. The fix removes a block, so success means the crawler gets a 200, the fact is in raw HTML, or the directive is gone. Whether an engine then retrieves or cites the page depends on the query and the model. Nobody outside the engine controls that step.

Who should own these tickets? Split ownership by system. Robots.txt and directive tickets usually belong to whoever owns templates and site config. WAF and CDN tickets belong to infrastructure or security, since they control bot management. Rendering tickets belong to the front-end team. Marketing owns the problem statement, the URL list and the training policy decision.