Scripts for AI SEO Checks You Can Run Yourself

An AI answer that cites your site has passed four stages. The model's built-in knowledge (parametric) is frozen at a training cutoff. Access decides whether a crawler may fetch and store your page. Retrieval decides whether your page enters the model's context for a given question. Generation decides what gets used and cited.

You control one of those stages directly: access. The levers that hold up under scrutiny all live there, and they work by removing blocks rather than adding anything. A critical survey of 45 GEO studies (arXiv, July 2026) found no technique with a stable, cross-platform causal effect on discoverability. Access problems, by contrast, are binary and documented by the engines themselves.

That makes access the right job for a script. The checks are deterministic, cheap, and repeatable. If you want the background on how crawlers see a page, start with how to see your site the way an AI crawler does.

Three scripts cover the ground:

1. Fetch a page as each AI crawler and record the status code.
2. Diff raw HTML against rendered text.
3. Scan for noindex and snippet controls in headers and markup.

robots.txt rules are per bot, and each bot does a different job. OpenAI's crawler documentation says each setting "is independent of the others". So the test has to run per agent, and a block means something different for each.

Put the full user-agent strings, copied from each operator's crawler page, in a file called agents.txt, one per line. Then loop:

URL="https://example.com/pricing"
while IFS= read -r UA; do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" -A "$UA" "$URL")
  echo "$CODE  $UA"
done < agents.txt

Read the results this way:

• 200 for every agent: the user-agent rules pass. Move on.
• 403 for some agents only: a firewall or bot-management rule is turning them away.
• 429: rate limiting.
• 200 with a challenge page: open the body. A CAPTCHA or JavaScript challenge is a fail, because crawlers can't solve it.

Run the same loop against /robots.txt itself. Anthropic warns that IP blocking can stop it reading your robots.txt at all. For per-bot rule parsing, see testing robots.txt for each AI bot.

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of their requests) but do not execute them. Content that only appears after JavaScript runs is invisible to them. Googlebot is the exception, because it renders. The reason crawlers download scripts they never run is covered in why AI crawlers download JavaScript.

The test: fetch the raw HTML, render the same URL in a headless browser, and check whether your key facts appear in each.

import re, requests
from playwright.sync_api import sync_playwright

url = "https://example.com/pricing"
raw = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text
raw_text = re.sub(r"<[^>]+>", " ", raw)

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto(url)
    rendered = page.inner_text("body")

for fact in ["Starter plan", "per month", "Founded in"]:
    print(fact, fact in raw_text, fact in rendered)

Replace the list with the facts you care about: prices, plan names, product names, the one-line description of what you do. Any fact that prints False for raw and True for rendered exists only after hydration.

Two caveats. A phrase found only inside a script block is a grey zone, since Vercel notes that JSON in the initial HTML response may still be indexed. And content behind tabs, accordions or "load more" buttons often fails this test for the same reason.

Google and Bing document four on-page controls that keep content out of AI answers: noindex, nosnippet, max-snippet:0 and data-nosnippet. Bing says content marked data-nosnippet is still indexed but excluded from snippets and AI summaries. These often arrive by accident, inherited from a template.

Check the response header and the markup:

curl -sI "$URL" | grep -i "x-robots-tag"
curl -s "$URL" | grep -ioE "noindex|nosnippet|max-snippet:[0-9]+|data-nosnippet"

Every hit needs an owner who can say it was intentional. A data-nosnippet wrapped around your pricing table removes it from Bing's AI summaries while it still ranks.

One limit: these greps read raw HTML. A robots meta tag injected by JavaScript behaves differently, which this note on JavaScript-added robots tags explains.

Payload size is rarely a problem. Google's March 2026 crawler post says Googlebot fetches up to 2MB per URL and ignores bytes past that, and also says most of the web never comes close. curl -s -o /dev/null -w "%{size_download}" "$URL" prints the byte count. Flag only pages over the limit.

A script reports what your server returns. That is useful, and it has edges.

In your control, and checkable

• robots.txt rules per agent.
• Firewall, CDN and bot-management rules.
• Whether key facts are in the HTML before JavaScript runs.
• Noindex and snippet directives.

Not checkable from outside

• The real crawler's path. OpenAI's help centre says eligibility needs both OAI-SearchBot allowed in robots.txt and your host or CDN allowing traffic from its published IP addresses. Your laptop is not on those IPs. A firewall that verifies bots by IP or reverse DNS treats your spoofed request differently from the real one, in either direction. Your server logs, filtered to verified bots, are the ground truth.
• Index membership. Search Console and Bing Webmaster Tools show it. A crawl cannot.
• Google's generative AI inclusion control. Inclusion is the default, but a property can be opted out or inherit an opt-out from a parent property. Only Search Console shows it.
• Retrieval and citation. Engines rewrite questions into their own queries, and retrieval fires at the model's discretion. No script sees whether your page was used.

Also watch for changes you didn't make. From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and all free-tier customers. A script run before and after that date would show the difference.

Run the checks in the order a crawler meets your site.

1. Pick five to ten URLs. Homepage, pricing, your main product or service pages, and one article. Problems are often page-specific.
2. Run script one against /robots.txt and each URL. Note any agent that gets a non-200.
3. Compare against robots.txt. An agent that robots.txt allows but the server blocks is the silent failure. The firewall access test covers how to trace it.
4. Run script two on pages with prices or product facts. List every fact missing from raw HTML.
5. Run script three on the same URLs. Confirm every suppression directive is intentional.
6. Check Search Console and Bing Webmaster Tools for index status and the inclusion control.
7. File each failure as its own ticket with the URL, the agent, the observed response and the expected one. The format in developer tickets for AI access fixes gets them done.
8. Re-run after every deploy and CDN change. How often to re-run checks sets a sensible rhythm.

If you would rather not maintain scripts, the AI Crawler Checker requests a page as each AI crawler and shows which ones the firewall turns away.

• Treating a 200 as proof. It proves your user-agent rules pass. It says nothing about IP-based verification. Confirm in logs.
• Testing one network. Results can differ between a datacenter and a home connection. Run from both if your firewall scores reputation.
• Using only the bot token. Some rules match the full user-agent string. Copy it exactly from the operator's page.
• Confusing training and search bots. Allowing GPTBot does not help ChatGPT search. Blocking it does not remove you from ChatGPT search. OAI-SearchBot controls that.
• Expecting robots.txt to stop user-triggered fetchers. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores it.
• Running the render diff on Google only. Googlebot renders JavaScript, so a page that looks fine in Google can still be blank to most AI crawlers.
• Scoring things that aren't levers. A script that awards points for schema or an llms.txt file is measuring the wrong thing. Google says structured data isn't required for generative AI search and that Google Search ignores llms.txt.
• Checking once. CDN defaults, template changes and deploys break access without anyone editing robots.txt.

Do I need the full user-agent string or just the bot name? Use the full string. Firewall and bot-management rules sometimes match on the complete user agent, not just the token. Copy each string exactly from the operator's crawler documentation: OpenAI, Anthropic, Perplexity, Meta, Google and Bing all publish them. Keep them in one text file so you can update a single place when an operator changes a string, which happens.

Can a script tell me whether ChatGPT or Perplexity cites my site? No. A crawler script shows whether a bot can reach and read your page. Citation happens at answer time, after the engine rewrites the user's question into its own queries and decides whether to search at all. To see what engines say, you need to ask them buyer questions directly. Treat that as a separate check from access, with its own limits.

Why does my script get a 200 while the real crawler is blocked? Your request comes from your own IP address with a borrowed user agent. Many firewalls verify bots by IP range or reverse DNS, so they treat the real crawler and your spoofed request differently. OpenAI says eligibility requires allowing its published search bot IP addresses, not just the user agent. Check your server or CDN logs for genuine crawler requests and their status codes.

Should my script check for schema markup or an llms.txt file? You can record them, but don't score them as visibility factors. Google says structured data isn't required for generative AI search and that Google Search ignores llms.txt files. No other engine documents reading llms.txt. Existing markup does no harm, so there's no reason to remove it. Spend script effort on access, rendering and snippet controls, which have documented consequences.

How long do these checks take to set up? The status-code loop and the grep checks take minutes with curl. The render diff needs Python and Playwright installed, which is a short setup on most machines. Running all three against ten URLs is quick. The slower part is reading results against robots.txt and logs, then writing clear tickets for each failure.