Free AI SEO Checker: Test One Page in Minutes

An AI engine can only quote a page it can fetch, store and read. That makes access the first thing to test, and the only part of AI visibility with strong evidence behind it. A critical survey of 45 GEO studies (arXiv, July 2026) found no technique with a stable cross-platform effect on discoverability. What does hold up is removing blocks.

On a single page, that means four checks:

• Reachability. Does each AI crawler get a 200, or a 403, a 429 or a CAPTCHA?
• robots.txt per bot. Rules apply to each user agent separately, and each bot has a different job.
• Content in raw HTML. Are your price, product names and key claims in the HTML before any script runs?
• Robots meta and snippet directives. Do noindex, nosnippet, max-snippet:0 or data-nosnippet hide content from AI answers?

If any of these fails, nothing else on the page matters to that engine.

You need a terminal and the URL of one page that matters, such as your pricing page.

1. Read robots.txt. Run curl -s https://example.com/robots.txt and look for groups naming OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Claude-SearchBot, Meta-WebIndexer and Googlebot. A healthy split for ChatGPT search with training opted out looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

2. Request the page as each crawler. Copy the full user-agent string from each operator's crawler page, then run curl -sI -A "<user-agent string>" https://example.com/pricing. Note the status code. Repeat per bot. Our guide to testing whether your firewall blocks AI bots covers the challenge pages to watch for.
3. Check the raw HTML. Run curl -s https://example.com/pricing | grep -i "49" (use your own price or product name). If the fact is visible in a browser but missing here, it loads through JavaScript.
4. Check directives. Run curl -sI on the page and look for an X-Robots-Tag header. Then search the HTML for name="robots" and data-nosnippet.
5. Check size. Run curl -s https://example.com/pricing | wc -c. Google's March 2026 crawler post says Googlebot fetches up to 2MB per URL and ignores the rest. The page size check explains when this matters, which is rarely.

Reachability

A 200 passes. A 403, a 429 or an HTML challenge page means the bot is blocked at the network layer, whatever robots.txt says. OpenAI's help centre states eligibility for ChatGPT search needs both: OAI-SearchBot allowed in robots.txt, and a host or CDN that lets OpenAI's published IP ranges through. Note that Cloudflare changed its defaults on 2026-09-15 to block training and agent crawlers on ad-displaying pages for several groups of sites, so a site can lose access without anyone editing it.

robots.txt

OpenAI's crawler page says each setting "is independent of the others". Blocking GPTBot stops training use. It does not remove you from ChatGPT search. Blocking OAI-SearchBot does. As of October 2026, there is no robots.txt token for Google AI Overviews or AI Mode. Googlebot rules and Search Console's controls cover those. Work through each agent with the robots.txt checker for AI crawlers.

Raw HTML

Vercel's crawler study found ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of requests) but do not execute them. Googlebot renders; most AI crawlers don't. Any fact missing from the raw HTML needs server-side rendering or prerendering. The JavaScript rendering check shows how to diff raw against rendered text.

Directives

Google and Bing document noindex, nosnippet, max-snippet and data-nosnippet as the controls that keep content out of AI answers. If you find one you didn't mean to set, often inherited from a template, remove it.

Many free checkers hand back a score built from schema presence, an E-E-A-T rating, readability or chunking. The primary sources do not back any of these as AI levers.

• Schema. Google says structured data isn't required for generative AI search. Ahrefs' matched-control test of 1,885 pages found AI Overview citations fell 4.6% after schema was added. That study only covered pages already heavily cited, so the effect on new pages is untested, not proven negative.
• E-E-A-T. Google says E-E-A-T isn't a specific ranking factor. There is no score for a tool to read.
• AI-friendly writing and chunking. Google's AI optimization guide says you don't need to write in a specific way for generative AI search, and that there is no ideal page length.

A score built on these inputs grades things the engines either ignore or never document. Existing schema does no harm, so there's no reason to strip it. It just isn't the thing to fix first. For more on what analyzer scores mean, read AI SEO analyzers explained.

Passing removes blocks. It does not get you retrieved or cited. Engines decide per query whether to search at all, and they rewrite prompts into their own queries, which nobody outside the engine can see.

A curl request from your laptop also isn't the real crawler. Firewalls may treat a spoofed user agent from a residential IP differently from the real bot's published IP range, which is one reason crawler checkers give different results. Index membership in Google or Bing needs Search Console or Bing Webmaster Tools, not a crawl.

And one test is a snapshot. CDN defaults and templates change, so re-run it after deploys and settings changes. Whether a one-time check is enough depends on how often your stack changes. To see what engines actually say about you, the free checker on this site asks up to three buyer questions and scores the answers.

Running curl for six bots on every key page gets slow. 99Visibility's free AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away. Use it for the reachability step, then confirm robots.txt rules and raw HTML content with the manual steps above. Fix what fails, starting with any 403 to a search bot, then any key fact that only appears after JavaScript.