What an AI SEO Expert Actually Knows

Real AI SEO expertise starts with knowing where an answer comes from. There are four stages, and a good expert keeps them apart:

• Parametric: what the model knows with no web access, frozen at a training cutoff. See parametric knowledge. Nobody changes this in a quarter.
• Access: whether a crawler may fetch and store your page. You and your CDN control this.
• Retrieval: whether your page enters the model's context for a given query. The engine decides, with some input from you.
• Generation: which retrieved pages get used and cited. The model decides.

Most weak advice blurs these. A tactic measured at the generation stage gets sold as a way to be found at all. An expert tells you which stage your problem sits in before recommending anything. That alone separates a diagnosis from a checklist.

robots.txt rules apply per bot, and each bot has a specific purpose. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-triggered fetches. It states that each setting is independent. So a site can opt out of training and still be eligible for ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The same split exists elsewhere, as of October 2026:

• Anthropic separates ClaudeBot (training) from Claude-SearchBot (search quality) and Claude-User.
• PerplexityBot builds Perplexity's search index. Perplexity says Perplexity-User generally ignores robots.txt.
• Meta's documentation is the only first-party statement that allowing a crawler helps citation: allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses."
• Google has no robots.txt token for AI Overviews or AI Mode. Googlebot rules are the control. Google-Extended covers Gemini apps training and grounding, not Search.

The most common real-world error is a blanket AI block that silently removes ChatGPT search eligibility. Your next move: read your robots.txt agent by agent and write down what each rule does.

Rendering is the strongest evidence in the field. Vercel's crawler study (published December 2024, updated February 2025) found ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of requests) but don't execute them. Independent re-testing in June 2026 found the same for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. Googlebot does render.

What it means: if your prices, plan names or product facts load only after JavaScript runs, those crawlers may never see them. The test is simple. Fetch the raw HTML with JavaScript off, compare it with the rendered page, and list what exists only after hydration.

The network layer is the gate most advice skips. OpenAI's help centre says eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses. A firewall can return 403 to a crawler that robots.txt allows, and no dashboard warns you. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, while Search stays allowed. A site can lose access without anyone editing it.

To see which crawlers your firewall turns away, the AI Crawler Checker requests a page as each one.

Google states that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet, with "no additional technical requirements." There is no separate AI path into Google.

The on-page controls that keep content out of AI answers are noindex, nosnippet, max-snippet:0 and data-nosnippet. Bing has supported data-nosnippet since October 2025, excluding marked content from snippets and AI summaries. These often arrive by accident, inherited from a template someone used to hide a cookie banner.

Google also offers a Search Console control for generative AI features. Inclusion is the default, so it is opt-out, and child properties inherit the parent's setting. It can't be seen from outside.

Check Google and Bing separately. In Otterly's 2026 experiment, Google deindexed two AI-generated sites and Google's AI surfaces stopped citing them, while Bing kept them and Copilot kept citing them. It was one pair of sites, but it shows one healthy index tells you nothing about the other.

An honest expert tells you what nobody can measure. As of October 2026:

• No AI rank exists for Google. Search Console's generative AI report shows impressions only, with no clicks, CTR or position.
• Retrieval is discretionary. ChatGPT searches at the model's discretion, and Claude.ai search is a user toggle. A single blended score averages answers given with and without live search.
• The real queries are hidden. OpenAI says ChatGPT search rewrites your prompt into one or more targeted queries. Prompt trackers test a proxy. Bing Webmaster Tools shows grounding queries, the queries the AI actually ran, but had no API as of August 2026.
• Cited is not the same as used. OpenAI's API returns more consulted sources than citations.

So a good report scores each engine separately, says whether search was on, and never claims a position. If someone shows you a single AI visibility number, ask what it averages.

Experts read the primary studies, not vendor summaries. A critical survey of 45 GEO studies (arXiv 2607.14035, July 2026) concluded that already-retrieved content can change its citation, but no technique shows a stable, cross-platform causal effect on being discovered. That is the central finding of the field.

Other results worth knowing:

• C-SEO Bench (NeurIPS 2025) found most conversational SEO methods largely ineffective and often harmful to ranking in the model's context.
• The original GEO paper's 30 to 40% band was measured in a lab engine with sources already loaded into context.
• Ahrefs' matched test of 1,885 pages found AI Overview citations fell 4.6% after adding schema. Every treated page already had 100+ AI Overview citations, so whether schema helps a page enter the set is untested.
• Ahrefs (March 2026) found 37.9% of URLs cited in AI Overviews also appeared in the first 10 results.
• Topical relevance is the factor several studies agree on.

The one positive on-page signal: stating a specific fact (a price, a number, a one-line definition) on a fetchable page gives the engine a sentence it can quote. For a fuller review, see what AI SEO research actually shows.

Many self-described experts still sell claims that primary sources contradict. If a pitch leans on these, treat it as a warning sign.

Don't overcorrect either. Google says existing markup does no harm, so ripping schema out isn't supported. The honest line is that it isn't a lever for AI answers. For guarantees specifically, read whether to trust an agency that guarantees AI citations.

Whether you hire or work in-house, the order is the same. Start with the gates, because nothing downstream matters if they are closed.

1. robots.txt per agent. Confirm OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer and Googlebot are allowed if you want those engines.
2. Network reachability. Request a page as each crawler and look for 403s, challenges or CAPTCHAs.
3. Rendering. Diff raw HTML against the rendered page. Server-render prices and key facts.
4. Suppression. Search templates for stray noindex, nosnippet and data-nosnippet. Check the Search Console generative AI setting.
5. Index status. Check Google Search Console and Bing Webmaster Tools separately.
6. Facts. Make prices, plans and company details consistent across your own pages, stated in plain sentences.
7. Measure per engine. The free checker on this site gives a snapshot of whether AI engines name you for up to three buyer questions. Where engines cite other sites instead of yours, being named on those sites is associated with appearing in the answer.

Steps one to five are mechanical and checkable. If you're weighing outside help, see do I need an expert for AI search, hiring an AI SEO consultant and what a fair AI SEO package covers.