SEO in the Age of AI: What Still Matters, and What Never Did

Search engines and AI answer engines share the same front door. Before any engine can quote your page, a crawler has to fetch it and an index has to hold it. That part of SEO did not go away. It now applies to many more crawlers than Googlebot.

It helps to split an AI answer into four stages:

• Parametric: what the model knows with no web access, frozen at a training cutoff. Nobody changes this in a quarter.
• Access: whether a crawler may fetch and store your page. You and your CDN control this.
• Retrieval: whether your page enters the model's context for a given question. The engine controls this.
• Generation: which retrieved pages the model uses and cites. The model decides.

For Google, the rules are written down. Google's AI features documentation says a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in AI Overviews or AI Mode, and that there are no additional technical requirements. Classic Google SEO is the entry ticket for Google's AI surfaces.

For ChatGPT, Perplexity and Claude, the entry ticket is a different set of crawlers, each with its own rules. What changed is the number of gates. What did not change is that the gates are technical, checkable, and mostly about removing blocks. For the broader argument, see whether SEO is still relevant with AI answers.

These are the levers with a documented mechanism. Each one works by removing a block, not by adding a signal.

Render the facts that matter in the HTML

Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of their requests) but do not execute them, so they cannot read client-side rendered content. Googlebot renders JavaScript, which is why this gap stayed hidden for years. If your prices, product names or definitions load only after hydration, most AI crawlers see an empty page. Server-side rendering or prerendering fixes that.

Set robots.txt per bot

OpenAI's crawler page says each setting is independent. GPTBot governs training use. OAI-SearchBot governs whether you appear in ChatGPT search answers. You can allow one and block the other:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

PerplexityBot, Claude-SearchBot and Meta-WebIndexer each have their own tokens too. There is no robots.txt token for Google AI Overviews: Googlebot rules cover all Google surfaces.

Make sure the network lets crawlers in

OpenAI's help page says eligibility needs two things: allow OAI-SearchBot, and confirm your host or CDN allows traffic from OpenAI's published IP ranges. A firewall can return 403 to a crawler that robots.txt allows. From 15 September 2026, Cloudflare set training and agent crawlers to blocked by default on ad-displaying pages for new domains and existing free-tier customers, so a site can lose AI access without anyone editing it.

Stay indexed, with snippets allowed

noindex, nosnippet, max-snippet:0 and data-nosnippet keep content out of AI answers on Google and Bing. Bing's index also backs Microsoft Copilot. Treat Google and Bing as two separate index checks.

State specific facts plainly

A price, a number or a one-line definition on a fetchable page gives an engine a sentence it can quote. That is the only on-page content habit with a positive experimental signal.

A lot of SEO advice was inherited, not measured. AI search did not kill these rules. It exposed that they were never levers.

Two cautions keep this honest. First, don't overcorrect. Google says existing schema does no harm, so removing it is not supported either. It is simply not a lever for AI answers. Ahrefs' study also only covered pages that were already cited heavily, so whether schema helps a page enter the set is untested.

Second, the famous GEO uplift numbers come from lab setups where the content was already inside the model's context. C-SEO Bench found most conversational SEO rewriting methods were largely ineffective and often harmful. Related questions on backlinks in AI search and domain age follow the same pattern: strong association with brand size, no proven lever.

Be plain with your team about where your effort lands.

In your control

• Whether each crawler is allowed in robots.txt.
• Whether your CDN and firewall return a 200 to verified crawlers.
• Whether key content exists in the raw HTML.
• Whether pages are indexed and snippet-eligible in Google and Bing.
• Whether your own pages state the same price, founding year and product names everywhere.
• Whether other sites that engines cite mention you. That is an association in the evidence, not a promised effect.

Not in your control

• What a model learned before its training cutoff.
• Whether an engine decides to search the web for a given question at all.
• The queries it actually runs. OpenAI says ChatGPT search rewrites your question into one or more targeted queries.
• Where your page lands in the model's context, and which retrieved pages it cites.

Organic rank is a partial bridge between the two lists. Ahrefs found that 37.9% of URLs cited in AI Overviews also appeared in the top 10 results (March 2026). That leaves most citations coming from further down, so ranking helps but does not decide. As of October 2026, Google's AI reporting in Search Console shows impressions only, with no position data, so any "AI rank" number has nothing to be checked against. The SERP itself has changed shape, and so has what you can measure on it.

Do these in order. Each step depends on the one before it.

1. Check index status engine by engine. Open Google Search Console and Bing Webmaster Tools separately. One being healthy tells you nothing about the other.
2. Read robots.txt per bot. Confirm OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot and bingbot are allowed. Decide training bots (GPTBot, ClaudeBot, Google-Extended) as a separate policy choice.
3. Test reachability as each crawler. Request a real page with each crawler's user agent and look for 403s, challenges or CAPTCHAs. The AI Crawler Checker does this and shows which crawlers your firewall turns away.
4. Diff raw HTML against the rendered page. List every price, heading and product name that appears only after JavaScript runs, then move it into the server response.
5. Hunt for accidental suppression. Search templates for noindex, nosnippet, max-snippet and data-nosnippet. These often arrive from a template meant to hide something else.
6. Make your facts consistent. Put the same price, plan names and company details on every page that states them, in plain sentences.
7. Then measure each engine separately. The free checker on this site shows whether AI engines name you for up to three buyer questions, as a snapshot.

Smaller teams can start with steps 1, 2 and 4. The guide to SEO basics for a small business site covers that shorter list.

Keep

• Writing pages that answer a real question for a real buyer. Topical relevance is the one content factor several independent studies agree on.
• Clean technical SEO: indexable pages, sensible canonicals, no accidental noindex.
• Server-rendered content for anything you want quoted.
• Clear, specific facts: prices, numbers, one-line definitions.
• Earning mentions on the sites engines already cite in your category.
• Checking Google and Bing as separate channels.

Drop

• Reporting a domain authority or E-E-A-T score as if an engine reads it.
• Adding schema, FAQ markup or llms.txt in the hope of AI citations.
• Rewriting pages into an "AI style", chunking them, or stuffing long-tail variants.
• Treating a blanket AI block in robots.txt as a harmless default.
• Tracking an "AI position" in Google.
• Testing only the prompts you typed and assuming those are the queries engines run.

Blocking every AI bot at once. A blanket block that includes OAI-SearchBot removes you from ChatGPT search answers. Many teams meant to stop training use only. The answer to whether to block GPTBot walks through the split.

Trusting robots.txt alone. Robots.txt is necessary but not sufficient. A WAF or CDN can block a crawler that robots.txt allows, and nothing in your dashboards will show it. Some user-triggered fetchers, such as Perplexity-User, generally ignore robots.txt rules by their operator's own account.

Mass-publishing to shortcut AI search. In Otterly's experiment, two new domains with roughly 1,000 AI-generated posts each were deindexed by Google after about 14 and about 90 days, with no manual action shown. Microsoft Copilot kept citing one site through Bing, accounting for 2,718 of its 3,934 tracked citations, while Google's AI surfaces almost never cited either. It is one pair of sites run by a tracking vendor, but it matches Google's scaled content abuse policy, and losing Google cost far more than Copilot gave back.

Reading one blended score. Engines behave differently. In the Otterly test, one engine cited the sites thousands of times while four barely did. Measure each platform on its own.

Gaming the answer. Hidden text, prompt injection and content cloaked for AI crawlers exist, and research shows manipulated pages can sway search agents once retrieved. They are illegitimate, they break spam policies, and they put your brand at risk. Don't use them.

For where this heads next, read what we can say about the future of SEO with AI.

Should I remove the schema markup I already have? No. Google says existing structured data does no harm, and it can still support rich results in classic search. The point is narrower: schema is not a lever for AI answers. Google says it isn't required for generative AI search, and Ahrefs' matched test found no uplift. Keep what works for your search listings, and stop adding markup in the hope of AI citations.

Does blocking GPTBot remove me from ChatGPT search? No. OpenAI says its crawler settings are independent. GPTBot controls whether your content may be used for training. OAI-SearchBot controls whether your pages can appear in ChatGPT search answers. You can block GPTBot and allow OAI-SearchBot. The reverse also holds: allowing GPTBot does nothing to help you appear in ChatGPT search.

If I rank in Google's top 10, will AI Overviews cite me? Not necessarily. Ranking makes you eligible, but Google says indexing and serving aren't guaranteed. Ahrefs found 37.9% of URLs cited in AI Overviews also appeared in the top 10 (March 2026), so most citations come from pages ranked lower. Good rankings help with eligibility. They do not decide which pages the AI feature picks.

Can I track my position inside Google's AI answers? Not reliably. As of October 2026, Google's Search Console report for AI Overviews and AI Mode shows impressions only, with no clicks, click-through rate or position. Google also says third-party tools don't have access to its internal ranking data. Any AI position number is a model built by a tool vendor, with no ground truth to check it against.

Do AI engines care about domain age or authority? No engine documents using domain age or a domain authority score. Domain Authority is a Moz metric, and Google publishes no site-level authority score. Large, well-known brands do appear far more often in AI answers, but that reflects brand prominence, which nobody can switch on. The practical route is being named on the sites engines already cite in your category.