XML sitemap: definition and how it works

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

An XML sitemap is a file that lists the URLs a site owner wants search engines to crawl. It helps crawlers discover pages, especially new or poorly linked ones. It does not get a page indexed, and it does not get a page cited in AI answers. The engine still decides what to fetch, what to store and what to use.

Key takeaways

An XML sitemap is a discovery aid: a list of URLs, not an instruction engines must follow.
Listing a page does not get it indexed, and an unindexed page can't appear as a supporting link in Google's AI Overviews or AI Mode.
Access gates come first: robots.txt permission per bot, firewall reachability and server-rendered content all matter more than the sitemap.
The lastmod date in a sitemap is self-declared, so changing it proves nothing about freshness.

Definition

An XML sitemap (usually served at /sitemap.xml) is a machine-readable list of the URLs on a site that the owner wants crawled. Each entry names a page. It can also carry an optional date, lastmod, saying when the page last changed.

The sitemap tells a crawler where pages are. It does not tell the crawler that a page is good, important or worth showing. Think of it as an address book handed to a visitor, not a guest list the visitor must honour.

For the specific question of whether AI engines use it, see does my XML sitemap help AI search engines find my pages.

How it works

A minimal sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
</urlset>

Crawlers find it in two ways:

A Sitemap: line in robots.txt, for example Sitemap: https://example.com/sitemap.xml.
Sitemap submission in Google Search Console or Bing Webmaster Tools.

The crawler reads the list, then decides on its own schedule which URLs to fetch. Most CMS platforms generate the file automatically. Your job is to keep it accurate: list canonical, indexable URLs and drop redirects, 404s and noindex pages.

Why it matters for AI visibility

A sitemap sits at the very front of the four stages of AI search: it helps a crawler discover a URL. Everything after discovery is decided elsewhere.

Access. A listed page still has to be fetchable. OpenAI's crawler docs treat each bot separately: OAI-SearchBot controls ChatGPT search, while GPTBot governs training only. A firewall or CDN can also turn away a crawler that robots.txt allows.
Rendering. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. A listed page whose key facts appear only after JavaScript runs may look empty to them.
Indexing. Google states a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode, and that indexing and serving isn't guaranteed.
Per engine. In Otterly's 2,000 AI blogs experiment, Google deindexed both test sites and its AI surfaces almost never cited them, while Bing kept them and Microsoft Copilot kept citing them. A sitemap submitted to both engines did not decide either outcome.

The next move: keep the sitemap clean, then check the gates that actually block pages. The AI Crawler Checker requests a page as each AI crawler and shows which ones get turned away.

Common misconceptions

"Submitting a sitemap gets my pages indexed." No. It gets them considered for crawling. Engines index what they choose.
"A sitemap gets me cited by ChatGPT or AI Overviews." Nothing in the evidence links sitemaps to citation. Citation depends on access, indexing, retrieval for a given query and the model's own choices.
"Bumping lastmod makes AI treat my pages as fresh." The date is publisher-declared and easy to change. No engine documents rewarding it.
"IndexNow replaces the sitemap." They do different jobs. As of October 2026, the IndexNow participating engines list has seven endpoints and Google is not one of them. See IndexNow vs XML sitemaps.
"A sitemap is a way to submit brand facts to AI." It lists URLs, nothing more. On that question, read whether there is an official way to submit brand facts.

Frequently asked questions

Do AI crawlers like OAI-SearchBot or PerplexityBot read sitemaps?

The AI vendors' crawler documentation in our evidence base doesn't describe how their bots use sitemaps. What they do document is per-bot robots.txt permission and, for OpenAI, the need for your host or CDN to allow their published IP addresses. Keep a sitemap in place as cheap hygiene, but treat bot permission and reachability as the checks that decide whether a page can be fetched at all.

Should I update lastmod every time I touch a page?

Only when the content meaningfully changes. The field is self-declared, so an engine has no reason to trust it blindly, and no engine documents using it to choose AI citations. An accurate date is useful internal hygiene. A date bumped on every deploy just makes the signal noisier and proves nothing about freshness to anyone.

Do I need separate sitemaps for Google and Bing?

No. One sitemap works for both. Submit it in Google Search Console and Bing Webmaster Tools, then check index status in each one separately. Otterly's experiment showed the two indexes can diverge completely, and Google's AI surfaces and Copilot followed their own engine's index. A healthy report in one tool tells you nothing about the other.

Can a page outside my sitemap still appear in AI answers?

Yes. Crawlers also discover pages through links, so a sitemap is not the only route in. The reverse is true too: a listed page can be skipped, left unindexed or never retrieved for a given query. Being in the sitemap is neither required nor sufficient for AI visibility.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.