What Is llms.txt, and Does Any AI Engine Actually Read It?

llms.txt is a proposal published at llmstxt.org. The idea is simple: a site puts a Markdown file at /llms.txt that gives a short summary of the site and links to the pages a language model should read first. The pitch is that a model gets a clean, curated map instead of wading through navigation, scripts and boilerplate.

The word that matters is proposal. robots.txt is a formal standard (RFC 9309) that crawlers commit to honoring. llms.txt has no equivalent commitment from the companies that run AI search. Anyone can publish the file. Whether anything reads it depends entirely on the engine, and engines decide that, not publishers.

For a short definition, see the llms.txt glossary entry. For how the two files differ in status and function, see llms.txt vs robots.txt.

Google is the only major operator that has said anything direct, and what it said is negative. Its guide to generative AI features on Search (July 2026) states: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

That statement covers Google Search only. It is not a statement about ChatGPT, Claude or Perplexity. But those vendors have not filled the gap. As of our August 2026 review of vendor documentation, none of OpenAI, Anthropic, Perplexity, Microsoft or Meta documents parsing llms.txt. What they do document are crawler user agents and robots.txt rules.

The research picture matches. A July 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found no technique with a stable, cross-platform causal effect on discoverability, and our evidence review found no controlled test of llms.txt at all. The honest summary: no engine says it reads the file, and nobody has measured an effect.

It helps to split an AI answer into four stages, because llms.txt would only matter at one of them.

1. Parametric. What the model knows with no web access, frozen at a training cutoff. Nothing you publish this quarter changes it.
2. Access. Whether a crawler is allowed to fetch and store your page. robots.txt, your firewall, your CDN and your rendering decide this.
3. Retrieval. Whether your page enters the model's context for a given question. The engine rewrites the user's prompt into its own queries and picks candidates from its index.
4. Generation. Which retrieved pages the model uses and cites.

A file like llms.txt would have to influence access or retrieval to matter. But access is governed by robots.txt and network rules, and retrieval runs on each engine's own index. If an engine does not fetch /llms.txt, the file sits outside all four stages.

The access stage is where documented mechanisms live. OpenAI says GPTBot (training) and OAI-SearchBot (ChatGPT search) are independent settings. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them, so content that only appears after scripts run can be invisible to them. Those are concrete, checkable facts. llms.txt has no comparable documentation.

In your control

• Per-bot robots.txt rules. You decide which retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Googlebot) can fetch your pages, separately from training bots like GPTBot.
• Network reachability. A firewall or CDN can block a crawler that robots.txt allows. OpenAI's help docs say eligibility needs both: allow OAI-SearchBot and make sure your host or CDN accepts traffic from OpenAI's published IPs. Since 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains and existing free-tier customers, so a site can lose AI access without anyone editing it.
• Rendering. Put your load-bearing facts (prices, product names, definitions) in the HTML the server sends, not only in JavaScript.
• Suppression controls. noindex, nosnippet, max-snippet:0 and data-nosnippet keep content out of AI answers on Google and Bing. Check they are not applied by accident.
• Specific facts. A clear sentence stating a price, a number or a one-line definition on a fetchable page gives an engine something it can quote.

Not in your control

• Whether an engine runs a web search for a given question at all.
• The queries it generates, and where your page lands in its candidate list.
• Which retrieved pages get cited.
• Whether any engine ever starts reading llms.txt.

Nobody can promise you a citation. What you can do is remove the blocks that make one impossible.

If you came here because llms.txt was on an AI SEO checklist, replace that line item with these, in this order.

1. Check robots.txt per bot. Confirm the retrieval bots you care about are allowed. A blanket AI block often removes ChatGPT search eligibility by accident. A setup that allows search but opts out of training looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

2. Check that crawlers can actually reach you. Request your pages as each AI crawler and look for 403s, challenges or CAPTCHAs. Make sure /robots.txt itself is reachable; the robots.txt reachability playbook walks through it. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
3. Diff raw HTML against the rendered page. Anything that appears only after JavaScript runs may not exist for most AI crawlers. Fix the facts that matter first.
4. Audit suppression directives. Search templates for noindex, nosnippet, max-snippet and data-nosnippet.
5. Check index status engine by engine. Use Google Search Console and Bing Webmaster Tools separately. Copilot grounds on Bing, and Google's AI features need a page to be indexed and snippet-eligible.
6. Make your key facts consistent. State prices, plans and product names the same way on every page, in plain HTML.

Then, if you still want an llms.txt, publish one. It costs little and, per Google, does no harm.

• Treating llms.txt as a ranking or citation lever. No engine documents reading it. Selling it as a fix sets an expectation nothing can meet.
• Confusing llms.txt with robots.txt. llms.txt cannot block or allow any crawler. Access rules live in robots.txt and in your network settings.
• Allowing GPTBot to get into ChatGPT search. GPTBot governs training. ChatGPT search uses OAI-SearchBot.
• Building parallel Markdown copies of every page. Google says you don't need Markdown versions to appear in Search. More on that in should you offer Markdown versions of your pages.
• Rewriting or chunking content for LLMs. Google rejects both as requirements and says there is no ideal page length. See should you chunk content for LLMs.
• Letting llms.txt drift from the site. If you keep one, an outdated price there is one more place for a wrong fact to live.

Google's same guide says: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files." Some developer tools and coding assistants let users point a model at an llms.txt by hand, and that is a legitimate reason to keep one for documentation sites.

So the decision is practical, not strategic. If you already have one, leave it and keep it accurate. If you don't, publishing one is optional and low priority. Do not pay for one as a visibility service, and do not let it displace the access checks above. If you are weighing a generator, read this note on free llms.txt generators first. For the wider picture on what moves ChatGPT specifically, see ChatGPT visibility factors, graded by evidence.