llms.txt: definition of the proposed file

llms.txt is a plain-text file written in Markdown that a site owner places at the root of a domain, at /llms.txt. It gives language models a short, curated map of the site: what the site is, and which pages matter most.

The idea comes from a public proposal published at llmstxt.org. It is a suggestion for how sites could help models, not a protocol any engine has committed to. That separates it from robots.txt, which is a formal standard (RFC 9309) that crawlers are built to read.

You will also see it called an "AI text file" or an "llms txt file". The terms mean the same thing.

The proposal describes a simple structure. A site owner writes a Markdown file that includes:

• the name of the site or project
• a brief summary of what it covers
• lists of links to the pages a model should read first, with short notes

The file sits next to robots.txt at the domain root. In theory, a model or agent fetching the site would read it and know where the useful content lives, without crawling every page.

That is the theory. In practice, the file only does something if a crawler or AI system is built to request and use it. A related idea is publishing Markdown copies of pages, covered in whether to offer Markdown versions of your pages for AI.

It mostly doesn't, on current evidence. Google's guide to generative AI features in Search (July 2026) says you don't need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI capabilities, because Google Search doesn't use them. The same guide says doing so will neither harm nor help visibility. It also says it is completely fine to keep llms.txt files for other services that use them. Our digest of Google's generative AI optimization guide covers the rest of that document.

Outside Google, no engine documents reading the file. OpenAI's crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, and none of them is described as consuming llms.txt.

So what should you check instead? The levers with evidence are access levers:

1. Your key facts are in the HTML before JavaScript runs. Vercel found ChatGPT and Claude crawlers fetch JavaScript files but don't execute them (see why AI crawlers download JavaScript they don't run).
2. Your robots.txt allows the retrieval bots, such as OAI-SearchBot, per bot.
3. Your firewall or CDN doesn't turn those bots away.

For the full picture, read the guide on what llms.txt is and whether any engine reads it.

"An llms.txt file gets you cited"

No engine documents using it, and no published study shows it changes citations. Nobody can promise a citation from it.

"llms.txt controls which AI bots can crawl you"

It controls nothing. Crawl permission lives in robots.txt, set per user agent. The differences are laid out in llms.txt vs robots.txt.

"Google proved llms.txt is useless everywhere"

Google's statement covers Google Search only. The accurate line is narrower: Google ignores it, and no other engine says it reads it.

"You should delete it if you have one"

Google says it does no harm. There is no reason to remove a file that costs nothing to keep. Before you generate a new one, read what a free llms.txt generator can and can't do.

• robots.txt: the standard file that sets crawl rules per bot. This is where AI crawler access is actually decided.
• llms.txt vs robots.txt: a side-by-side of a standard and a proposal.
• Markdown versions of pages: the companion idea of serving pages as Markdown for models.
• The llms.txt guide: the longer explanation, with what each engine documents.