llms.txt vs robots.txt: A Standard vs a Proposal
99Visibility Research · Updated October 21, 2026 · 6 min read
Short answer
robots.txt is a published standard (RFC 9309), and OpenAI, Google, Perplexity, Anthropic and Meta each document how their crawlers treat it, bot by bot. llms.txt is a proposal for a Markdown file that summarizes a site for language models. Google Search ignores it, and as of October 2026 no AI engine documents reading it. Keep an llms.txt if you want one, but put the effort into robots.txt.
Key takeaways
- robots.txt is an IETF standard with per-bot rules, and AI crawler operators publish the bot names it controls.
- llms.txt is a community proposal that Google Search ignores and that no AI engine documents reading as of October 2026.
- Google says an llms.txt will neither harm nor help your visibility in Google Search, so keeping one is harmless but not a lever.
- A robots.txt mistake, such as a blanket AI block, can remove you from ChatGPT search, so it deserves your attention first.
- robots.txt is necessary but not sufficient: firewalls, CDNs and user-triggered fetchers sit outside it.
What each file is
robots.txt
robots.txt is a plain text file at the root of your domain. It tells named crawlers which paths they may fetch. Its format is the Robots Exclusion Protocol, published by the IETF as RFC 9309. Our page on the Robots Exclusion Protocol covers the spec. Each group of rules targets a user agent, so you can allow one bot and block another.
llms.txt
llms.txt is a proposal, published at llmstxt.org, for a Markdown file served at /llms.txt. The idea is to hand language models a curated summary of a site with links to its key pages. It is not an IETF standard. No AI engine has documented that it reads the file. Our guide to llms.txt walks through the proposal itself.
The difference that matters
robots.txt is a control that crawler operators document. llms.txt is a suggestion that nobody has documented consuming. That gap decides where your time goes.
llms.txt vs robots.txt, side by side
| robots.txt | llms.txt | |
|---|---|---|
| Status | IETF standard, RFC 9309 | Community proposal (llmstxt.org) |
| What it does | Allows or blocks named crawlers by path | Offers a Markdown summary of the site for LLMs |
| AI engine documentation | OpenAI, Google, Perplexity, Anthropic and Meta publish their bot names and what each one controls | No engine documents reading it |
| Google Search | Googlebot rules control all Google surfaces, including AI Overviews and AI Mode | Ignored, per Google's AI optimization guide |
| Granularity | Per bot, per path | Whole site |
| Cost of a mistake | High: a blanket AI block can remove you from ChatGPT search | Low: Google says it neither helps nor harms in Search |
Read the last row first. One file can quietly take you out of an AI answer engine. The other, as far as any engine has documented, does nothing either way.
Why robots.txt is where the effort pays
robots.txt rules are per bot, and the bot names map to specific products. OpenAI's crawler documentation separates them. GPTBot is for training. OAI-SearchBot surfaces sites in ChatGPT search. OpenAI states that "Each setting is independent of the others," and that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."
Other operators split their bots the same way. PerplexityBot indexes for Perplexity search. Anthropic separates ClaudeBot (training) from Claude-SearchBot (search quality). Google-Extended governs Gemini training and grounding, and Google says it does not affect inclusion in Google Search.
Say you want to keep your content out of OpenAI model training but stay eligible for ChatGPT search. The rules look like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The common error runs the other way. A site blocks every AI bot in one sweep, the search bots go with the training bots, and the site drops out of ChatGPT search with no warning. Google works differently again: there is no robots.txt token for AI Overviews or AI Mode. Googlebot rules are the control there. Our per-bot guide to robots.txt for AI crawlers lists the tokens and what each one governs.
Keep expectations straight. Allowing a bot buys eligibility, not citations. You remove a block. What the engine then retrieves and cites is outside your control.
What the evidence says about llms.txt
Google's guide to optimizing for generative AI features (July 2026) is direct. It says you don't need "new machine readable files, AI text files, markup, or Markdown" to appear in Google Search, including its generative AI capabilities, because "Google Search ignores them." It adds that doing so "will neither harm nor help" your visibility in Google Search. Our digest of Google's guide covers the rest of it.
That statement covers Google Search only. The same paragraph says it is "completely fine" to maintain an llms.txt for other services that use such files. So the honest position is narrower than "proven useless." As of October 2026, no other engine (OpenAI, Anthropic, Perplexity, Microsoft or Meta) documents parsing it. The 2026 critical survey of 45 GEO studies (arXiv:2607.14035) found no controlled test of it at all.
This matters if someone sells you llms.txt SEO. A file no engine documents reading cannot be shown to move AI answers. If a vendor promises citations from it, ask which engine says it reads the file. Before you generate one, read what to know before using an llms.txt generator.
Where robots.txt stops working
robots.txt is necessary, not sufficient. Three gaps sit outside it.
- Firewalls and CDNs. OpenAI says ChatGPT search eligibility means allowing OAI-SearchBot and confirming your host or CDN allows traffic from its published IP addresses. A bot-protection rule can return a 403 to a crawler that robots.txt allows. WordPress sites hit this too, as covered in whether WordPress security plugins block AI crawlers.
- User-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules." Index-time access and answer-time fetching are different controls.
- Defaults you didn't set. From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages for new domains, new sites on existing accounts and existing free-tier customers. Search crawlers stay allowed by default.
None of these shows up by reading robots.txt alone. If your file allows the right bots and you still never appear, start with the usual blockers behind a correct robots.txt.
Which to use when
- robots.txt: always. Every site needs a deliberate, per-bot robots.txt. Decide separately for training bots (GPTBot, ClaudeBot, Google-Extended) and search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot). Then confirm your firewall or CDN agrees with it.
- llms.txt: optional. Keep one if you already have it or it costs you minutes. Google says it does no harm in Search. Do not expect it to change any AI answer, and do not report it as a win.
- llms.txt: skip it if building it competes with fixing access. A blocked search bot is a documented failure. A missing llms.txt is not.
- AI SEO plugins: many generate llms.txt files and robots.txt rules automatically. Check what a plugin writes to robots.txt before you trust it, using our notes on AI SEO plugins.
The next move is a per-bot read of your current file. The robots.txt Checker for AI Crawlers on 99Visibility shows how your rules treat each AI crawler, so you can fix a blanket block before it costs you a channel.
Frequently asked questions
Does Google use llms.txt for AI Overviews or AI Mode?
No. Google's generative AI optimization guide says Google Search, including its generative AI capabilities, ignores AI text files such as llms.txt. To be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet. Google says there are no additional technical requirements, so robots.txt rules for Googlebot and your snippet controls are what matter there.
Will an llms.txt file hurt my site?
Google says creating one will neither harm nor help your visibility or rankings in Google Search. No other engine documents reading the file, so there is no documented effect in either direction elsewhere. The real cost is opportunity: time spent on llms.txt is time not spent checking that your robots.txt and firewall let the search crawlers in.
Does blocking GPTBot remove me from ChatGPT search?
No. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT search. OpenAI states each setting is independent of the others. You can block GPTBot to opt out of training and still allow OAI-SearchBot to stay eligible for search answers. Blocking OAI-SearchBot is what removes you from ChatGPT search.
Can robots.txt block Google AI Overviews without blocking Search?
No. As of October 2026 there is no robots.txt token for AI Overviews or AI Mode. Google says robots.txt directives for Googlebot are the control, and they apply to Search as a whole. Google-Extended only governs Gemini training and grounding. For page-level limits inside Google Search, the documented tools are noindex, nosnippet, max-snippet and data-nosnippet.
Is it worth paying for an llms.txt service?
The evidence gives no reason to. No engine documents reading llms.txt, and Google says Search ignores it. A generator can produce a valid file in minutes, so paying for one buys a file with no documented effect. If a vendor claims it earns AI citations, ask them to name the engine that says it reads the file and the test that showed a change.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.