robots.txt: definition and how it works
99Visibility Research · Updated October 21, 2026 · 4 min read
Short answer
robots.txt is a plain-text file at the root of a website that tells named crawlers which paths they may fetch. Each group of rules targets a user agent, such as Googlebot or OAI-SearchBot. Compliant crawlers read it before crawling. It is a request, not a lock. Some user-triggered fetchers may ignore it, and a firewall or CDN can still block a bot the file allows.
Key takeaways
- robots.txt lives at /robots.txt on each host and groups Allow and Disallow rules by user agent.
- Rules are per bot: OpenAI says its crawler settings are independent, so blocking GPTBot (training) does not block OAI-SearchBot (ChatGPT search).
- User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, by their operators' own documentation.
- An allowed bot can still be turned away by a firewall or CDN, so check network reachability as well as the file.
Definition
robots.txt is the file that implements the Robots Exclusion Protocol, standardized as RFC 9309. A site owner places it at the root of a host, for example https://example.com/robots.txt. Crawlers that follow the protocol fetch it first and obey the group of rules that matches their user agent.
It controls crawling, meaning whether a bot may request a URL. It does not enforce anything. A bot that ignores the file can still fetch the page, and a page you want out of search results needs other controls, such as noindex. This entry is a definition. For per-bot rules and examples, read the full guide to robots.txt for AI crawlers.
How it works
The file is a list of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules for paths.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
This example lets OpenAI's search crawler in and keeps its training crawler out. A crawler uses the group that names it and falls back to User-agent: * if none does. A single misplaced Disallow directive can close a whole site to a bot.
The file only works if the crawler can fetch it. Anthropic warns that blocking its IP addresses impedes its ability to read your robots.txt. If the file itself returns an error to a bot, that bot cannot see your rules.
Why it matters for AI visibility
AI platforms run several crawlers, and each has a different job. OpenAI's crawler documentation says "Each setting is independent of the others." As of October 2026:
- GPTBot governs training use only. OAI-SearchBot governs ChatGPT search. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."
- PerplexityBot indexes for Perplexity search. Claude-SearchBot serves Claude's search. Anthropic's ClaudeBot collects training data.
- Googlebot is the control for Google AI Overviews and AI Mode. There is no separate robots.txt token for those surfaces.
- Meta states that allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses."
The common real-world error is a blanket block on every AI bot. It removes training access, which may be the intent, and silently removes search eligibility too. Allowing a bot buys eligibility only. It does not guarantee retrieval, citation or ranking.
Common misconceptions
- "robots.txt is decisive." It is necessary, not sufficient. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a person triggers those fetches. Perplexity says Perplexity-User "generally ignores robots.txt rules."
- "If robots.txt allows a bot, the bot gets in." OpenAI's help center says eligibility needs both: allow OAI-SearchBot, and confirm your host or CDN allows traffic from its published IP addresses. A firewall returning 403 blocks the bot without any change to the file. If you allow the bots and still never appear, see what else can block ChatGPT.
- "Allow GPTBot so ChatGPT can find you." GPTBot is for training. Search access runs through OAI-SearchBot.
- "llms.txt replaces robots.txt." llms.txt is a proposal. Google Search ignores it, and no engine documents reading it.
Frequently asked questions
Where does robots.txt have to be placed?
At the root of each host, at the path /robots.txt. A file in a subfolder is not read. Each subdomain is a separate host, so blog.example.com needs its own file if you want rules there. The file must also return successfully to the bots you care about. If a firewall blocks a crawler from fetching it, that crawler cannot read any of your rules.
Does blocking GPTBot remove me from ChatGPT search?
No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as its search crawler, and says each setting is independent. You can disallow GPTBot to opt out of training while allowing OAI-SearchBot to stay eligible for ChatGPT search answers. Check that OAI-SearchBot is not caught by a wildcard rule meant for other bots.
Can robots.txt keep my content out of Google AI Overviews?
Only through Googlebot, which also governs regular Google Search. There is no robots.txt token specific to AI Overviews or AI Mode. Google says a page must be indexed and eligible for a snippet to appear as a supporting link there. To limit display without leaving Search, Google documents snippet controls such as nosnippet, max-snippet and data-nosnippet.
Why would a bot ignore my robots.txt?
Some fetchers act on a user's request rather than crawling on their own. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. The protocol is also voluntary, so non-compliant bots can disregard it. If you need to stop a fetcher, that has to happen at the server, firewall or CDN.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.