User agent: definition, AI bot names and how to verify them
99Visibility Research · Updated October 21, 2026 · 4 min read
Short answer
A user agent is the name a client sends with every web request to identify itself. Browsers, search crawlers and AI bots all send one in the User-Agent header. robots.txt rules address bots by this name, so the user agent decides which rules a crawler reads. Any client can send any name, so a user agent is a claim, not proof. Verifying a real crawler needs a second signal, such as its published IP ranges.
Key takeaways
- A user agent is a self-reported name sent in the User-Agent header of each request.
- robots.txt groups rules by user-agent token, so each AI bot reads only the rules written for its name or the catch-all group.
- AI vendors run several bots with different jobs, and blocking one name does not block the others.
- User agents are easy to fake, so verification needs IP ranges, reverse DNS or a cryptographic signature.
Definition
A user agent is the identifier a client attaches to an HTTP request. The full value is the user-agent string, which usually includes a product name, a version and sometimes a contact URL. Inside that string sits a short product token, such as GPTBot or OAI-SearchBot. robots.txt rules match on that token, not on the whole string.
A bot user agent is simply the name a crawler or fetcher chooses to announce. Your server logs record it, your firewall can filter on it, and your robots.txt can address it.
How it works
Every request carries a header like User-Agent: ... OAI-SearchBot/.... The server can log it, serve the page, or refuse.
robots.txt, defined in RFC 9309, groups rules under User-agent lines. A compliant crawler looks for the group matching its token and follows those rules. If no group names it, it follows the User-agent: * group. That is why an unlisted AI bot is governed by your catch-all rules.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This file opts out of OpenAI training while staying eligible for ChatGPT search. robots.txt is a request to the crawler. Nothing in the protocol forces a client to obey it.
Why it matters for AI visibility
AI vendors split their work across several user agents, and each one controls something different. As of October 2026, OpenAI documents three:
- GPTBot crawls content that may be used for training.
- OAI-SearchBot surfaces sites in ChatGPT search. OpenAI says sites opted out of it "will not be shown in ChatGPT search answers."
- ChatGPT-User fetches pages when a person asks, and OpenAI says robots.txt rules "may not apply" to it.
OpenAI states that "each setting is independent of the others." Perplexity says its Perplexity-User agent "generally ignores robots.txt rules," which is typical of a user-triggered fetcher.
The user agent also reaches your firewall. OpenAI's help centre says eligibility means allowing OAI-SearchBot in robots.txt and confirming your host or CDN allows its published IP addresses. A robots.txt allow paired with a firewall 403 is a silent failure. The OpenAI crawler and firewall guide covers that setup.
Spoofing and common misconceptions
Any script can send User-Agent: GPTBot. So a log line that says GPTBot is a claim. Real verification uses a second signal. Cloudflare's verified-bot criteria accept a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS. OpenAI publishes its search bot ranges at openai.com/searchbot.json, but also warns against IP-only allowlisting. Combine the user agent with a verified signal instead. The playbook on verifying real AI crawlers walks through each method.
Misconceptions to drop:
- "Allow GPTBot so ChatGPT can find you." GPTBot governs training. ChatGPT search uses OAI-SearchBot.
- "Blocking by user agent stops scrapers." It stops honest bots. Spoofers change the name.
- "A robots.txt allow means the bot gets in." A CDN or firewall can still refuse it.
Before you block AI crawlers you don't recognize, check what each one does.
Frequently asked questions
What is the difference between a user-agent string and a robots.txt user-agent?
The user-agent string is the full header value a client sends, often long and versioned. The robots.txt user-agent line names only the short product token, such as GPTBot. A crawler matches its token against your groups and ignores the rest of the string. Write robots.txt rules with the exact tokens each vendor documents.
Can I verify a bot by its user agent alone?
No. The header is self-reported and trivial to fake. Check a second signal: the vendor's published IP ranges, a reverse DNS lookup that resolves to the vendor, or a Web Bot Auth signature. OpenAI recommends against relying on IP alone, so pair the user agent with a verified-bot signal rather than trusting either one by itself.
Does blocking a bot's user agent in robots.txt remove me from that AI product?
It depends on which bot. Blocking OAI-SearchBot removes you from ChatGPT search answers, per OpenAI. Blocking GPTBot only signals that content should not be used for training. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all. Match each rule to the specific job of that user agent.
Why does my robots.txt allow a bot but it still can't reach my site?
robots.txt and your firewall are separate gates. A CDN or bot-management rule can return a 403 or a challenge page to a crawler that robots.txt allows. Anthropic also warns that IP blocking can stop it reading your robots.txt file. Test the page with each crawler's user agent and check the status code it receives.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.