What is an AI agent?

An AI agent is a model that acts, not just answers. You give it a goal ("find three dentists open on Saturday near me and compare prices"), and it breaks the goal into steps, runs searches, opens pages, reads them and keeps going until it has a result.

The word "agentic" describes this behavior. Agentic AI means a system that chooses its own next action instead of following one fixed request and response. When the actions are mostly searching and reading the web, the narrower term is agentic search.

Most agents run the same loop:

1. Plan. The model decides what it needs to know or do next.
2. Act. It calls a tool: a web search, a page fetch, a calculator, a form.
3. Observe. It reads the tool's output into its context.
4. Repeat or stop. It either takes another step or writes the final answer.

When an agent opens a web page, the request comes from a user-triggered fetcher, not the vendor's indexing crawler. OpenAI uses ChatGPT-User, Perplexity uses Perplexity-User, and Anthropic uses Claude-User. These are separate from training bots (GPTBot, ClaudeBot) and search indexing bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot).

Retrieval is also discretionary. The model decides per task whether to search at all, and on some developer APIs web search is off until a developer turns it on. See whether AI model APIs search the web by default.

Agents change who reads your site and under what rules.

robots.txt may not stop them

OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules "may not apply". Perplexity says Perplexity-User "generally ignores robots.txt rules." Meta documents that Meta-ExternalFetcher may bypass robots.txt too. A disallow line is not a reliable way to keep agents out. The PerplexityBot vs Perplexity-User breakdown shows the split for one vendor.

The firewall is often the real gate

A CDN or bot-management rule can block an agent that robots.txt allows. As of October 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-displaying pages for new domains and existing free-tier customers, a change that took effect on 15 September 2026. You can lose agent access without editing anything. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.

Agents read raw HTML

Vercel reported that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. Prices or product details that only appear after JavaScript runs may be invisible to an agent reading your page.

Agents can be misled

Research on LLM search agents (arXiv:2606.16821, June 2026) found that manipulated web content can steer what agents endorse, with vulnerability varying widely by model and setup. That study used a mock search API, not the live web. The practical risk for you is defensive: a third-party page could misrepresent your brand to an agent.

• "Blocking GPTBot keeps ChatGPT agents off my site." No. GPTBot governs training only. OpenAI says each bot setting is independent. The OpenAI crawlers and firewall guide covers each one.
• "An agent visit means I'll be cited." A fetch only means the page entered the agent's context for one task. What the model uses or cites is its decision.
• "Agents see what a browser sees." Many don't execute JavaScript. Check your raw HTML.
• "Hidden instructions can make agents favor me." Prompt injection and cloaking are illegitimate, and they put your site and reputation at risk. Don't use them.
• "Unknown bots are probably agents I should allow." Verify first. See whether to block AI crawlers you don't recognize.

• Agentic search: agents that search and read the web to answer a question.
• SEO for AI agents: the full guide to user-triggered fetchers, robots.txt and access checks. Start here for depth.
• AI SEO agents: agents built to do SEO work, and their risks.
• User-triggered fetcher: the user agent an AI tool sends when it opens a page for a person, such as ChatGPT-User or Perplexity-User.

Is an AI agent the same as an AI crawler? No. A crawler visits pages on a schedule to build a training set or a search index. An agent fetches pages in real time because a user asked it to do something. Vendors run them under different user agents, such as OAI-SearchBot for indexing and ChatGPT-User for user actions, and they follow different rules.

Can I block AI agents from my site? Partly. robots.txt is not reliable for user-triggered fetchers, since OpenAI says rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them. A firewall or CDN rule can block them at the network layer. Decide first whether you want to, because blocking agents also stops them reading your pricing and product pages for buyers.

Do AI agents execute JavaScript? Many don't. Vercel reported that ChatGPT and Claude crawlers download JavaScript files but don't run them, so they can't read client-side rendered content. If your key facts only appear after JavaScript runs, serve them in the initial HTML. Googlebot is an exception because it renders pages.

How can a page mislead an AI agent? An agent reads page text into its context and may treat instructions or claims there as trustworthy. Hidden text or injected instructions can push it toward a conclusion. Research published in June 2026 found this works on some models more than others. Treat it as a risk to monitor, especially on third-party pages that mention your brand.