Web crawler: definition and how it works

99Visibility Research · Updated October 21, 2026 · 4 min read

Short answer

A web crawler is software that automatically requests web pages and stores what it gets back, either to build a search index or to collect training data for an AI model. People also call it a spider or a bot. Crawling is really four separate steps: discovery, fetching, rendering and indexing. Each can fail on its own, and any one failure can keep a page out of AI answers.

Key takeaways

A web crawler fetches pages for one of two jobs: building a search index or gathering training data, and AI companies run separate bots for each.
Crawl, fetch, render and index are distinct steps, so a page can pass robots.txt and still never reach an index.
Most AI crawlers do not execute JavaScript, so content that only appears after scripts run may be invisible to them.
A firewall or CDN can block a crawler that robots.txt allows, and nothing in robots.txt will tell you.
Being crawled makes a page eligible for an answer; it does not get the page retrieved or cited.

What a web crawler is

A web crawler is a program that requests URLs over HTTP, reads the responses, and passes the content to a system that stores it. Search engines use crawlers to build the index they answer queries from. AI companies use crawlers for two different purposes: collecting training data and building the search index their assistants pull from at answer time.

The purpose matters more than the name. OpenAI's crawler documentation says GPTBot collects content that may be used to train its foundation models, while OAI-SearchBot is used to surface websites in ChatGPT search. OpenAI states that "each setting is independent of the others." Anthropic, Perplexity, Google and Meta run similar splits. Some bots also fetch pages live when a user asks a question, which is a third job again.

How it works: four steps that fail separately

Treat crawling as a chain. A break at any link stops the page, and each link breaks for a different reason.

Crawl (discovery and permission). The crawler finds a URL through links or sitemaps, then checks robots.txt for its own user-agent. A Disallow for that bot ends the process here.
Fetch. The crawler requests the page. A web application firewall, CDN or bot-management rule can answer with a 403, a challenge page or a CAPTCHA. OpenAI's help documentation says eligibility for ChatGPT search requires allowing OAI-SearchBot and confirming your host or CDN allows traffic from OpenAI's published IP addresses.
Render. The crawler turns the response into readable text. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files (11.50% and 23.84% of their requests) but do not execute them. Googlebot does render. So a price or product name injected by client-side JavaScript can exist for Google and be missing for ChatGPT.
Index. The system decides whether to store the page and how it may be shown. noindex, nosnippet, max-snippet:0 and data-nosnippet control this. Google says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode.

The next move: test each step on its own. A pass at one step tells you nothing about the others.

Why crawlers matter for AI visibility

Crawler access is the part of AI visibility you actually control. The evidence for it is strong, and it works by removing blocks rather than adding anything.

The most common failure is a blanket AI block. A site owner disallows every AI bot to stop training, and with it removes search eligibility too. OpenAI's documentation says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Blocking GPTBot alone does not do that. The right rule is per bot, per purpose.

The second failure is silent. robots.txt says allow, the firewall says 403, and no dashboard flags it. Answer-time fetchers add a wrinkle: OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

What you can't control: whether an engine retrieves your page for a given question, or cites it once retrieved. Fixing access makes you eligible. It promises nothing beyond that. If you are about to start measuring, read whether to fix crawler access before tracking.

Common misconceptions

"Allow GPTBot so ChatGPT can find you." GPTBot governs training only. ChatGPT search eligibility runs through OAI-SearchBot.
"robots.txt enforces access." It is a request that well-behaved bots honour. It does not stop a firewall from blocking a bot you allowed, and some user-triggered fetchers do not follow it.
"Crawled means indexed means cited." These are three different outcomes. Each one narrows the set.
"If Google sees my content, every AI crawler does." Googlebot renders JavaScript. Most AI crawlers tested do not.
"Crawler hits show AI visibility." A training crawler has no referral surface, so heavy crawling can coexist with zero citations. In Otterly's 2,000-blog experiment, ChatGPT's crawlers kept visiting the test sites while ChatGPT barely cited them.

Frequently asked questions

Is a spider bot the same as a web crawler?

Yes. Spider, spider bot, crawler and bot all describe software that requests pages automatically and stores the results. The useful distinction is not the name but the job: training data collection, search indexing, or live fetching when a user asks a question. AI companies document a separate user-agent for each job, and robots.txt rules apply to each user-agent independently.

Do AI crawlers read JavaScript-rendered content?

Mostly not. Vercel's study found ChatGPT and Claude crawlers download JavaScript files but do not execute them, so they can't read client-side rendered content. Googlebot and AppleBot are the exceptions because they render pages. The fix is to put load-bearing facts, such as prices and product names, in the HTML the server sends before any script runs.

How do I check whether an AI crawler can reach my site?

Check two things separately. First, read your robots.txt rules for each named bot, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot. Second, request a page using each crawler's user-agent and record the status code. A 200 means the fetch worked. A 403, challenge page or CAPTCHA means your firewall or CDN is turning the bot away despite robots.txt.

Does blocking a training crawler remove me from AI search?

Not by itself. OpenAI says GPTBot and OAI-SearchBot settings are independent, so you can block training while staying eligible for ChatGPT search. The risk comes from blanket rules that block every AI user-agent at once, which also removes the search bots. Write rules per bot so each one does only what you intend.

Sources

This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.