Why do AI crawlers visit my site but AI never cites it?

A crawler visit and a citation happen at different stages, run by different systems.

• Access (index time). A bot fetches your page and may store it. This is the only stage your logs show.
• Retrieval (answer time). For a specific question, the engine decides whether to search at all, rewrites the question into its own queries, and pulls a set of candidate pages.
• Generation. The model reads the candidates and picks which ones to cite.

A visit proves only that the first stage worked. As of October 2026, retrieval is discretionary on every major platform. ChatGPT searches at the model's discretion. Claude.ai search is a user toggle. Google AI Overviews often do not trigger. So a page can be crawled, indexed and eligible, and still never be pulled into an answer.

There is also a simpler reason: many visits come from bots that can never cite you. Training crawlers collect text for future models. They have no answer page and no link to send anyone, so their visits produce zero citations by design.

What this means: a log full of AI bot hits is not a visibility signal on its own. The next move is to find out which bots those hits came from.

OpenAI, Anthropic and Perplexity each run separate bots for training, search indexing and user-triggered fetches. OpenAI states that "each setting is independent of the others."

If your logs show mostly GPTBot, you are seeing training traffic, which explains the silence. We cover that exact pattern in why logs show GPTBot but not OAI-SearchBot.

A user-agent string is just a header, and any client can send it. To confirm a visit is real, check the requesting IP against the operator's published list. OpenAI publishes OAI-SearchBot ranges at openai.com/searchbot.json. The server log playbook walks through filtering by user-agent, verifying IPs and reading status codes.

OtterlyAI ran a public test in 2026: two new domains, about 1,000 AI-generated posts each, tracked across seven AI engines for roughly four months. Google deindexed both sites. Bing kept them.

The ChatGPT result is the one that matters here. Otterly reports that ChatGPT's crawlers kept visiting the sites, yet ChatGPT citations were effectively zero, even while the sites ranked in Bing. Meanwhile, of the 3,934 citations Otterly tracked for one site, 2,718 came from Microsoft Copilot.

Two lessons follow:

1. Crawler visits from an engine did not predict citations from that engine.
2. Index presence and citation varied engine by engine. One healthy engine told you nothing about the others.

Keep the limits in view. It was one pair of sites on a low-stakes topic, run by a vendor that sells AI tracking. It shows the gap between crawling and citing can be total. It does not tell you how often that happens.

Sometimes the right bot visits and still leaves with nothing useful.

• JavaScript-only content. Vercel found that ChatGPT and Claude crawlers fetch JavaScript files but do not execute them. If your pricing or product facts appear only after hydration, the bot saw an empty shell. See Vercel's crawler study for the details.
• Firewall or CDN blocks. OpenAI says eligibility requires allowing OAI-SearchBot in robots.txt and making sure your host or CDN accepts traffic from its published IPs. A 403 or a CAPTCHA page counts as a visit in some logs but delivers no content. More on this in what blocks ChatGPT when robots.txt allows AI bots.
• Retrieved but not cited. OpenAI's API notes that the number of sources consulted is often greater than the number of citations. Your page may be read and used without a link. Our guide to cited vs retrieved sources explains the gap.
• Training lag. Anything GPTBot collects can only reach a future model. As of October 2026, OpenAI lists a February 16, 2026 knowledge cutoff for its GPT-5.6 models.

Work through the checks in order. Each one removes a possible block.

1. Sort your logs by bot. Separate training bots from search and user-triggered bots. The free AI bot log check shows how to do this without paid tools.
2. Check status codes for search bots. OAI-SearchBot, Claude-SearchBot and PerplexityBot should get a 200 on your key pages, not a 403 or a challenge page. The AI Crawler Checker requests a page as each AI crawler and shows which ones your firewall turns away.
3. Put key facts in the raw HTML. Prices, product names and one-line definitions should be readable before JavaScript runs. A specific fact on a fetchable page gives an engine a sentence it can quote.
4. Look at who gets cited instead. When engines cite other sites for your buyers' questions, being named on those sites is associated with appearing in the answer. The third-party citation playbook shows how to find them.
5. Test real questions. The free checker on this site shows whether AI engines name you for up to three buyer questions. It is a snapshot, not tracking.

What stays outside your control: whether an engine searches for a given question, which queries it runs, and what it cites. Fixing access makes you eligible. Nobody can promise the citation.