How does a brand end up in an AI model's training data?
99Visibility Research · Updated October 21, 2026 · 5 min read
Short answer
A brand ends up in an AI model's training data when training crawlers such as GPTBot, ClaudeBot or Google-Extended are allowed to collect pages that mention it, and the vendor then uses that content in a future model. Any effect appears only in a later model generation, and only for content collected before that model's knowledge cutoff. No vendor shows whether your pages were included, and nobody has measured what allowing these crawlers changes.
Key takeaways
- Training crawlers collect content that may be used in future models, and each one is controlled separately in robots.txt.
- Anything collected lands a model generation later and stops at that model's knowledge cutoff.
- No AI vendor exposes whether your content is in its training data, and the effect of allowing training crawlers is unmeasured.
- Training crawlers do not control search: OAI-SearchBot, not GPTBot, governs ChatGPT search eligibility.
- The faster, checkable route into AI answers is live retrieval, which depends on access, rendering and being named on sites engines already cite.
The full answer: training is a slow, invisible pipeline
Your brand reaches a model's training data through a chain of steps you only partly control.
- A training crawler requests your pages. You control whether it may, through robots.txt and your firewall or CDN.
- The vendor stores some of what it collected. You can't see what it kept.
- The vendor trains a future model on a selection of stored content. You can't see that selection either.
- The model ships with a fixed knowledge cutoff. Anything after that date isn't in it.
What the model learned becomes its parametric knowledge: what it says about you with no web search. That knowledge is frozen until the next model replaces it.
So allowing a training crawler is a permission, not a placement. OpenAI's crawler documentation describes GPTBot as crawling content that "may be used in training" its foundation models. It makes no promise about what gets used, and it claims no visibility benefit.
Which crawlers collect training data
Each vendor runs separate crawlers for training and for search. As of October 2026, the main training agents are these:
| Crawler | Vendor | What it governs |
|---|---|---|
| GPTBot | OpenAI | Content that may be used to train foundation models |
| ClaudeBot | Anthropic | Training, separate from Claude-SearchBot and Claude-User |
| Google-Extended | Gemini Apps and Vertex AI training and grounding, not Google Search | |
| Meta-ExternalAgent | Meta | Training, separate from Meta-WebIndexer |
OpenAI states that "each setting is independent of the others". A site can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
A split policy looks like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Rendering matters too. Vercel's crawler study found that ChatGPT and Claude crawlers fetch JavaScript files but don't execute them. If your brand facts appear only after JavaScript runs, these crawlers never see them, whatever robots.txt says.
Why any effect lands a model generation later
A model only knows what was collected before its knowledge cutoff. Content you publish today can't reach a model that has already shipped.
Cutoffs also differ inside one product. OpenAI's model documentation lists a February 16, 2026 cutoff for the GPT-5.6 family and August 31, 2025 for the Instant path (chat-latest) used in ChatGPT. The same app can answer from knowledge roughly five and a half months apart, depending on routing. That's why ChatGPT's knowledge cutoff varies.
The practical result: a rebrand, a price change or a new product won't appear in search-off answers until a newer model ships with a later cutoff. The guide to knowledge cutoffs and old news covers how stale facts persist.
What nobody can measure
Be plain about the limits here, because most advice isn't.
- Membership is hidden. No vendor exposes whether a given page or brand is in a training set.
- The effect is unmeasured. Nothing published tests whether allowing training crawlers improves brand recall, or whether blocking them reduces it. The honest label is unmeasured, not proven useless.
- Retrieval state is mixed. A critical survey of 45 GEO studies (arXiv, July 2026) found no technique with a stable, cross-platform causal effect on discoverability. Any visibility number averages answers given with and without live search.
Your own configuration can also change without your involvement. From September 15, 2026, Cloudflare set Training and Agent crawlers to blocked by default on ad-displaying pages for new domains and existing free-tier customers. A site can stop serving training crawlers without anyone editing robots.txt.
What to do next
Decide your training policy on business grounds, then spend effort where the mechanism is documented.
- Set training and search crawlers separately. Choose GPTBot, ClaudeBot and Google-Extended rules as a content-licensing decision. Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want to appear in AI search.
- Test real reachability. Confirm your CDN or firewall doesn't return 403 to crawlers that robots.txt allows.
- Server-render the facts that matter. Put pricing, product names and a one-line description in the HTML before JavaScript runs.
- Get named on sites engines already cite. The evidence shows a strong association between being mentioned there and appearing in answers, with no promised effect.
- Compare search-on and search-off answers. The search on vs search off guide explains the gap. The free checker on this site shows whether AI engines name you for three buyer questions, as a snapshot.
Frequently asked questions
Does allowing GPTBot help my brand appear in ChatGPT search?
No. OpenAI documents GPTBot as a training crawler only and asserts no search benefit from allowing it. ChatGPT search eligibility is governed by OAI-SearchBot, plus a host or CDN that lets OpenAI's published search IP addresses through. If you want to appear in ChatGPT search answers, check those two things. Your GPTBot setting is a separate decision about whether your content may be used to train future models.
Will blocking training crawlers make AI forget my brand?
Blocking stops future collection, not past learning. A model already trained keeps what it learned until it's replaced. Whether exclusion lowers brand recall in later models is untested in any published study. Blocking GPTBot also doesn't remove you from ChatGPT search, because OpenAI states each crawler setting is independent. There's more detail in [whether blocking training bots affects brand recall](/questions/blocking-training-bots-brand-recall).
Can I check whether my content is in a model's training data?
Not directly. No AI vendor exposes training-set membership for any page or brand. What you can observe is behaviour: ask a model about your brand with web search turned off and compare its answer with your current facts. That shows what the model says from memory, not why. It can't tell you which pages were used or whether allowing a crawler made any difference.
How long until new brand information reaches a model's memory?
At least one model generation, and nobody publishes a schedule. The content has to be collected before the next model's cutoff, selected for training, and shipped in a release. Until then, search-off answers reflect the older model. For current facts, live retrieval is the faster route, since engines with search on can fetch your pages at answer time if crawlers can reach them.
Sources
This page is AI-assisted. It was written from a verified evidence base and checked against our editorial policy. Engine behaviour changes often; facts are as of October 21, 2026.