Training data: definition and its link to AI visibility

Training data (also called the training corpus, or LLM training data) is the collection of text a model is trained on before it ships. What the model absorbs becomes its parametric knowledge: the facts it can state when no search runs.

That knowledge stops at a knowledge cutoff. OpenAI's model docs list a Feb 16, 2026 cutoff for the GPT-5.6 family and Aug 31, 2025 for the Instant path (chat-latest). So "what ChatGPT knows" is not one value. The ChatGPT knowledge cutoff explainer covers why it varies by route.

AI companies run crawlers that fetch public web pages for possible training use. Each one has its own robots.txt token, and as of October 2026 the main ones are:

• GPTBot (OpenAI). OpenAI says it crawls "content that may be used in training our generative AI foundation models."
• ClaudeBot (Anthropic). Training, distinct from Claude-SearchBot and Claude-User.
• Google-Extended (Google). Controls training and grounding for Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search.
• Meta-ExternalAgent (Meta).

These settings are per bot. OpenAI states that "each setting is independent of the others," so a site can disallow GPTBot and still allow OAI-SearchBot for ChatGPT search. Allowing a crawler signals permission. It does not guarantee your pages end up in the corpus, and no vendor exposes corpus membership.

Any effect also lags. Content collected now can only reach answers after a new model generation is trained and released. For the mechanics, see how a brand ends up in training data.

Training data is the only theoretical path into answers given with search turned off. Nothing you publish this quarter reaches those answers until the next model ships.

Here is the honest boundary. No study in the evidence base tests whether being in training data makes a model name a brand more often or describe it more accurately. OpenAI does not claim allowing GPTBot improves visibility. The effect is unmeasured, which is different from zero.

What you can control is access for live retrieval: search-bot permissions, firewall reachability and server-rendered content. Those decide whether engines can fetch you at answer time. Our guide to training data vs live retrieval goes deeper on where to spend effort. To see what ChatGPT says today, the free checker on this site runs up to three buyer questions and scores the answers.

• "Allow GPTBot so ChatGPT can find you." GPTBot governs training only. ChatGPT search uses OAI-SearchBot.
• "Blocking GPTBot removes you from ChatGPT." It does not affect search eligibility. The two settings are independent.
• "Blocking training bots makes AI forget your brand." Nobody has measured this. See whether blocking training crawlers affects brand recall.
• "A block deletes what was already collected." A model already trained keeps what it learned. More in what happens to pages already crawled.
• "Getting into training data fixes wrong facts." There is no evidence it does, and errors can come from any stage. See whether tools can prevent hallucinations.

• Parametric knowledge: what a model knows from training alone.
• Knowledge cutoff: the date training data stops.
• Training data vs live retrieval: the two channels into AI answers, compared.

Should I block GPTBot? It depends on whether you want your content used for model training. Blocking GPTBot tells OpenAI not to use crawled content for training. It does not remove you from ChatGPT search, which OAI-SearchBot controls. No study shows that blocking or allowing GPTBot changes how ChatGPT describes your brand, so decide on licensing and policy grounds, not on promised visibility.

Can I check whether my site is in a model's training data? No. No AI vendor exposes training corpus membership. You can confirm what your robots.txt tells each training crawler, and you can see whether a model mentions your brand with search off. Neither proves your pages were or were not in the corpus, because a model can learn about a brand from other sites.

How long until new content reaches training data? At least one full model generation. Content published after a model's knowledge cutoff cannot appear in that model's retrieval-off answers. It can only show up in a later model trained on newer data, and vendors do not publish when or whether specific pages were included.

Does Google-Extended affect Google Search or AI Overviews? Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It controls training and grounding for Gemini Apps and Vertex AI. AI Overviews and AI Mode follow Googlebot rules and Search eligibility, not Google-Extended.