Pretraining is the first and typically most computationally expensive phase of training a modern AI model, in which a model learns broad statistical patterns from a large, general corpus using a generic training objective, before any adaptation to a specific downstream task occurs. For large language models, pretraining almost always means Self-supervised learning: the model is given massive quantities of text and trained to predict a token given its surrounding context, most commonly the next token in a sequence, an objective that requires no human-provided labels since the "answer" is simply the next word in the existing text.
Why pretraining works
The insight behind pretraining is that predicting the next token accurately across a huge and varied corpus forces a model to implicitly learn grammar, facts, reasoning patterns, and even some rudimentary world knowledge, because all of these are useful for reducing prediction error. A model trained this way is called a base or foundation model and, before any further adaptation, behaves largely as a highly capable text completion engine rather than an assistant that follows instructions. The Foundation model terminology, coined by Stanford researchers in 2021, reflects the fact that a single pretrained model can serve as the starting point for many different downstream applications through Fine-tuning or In-context learning.
Scale and cost
Pretraining is dominated by Scaling laws, empirical relationships showing that a model's loss decreases predictably as the amount of training compute, data, and parameters increases together, following the framework established by researchers at OpenAI in 2020 and refined by DeepMind's Chinchilla paper in 2022, which argued that many earlier models were undertrained relative to their size and that data and parameters should scale together. Pretraining runs for the largest frontier models consume on the order of thousands of GPUs or TPUs running for weeks or months, at costs reported to run into the tens or hundreds of millions of dollars, and require training corpora built from sources such as Common Crawl and licensed or proprietary datasets.
From pretraining to a usable assistant
A freshly pretrained base model is not yet suitable as a conversational assistant: it will continue a prompt in whatever style seems statistically likely, rather than answering a question directly or refusing an unsafe request. Turning a base model into a product like ChatGPT or Claude (AI model family) requires additional post-training stages, typically supervised fine-tuning on curated instruction-response pairs followed by a preference-alignment step such as RLHF or Direct Preference Optimization. Pretraining nonetheless supplies the large majority of a model's raw knowledge and capability; post-training mostly shapes how that capability is expressed and controlled.
Trends and open questions
Because pretraining relies on ever-larger quantities of text, researchers have raised concerns about a coming "data wall," the point at which the supply of high-quality, human-generated text is exhausted relative to what scaling laws demand, prompting increased interest in Synthetic data generated by models themselves and in multimodal pretraining that incorporates images, audio, and video alongside text. Pretraining objectives have also diversified beyond simple next-token prediction, with some approaches incorporating denoising, masked-token prediction as used by BERT, or multimodal contrastive objectives similar to CLIP. Despite these variations, next-token prediction over web-scale text remains the dominant pretraining recipe behind the majority of deployed large language models as of the mid-2020s.