Next-token prediction is a fundamental training objective used in autoregressive language models, particularly large language models (LLMs). In this paradigm, a model is trained to estimate the probability of the next token in a sequence given all preceding tokens, effectively learning the conditional probability distribution P(token_n | token_1, ..., token_{n-1}). This objective allows the model to generate text by iteratively sampling or selecting the most likely next token and appending it to the input sequence.
The approach is central to modern large language models built on the Transformer (architecture) architecture, which have superseded earlier recurrent neural network-based models and purely statistical models such as word n-gram language models. By training on vast datasets, often scraped from the public internet, these models learn to produce coherent and contextually relevant text across a wide range of tasks, including natural language generation, machine translation, and question answering.
Historical Development
The conceptual roots of next-token prediction trace back to the 1950s, when Noam Chomsky developed formal approaches to describing language using formal grammar. However, these early models were rule-based and did not involve probabilistic prediction. In the 1970s, Frederick Jelinek and colleagues at IBM Research introduced statistical approaches to language modeling as part of speech recognition work. Their group used probabilistic n-gram models of word sequences and introduced perplexity as an information-theoretic measure of model uncertainty.
In 1980, the first significant statistical language model was proposed, and during that decade IBM conducted 'Shannon-style' experiments, where human subjects predicted or corrected text to identify potential improvements for language modeling. These statistical methods dominated until the rise of neural approaches in the 2000s.
Statistical Foundations
Pure statistical models, particularly those based on word n-grams, estimate the probability of a word given the previous n-1 words. These models rely on counting occurrences in training corpora and suffer from the curse of dimensionality, as the number of possible sequences grows exponentially with vocabulary size, leading to data sparsity.
Exponential language models, such as maximum entropy models, address some limitations by encoding relationships between words and n-gram histories using feature functions. The probability of a word given its history is computed as P(w_m | w_1, ..., w_{m-1}) = 1/Z(w_1, ..., w_{m-1}) * exp(a^T f(w_1, ..., w_m)), where Z is a partition function, a is a parameter vector, and f is a feature function. These models often require regularization or a prior on parameters. The log-bilinear model is another example of an exponential language model.
Skip-gram models, introduced later, focus on predicting surrounding words given a central word, which differs from strict next-token prediction but shares the idea of learning word distributions from context.
Neural Language Models
Neural language models became increasingly important in the 2000s. Yoshua Bengio and co-authors introduced a probabilistic neural network language model that learned distributed word representations while estimating the probability of word sequences. These continuous space embeddings helped alleviate the curse of dimensionality by representing words as non-linear combinations of weights in a neural network, avoiding the data sparsity problem inherent in n-gram models.
Recurrent neural networks (RNNs) extended this approach by retaining information from earlier words as the sequence was processed. In 2010, Tomáš Mikolov and colleagues demonstrated that recurrent neural-network language models could substantially outperform conventional n-gram models. RNN-based models process sequences sequentially, maintaining a hidden state that captures context, and predict the next token based on this state and the current input.
The Transformer Revolution
The introduction of the Transformer (architecture) architecture marked a significant shift in next-token prediction. Transformers, which rely on self-attention mechanisms, allow models to weigh the importance of all previous tokens in a sequence simultaneously, rather than processing sequentially. This parallelization enabled training on much larger datasets and led to the development of large language models.
As of 2026, large language models are predominantly based on transformers trained on larger datasets, frequently using texts scraped from the public internet. These models have superseded recurrent neural network-based models, which had previously superseded purely statistical models. Companies such as OpenAI, Anthropic, and Google DeepMind have developed prominent LLMs that use next-token prediction as their core training objective.
Training and Generation Process
During training, a model is presented with sequences of tokens and learns to predict each token given the preceding ones. The loss function, typically cross-entropy, measures the difference between predicted probabilities and actual next tokens. Through backpropagation and optimization techniques, the model adjusts its parameters to minimize this loss across the training corpus.
At inference time, text generation proceeds autoregressively: the model receives an initial prompt, predicts the next token, appends it to the prompt, and repeats the process. This generates text token by token, with each new token conditioned on all previously generated tokens. The choice of token can be deterministic (selecting the highest probability) or stochastic (sampling from the probability distribution), with techniques like temperature scaling and top-k sampling affecting output diversity.
Applications and Impact
Next-token prediction enables a wide variety of natural language tasks beyond text generation. These include speech recognition, machine translation, optical character recognition, handwriting recognition, grammar induction, information retrieval, and disaster response. The ability to predict sequences in natural language makes language models useful for route optimization and other sequence-based tasks.
Large language models trained with next-token prediction have demonstrated capabilities in tasks such as summarization, code generation, and conversational AI. However, it remains unclear whether these models are plausible cognitive models of human language processing. At least for recurrent neural networks, research has shown that they sometimes learn patterns that humans do not, but fail to learn patterns that humans typically do.
Evaluation and Benchmarks
Evaluation of language models is mostly done by comparison to human-created sample benchmarks derived from typical language-oriented tasks. Other, less established quality tests examine the intrinsic character of a model or compare two models. Since language models are intended to be dynamic and learn from data, some proposed models investigate learning rates through inspection of learning curves.
Numerous datasets have been developed for evaluating language processing systems. These include Massive Multitask Language Understanding (MMLU), Corpus of Linguistic Acceptability, GLUE benchmark, Microsoft Research Paraphrase Corpus, Multi-Genre Natural Language Inference, Question Natural Language Inference, Quora Question Pairs, Recognizing Textual Entailment, Semantic Textual Similarity Benchmark, SQuAD question answering Test, Stanford Sentiment Treebank, Winograd NLI, BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, NaturalQuestions, TriviaQA, RACE, BIG-bench hard, GSM8k, RealToxicityPrompts, WinoGender, and CrowS-Pairs. These benchmarks assess various aspects of language understanding and generation, providing standardized measures for comparing model performance.
Future Directions
As large language models continue to evolve, next-token prediction remains a foundational objective, but researchers are exploring variations and extensions. These include objectives that incorporate reinforcement learning from human feedback, as well as architectures that improve efficiency and context handling. The ongoing development of specialized hardware, such as AWS Trainium and Cerebras systems, aims to support the computational demands of training and deploying these models at scale.