An autoregressive model is a generative model that produces a sequence one element at a time, with each new element conditioned on everything generated before it. In natural language processing, this means predicting the next token given the sequence of tokens already produced, then feeding that token back in to predict the one after it. The term originates in statistics, where autoregressive models predict a value in a time series from its own past values; modern large language models apply the same left-to-right factorization to text.
Autoregressive generation is the default decoding strategy behind most modern chat assistants built on large language models, from OpenAI's GPT series to Anthropic's Claude and Meta's Llama. It is not the only way to generate sequences, but as of 2025 it remains the dominant paradigm for text, and variants of it are used for audio, code, and even some image models.
History
The statistical AR model dates to 1920s work on time series, long before computing existed. In language modeling, n-gram models of the 1980s and 1990s were autoregressive in spirit, predicting a word from a fixed window of preceding words using counted probabilities. Neural autoregressive language models emerged with recurrent neural networks and later LSTM networks, which processed a sequence step by step and predicted the next word at each step. The paradigm carried over largely unchanged into the transformer architecture: GPT-2 and its successors are decoder-only transformers trained to predict the next token, an objective sometimes called causal language modeling.
How it works
During training, an autoregressive model is shown a corpus of text and, at every position, asked to predict the following token given all prior tokens, an approach known as self-supervised learning because the labels come from the text itself rather than human annotation. The model's loss function is typically cross-entropy between the predicted next-token distribution and the actual next token. At inference time, sampling strategies such as greedy decoding, top-k, or nucleus sampling determine how the next token is chosen from the model's predicted probability distribution, and the chosen token is appended to the context before the next prediction, a process repeated until an end-of-sequence token or length limit is reached.
Autoregressive versus other approaches
Autoregressive models contrast with non-autoregressive and diffusion-based approaches. Diffusion models, used for most image and video generation, produce an entire output at once and iteratively refine it by removing noise, rather than generating element by element. Some research systems have explored non-autoregressive text generation for speed, and by the mid-2020s a handful of labs experimented with diffusion-based text models, but none had displaced autoregressive decoding as the mainstream approach for language by 2025. State space models such as Mamba are also autoregressive in the sense of predicting sequentially, but replace the transformer's attention mechanism with a different computational structure.
Limitations
Autoregressive generation is inherently sequential: each token depends on the one before it, which limits parallelism at inference time compared to training, where all positions can be computed at once using teacher forcing. This sequential bottleneck motivated techniques such as speculative decoding, which uses a smaller draft model to propose multiple tokens that the larger model verifies in parallel. Autoregressive models can also compound small errors: an unlikely or incorrect token early in a generation becomes part of the context for every subsequent token, a dynamic connected to hallucination and to degraded output quality in very long generations.
Impact
The autoregressive, next-token-prediction objective is central to the story of modern AI: it is simple, requires no labeled data, and scales predictably with data and compute, a relationship studied under scaling laws. Its simplicity is part of why a single training objective could produce models capable of translation, coding, and reasoning without task-specific architectures, a generality later formalized in the idea of the foundation model.