Speculative decoding is an inference-time optimization technique for large language models that accelerates autoregressive text generation by using a smaller, faster draft model to propose several candidate tokens ahead of the main model, which then verifies or rejects them in a single parallel pass. Because verification can check multiple tokens at once, the technique can produce the same output distribution as standard decoding while requiring fewer sequential passes through the large model, reducing latency without changing what the model would have generated.
The approach was described in research from Google and DeepMind in 2022 and 2023 and adopted across major inference systems by 2024, becoming a standard technique for reducing the cost and latency of serving large models in production.
How it works
In standard autoregressive inference, a model generates one token at a time, and each token requires a full forward pass before the next can be produced, an inherently sequential process. Speculative decoding breaks this bottleneck by having a smaller draft model, which is much cheaper to run, generate a short sequence of candidate tokens. The large target model then evaluates all of these candidates in a single batched forward pass, checking whether it would have assigned similar probability to each proposed token. Tokens that pass this check are accepted; the first token that fails is corrected by the target model's own prediction, and any remaining draft tokens after that point are discarded. Because a full forward pass over several tokens costs little more than a pass over one token, since the computation is often dominated by memory bandwidth rather than arithmetic on modern GPUs, accepted multi-token steps yield a substantial net speedup.
Design variants
Several variants of the core idea have been proposed. Some systems use a distilled or otherwise related small model as the draft model; others use a subset of the target model's own layers, or a lightweight lookup based on n-grams from the recent context, avoiding the need to train and maintain a separate draft model. Medusa and similar approaches attach extra prediction heads to the original model itself to propose multiple future tokens without a separate model at all. The choice of draft strategy affects both the acceptance rate, meaning how often draft tokens match what the target model would have produced, and the additional cost of running the draft step.
Impact
Speculative decoding is notable for being close to a free optimization, in the sense that, implemented correctly, it does not change a model's output distribution or require retraining; it only changes how quickly that output is computed. This distinguishes it from lossy efficiency techniques such as quantization or knowledge distillation, which can affect output quality. By the mid-2020s, speculative decoding was widely deployed by major model providers as part of standard serving infrastructure, particularly for latency-sensitive applications such as chat interfaces and coding assistants like GitHub Copilot and Claude Code, and was frequently combined with other serving optimizations to reduce the cost of running frontier models at scale.