# Greedy Decoding

Greedy decoding is a simple text generation strategy in autoregressive language models that selects the token with the highest probability at each step, producing deterministic output without considering future alternatives.

Greedy decoding is a fundamental decoding strategy used in autoregressive language models, including those based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. At each generation step, the model computes a probability distribution over the vocabulary for the next token, and greedy decoding selects the token with the highest probability. This process repeats until an end-of-sequence token is generated or a predefined maximum length is reached. Because it always picks the most likely token, greedy decoding is deterministic: given the same input and model weights, it produces the same output every time. It is computationally efficient and easy to implement, making it a common baseline in [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) tasks. However, it often leads to repetitive or suboptimal text because it does not consider the impact of a choice on future tokens; a slightly less probable token at an early step might enable a much more coherent continuation. Greedy decoding is contrasted with stochastic methods like [top-k sampling](https://www.wikiprompt.org/wiki/top-k-sampling) and [top-p sampling](https://www.wikiprompt.org/wiki/top-p-sampling), which introduce randomness, and with [beam search](https://www.wikiprompt.org/wiki/beam-search), which maintains multiple candidate sequences to find a more globally optimal output.

## How Greedy Decoding Works

In an autoregressive model, the probability of a sequence of tokens \(x_1, x_2, \ldots, x_T\) is factorized as the product of conditional probabilities: \(P(x_1, \ldots, x_T) = \prod_{t=1}^T P(x_t | x_1, \ldots, x_{t-1})\). Greedy decoding approximates the most likely sequence by choosing, at each time step \(t\), the token \(x_t\) that maximizes \(P(x_t | x_1, \ldots, x_{t-1})\). This is a local maximization, not a global one. The algorithm is straightforward: start with a prompt or a start token, feed it into the model, obtain the probability distribution for the next token, pick the argmax, append it to the input, and repeat. This process is sometimes called "argmax decoding" or "maximum likelihood decoding" at each step.

## Advantages and Disadvantages

The primary advantage of greedy decoding is its simplicity and speed. It requires no additional parameters or search structures, making it suitable for real-time applications where latency is critical, such as interactive chatbots or code completion. It also produces deterministic outputs, which can be desirable for debugging or reproducibility. However, greedy decoding has significant drawbacks. Because it never backtracks, it can get stuck in loops, generating repetitive phrases (e.g., "I love you love you love you"). It also tends to produce bland or generic text, as it always chooses the most common word, which may not be the most informative or creative. Research has shown that greedy decoding often yields lower-quality outputs compared to beam search or sampling methods, especially for open-ended generation tasks like storytelling or dialogue.

## Comparison with Beam Search

[beam search](https://www.wikiprompt.org/wiki/beam-search) is a more sophisticated decoding strategy that maintains a set of \(k\) partial hypotheses (beams) at each step. At each time step, it expands all beams by considering all possible next tokens, then keeps the \(k\) sequences with the highest cumulative log-probability. This allows the model to explore multiple paths and avoid the local optima that greedy decoding falls into. Beam search generally produces more coherent and higher-scoring sequences than greedy decoding, but it is computationally more expensive, as it requires evaluating \(k\) times more candidates per step. In practice, beam search with a moderate beam size (e.g., 4 or 8) is often used for tasks like machine translation, where the output length is constrained and global coherence matters. Greedy decoding can be seen as beam search with \(k=1\). However, even beam search can suffer from repetition and lack of diversity, which is why sampling-based methods are preferred for creative generation.

## Use Cases and Implementation

Greedy decoding is widely used in production systems where speed is more important than output quality, such as in some [large language model](https://www.wikiprompt.org/wiki/large-language-model) inference pipelines. For example, when a user asks a simple factual question, greedy decoding might be sufficient to provide a correct answer. It is also used as a baseline in research papers to compare against more advanced methods. Implementation is trivial in most deep learning frameworks: after obtaining the logits from the model, apply `argmax` over the vocabulary dimension. Many libraries, such as Hugging Face's Transformers, provide a `do_sample=False` parameter that triggers greedy decoding. Despite its limitations, greedy decoding remains a foundational technique in the field of [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and is often the first method taught to students learning about text generation.

## Limitations and Alternatives

Greedy decoding's main limitation is its inability to recover from early mistakes. For instance, in a sentence like "The cat sat on the...", if the model predicts "mat" with high probability but "floor" with slightly lower probability, greedy decoding will choose "mat". If the subsequent context would have been more natural with "floor", the model cannot go back. This is why alternatives like sampling with temperature, top-k, or nucleus (top-p) sampling are used to introduce randomness and increase diversity. These methods sample from the probability distribution rather than taking the argmax, allowing for less likely tokens to be selected. Another alternative is contrastive search, which balances between the model's confidence and the diversity of the generated text. In practice, the choice of decoding strategy depends on the task: for factual generation, greedy or beam search is preferred; for creative writing, sampling methods are better.

## References

- Greedy decoding is described in standard textbooks on natural language processing, such as "Speech and Language Processing" by Daniel Jurafsky and James H. Martin.
- The limitations of greedy decoding are discussed in the context of neural machine translation in papers by Sutskever et al. (2014) and Bahdanau et al. (2015).
- For a comprehensive comparison of decoding strategies, see the paper "The Curious Case of Neural Text Degeneration" by Holtzman et al. (2019), which highlights the pitfalls of greedy and beam search and proposes nucleus sampling.

---

*Note: This article focuses on the concept of greedy decoding as used in modern neural language models. It is not to be confused with greedy algorithms in general, which are a broader class of optimization techniques.*

---
Source: https://www.wikiprompt.org/wiki/greedy-decoding
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T02:00:31.583116+00:00