# Top-p Sampling

Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for text generation that samples from a dynamically sized set of high-probability tokens whose cumulative probability exceeds a threshold p, introduced in 2019 to improve text quality.

Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy used to generate sequences from autoregressive probabilistic models, particularly in natural language generation. It was originally proposed by Ari Holtzman, Yejin Choi, and colleagues in 2019 to address the issue of repetitive and nonsensical text produced by deterministic decoding methods like beam search. The technique has since been applied in other scientific fields, such as protein engineering and geophysics.

In top-p sampling, a probability threshold p is set, and the next item in a sequence is sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceeds p. This method adapts the size of the candidate pool based on the model's certainty, making it more flexible than top-k sampling, which samples from a fixed number of candidates. Due to its effectiveness, top-p sampling is widely used in many [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) applications.

## Technique

At each step of the text generation process, a language model calculates a probability distribution over its entire vocabulary for the next token. While simply picking the token with the highest probability (greedy search) or a limited set of high-probability sequences (beam search) is possible, these deterministic methods often produce text that is dull, repetitive, or nonsensical. Top-p sampling introduces randomness to avoid these issues while maintaining quality.

The core idea is to sample from a smaller, more credible set of tokens at each step, called the nucleus. This nucleus contains the most likely next tokens whose combined, or cumulative probability, just exceeds the threshold p. By sampling only from this dynamically-sized group, the model can adapt to different situations. When the model is confident about the next token (e.g., one token has a very high probability), the nucleus will be small. When the model is uncertain (the probabilities are more evenly distributed), the nucleus will be larger, allowing for more diversity.

The process at each step is as follows:

1. The model calculates the probabilities for all possible next tokens.
2. The tokens are sorted by their probability in descending order.
3. The nucleus is formed by selecting tokens from the top of the list until their cumulative probability exceeds the predefined threshold, p.
4. The probabilities of tokens within this nucleus are then rescaled so that they sum to 1. All tokens outside the nucleus are discarded (given a probability of 0).
5. The final next token is randomly sampled from this new, smaller distribution.

Formally, the nucleus, \(V^{(p)} \subseteq V\), is defined as the smallest set of tokens satisfying:

\[\sum_{x \in V^{(p)}} P(x|x_1, \dots, x_{t-1}) \geq p\]

In this formula, \(P(x|x_1, \dots, x_{t-1})\) represents the probability of a token \(x\) given the preceding tokens \(x_1, \dots, x_{t-1}\).

### Example

Imagine at a certain step, a language model has a vocabulary of five words: `[the, a, cat, dog, eats]` and produces the following probabilities:

- the: 0.5
- a: 0.2
- cat: 0.1
- dog: 0.1
- eats: 0.1

If we set \(p = 0.8\):

1. The tokens are sorted by probability: [the, a, cat, dog, eats].
2. The cumulative probability is calculated:
  - the: 0.5
  - the + a: 0.5 + 0.2 = 0.7
  - the + a + cat: 0.7 + 0.1 = 0.8
3. The nucleus is the smallest set with cumulative probability ≥ 0.8, which is \(V^{(0.8)} = \{\text{the, a, cat}\}\).
4. The probabilities for this set are rescaled to sum to 1:
  - P(the) = 0.5 / 0.8 = 0.625
  - P(a) = 0.2 / 0.8 = 0.25
  - P(cat) = 0.1 / 0.8 = 0.125
5. The next token is then sampled from this new distribution, meaning dog and eats have a 0% chance of being chosen.

### Top-k sampling

Top-k sampling is a similar technique where the pool of candidate tokens is restricted to the \(k\) most likely tokens. The main advantage of top-p is its adaptability. When the model is very certain about the next token (a peaked distribution), the nucleus \(V^{(p)}\) can be very small. When the model is uncertain (a flat distribution), the nucleus can be much larger, allowing for more diversity. In contrast, top-k always samples from a fixed number of tokens, which may be too restrictive or too broad depending on the context.

## Applications

While top-p sampling is most famously used as a decoding strategy for large language models, the technique has also been adapted for use in other scientific domains that involve generating or analyzing sequential data from probabilistic models.

### Natural language generation

In its original domain of natural language generation, top-p sampling is valued for its ability to produce more diverse and coherent text compared to deterministic methods. It has been shown to be beneficial in tasks like automatic question generation, where sample diversity is important for creating effective training data for question answering models.

### Drug and protein design

Top-p sampling is used in computational biology to generate novel molecular and protein sequences from specialized language models. In de novo drug design, chemical language models trained on molecular structures use nucleus sampling to generate focused libraries of new, valid drug candidates. Similarly, protein language models leverage top-p sampling to propose novel protein sequences with desired properties, aiding in protein engineering efforts.

### Geophysics

In geophysics, top-p sampling has been applied to generate sequences of geological events or model subsurface structures. For example, it can be used in seismic inversion or reservoir characterization to sample from probabilistic models that predict subsurface properties, helping to quantify uncertainty in geological interpretations.

## Implementation and Usage

In practice, top-p sampling is often combined with other decoding strategies, such as temperature scaling, to fine-tune the randomness and quality of generated text. Temperature scaling adjusts the sharpness of the probability distribution before applying top-p, allowing for further control over diversity. Many [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) frameworks and libraries provide built-in support for top-p sampling, making it easy to integrate into existing pipelines.

Top-p sampling is a standard feature in the APIs of major AI companies, including [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), as well as in open-source libraries like Hugging Face's Transformers. It is typically specified as a parameter (e.g., `top_p`) in text generation functions, with common values ranging from 0.9 to 0.95 for balanced output.

The choice of p significantly affects the output. A lower p (e.g., 0.5) makes the model more conservative, focusing on high-probability tokens, while a higher p (e.g., 0.99) allows for more diversity but may increase the risk of incoherence. Researchers and practitioners often tune p based on the specific task and desired output characteristics.

## See Also

- [top-k-sampling](https://www.wikiprompt.org/wiki/top-k-sampling)
- temperature-sampling
- [beam-search](https://www.wikiprompt.org/wiki/beam-search)
- greedy-decoding

---
Source: https://www.wikiprompt.org/wiki/top-p
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:33:20.37872+00:00