# Temperature (language model)

Temperature is a sampling parameter in language models that controls output randomness by scaling logits before softmax, with lower values producing deterministic text and higher values increasing diversity. It is often used alongside top-k and top-p sampling.

In the context of [large language models](https://www.wikiprompt.org/wiki/large-language-model), **temperature** is a hyperparameter that governs the randomness of generated text. It is applied during the decoding phase, after the model computes a probability distribution over the next possible token but before the final token is selected. By scaling the logits (the raw, unnormalized scores) before they are passed through a softmax function, temperature reshapes the probability distribution, making it either more peaked (deterministic) or more flat (diverse). The parameter is typically a positive floating-point number, with a default value of 1.0 representing the model's native distribution. Values below 1.0 sharpen the distribution, favoring high-probability tokens, while values above 1.0 flatten it, giving lower-probability tokens a greater chance of being selected. Temperature is a core control in [generative AI](https://www.wikiprompt.org/wiki/generative-ai) systems, widely used in APIs from providers such as [OpenAI](https://www.wikiprompt.org/wiki/openai), [Anthropic](https://www.wikiprompt.org/wiki/anthropic), and [Google DeepMind](https://www.wikiprompt.org/wiki/google-deepmind) to tune creative versus factual outputs.

Temperature is conceptually distinct from other sampling strategies, though it is often combined with them. While [top-k sampling](https://www.wikiprompt.org/wiki/top-k-sampling) restricts the candidate pool to the k most likely tokens and [top-p sampling](https://www.wikiprompt.org/wiki/top-p-sampling) (also called nucleus sampling) selects from the smallest set whose cumulative probability exceeds a threshold, temperature modifies the entire distribution before any truncation occurs. In practice, developers frequently set temperature alongside top-p to balance creativity and coherence. For instance, a low temperature (e.g., 0.2) with a moderate top-p (e.g., 0.9) is common for factual tasks like summarization, whereas a high temperature (e.g., 0.8) with a higher top-p (e.g., 0.95) is used for creative writing or brainstorming.

## Mathematical formulation

The effect of temperature is defined through a scaling operation on the logits. Given a model's logits \( z_i \) for each token \( i \) in the vocabulary, the temperature-scaled probability \( p_i \) is computed as:

\[ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} \]

where \( T \) is the temperature value. When \( T = 1 \), the expression reduces to the standard softmax. As \( T \) approaches 0, the distribution converges to a point mass on the token with the highest logit, effectively making the model greedy and deterministic. As \( T \) increases toward infinity, the distribution approaches a uniform distribution over all tokens, yielding maximally random outputs. In practice, temperatures are rarely set above 2.0 because extreme values produce gibberish, and values below 0.1 are often treated as equivalent to greedy decoding. The scaling is applied before any sampling method, so the final selection can still use stochastic sampling from the modified distribution.

## Historical origins

The concept of temperature in probabilistic models predates modern deep learning. It was introduced in statistical physics, where temperature controls the sharpness of the Boltzmann distribution. In machine learning, temperature scaling was popularized in the context of [neural networks](https://www.wikiprompt.org/wiki/neural-network) for calibrating confidence scores, notably in the 2017 paper "On Calibration of Modern Neural Networks" by Chuan Guo and colleagues, which used a single temperature parameter to rescale logits for better uncertainty estimates. The idea was quickly adopted in sequence generation, particularly for [sequence-to-sequence](https://www.wikiprompt.org/wiki/sequence-to-sequence) models and [transformers](https://www.wikiprompt.org/wiki/transformer), as a practical knob for controlling output variability. Early adoption in [machine learning](https://www.wikiprompt.org/wiki/machine-learning) libraries, such as TensorFlow and PyTorch, made temperature a standard argument in sampling functions, and it became a default parameter in many [AI](https://www.wikiprompt.org/wiki/artificial-intelligence) text generation interfaces.

## Role in language model inference

During inference, a language model produces a probability distribution over its vocabulary for each position in the output sequence. Without temperature, the model would always choose the most likely token, leading to repetitive and often bland text. Temperature introduces stochasticity, which is essential for tasks that require variety, such as dialogue, story generation, or code completion where multiple valid continuations exist. In [deep learning](https://www.wikiprompt.org/wiki/deep-learning) frameworks, temperature is applied at the decoding step, not during training, meaning it does not affect the model's learned weights. This makes it a lightweight, runtime-only adjustment that can be changed per request without retraining.

For example, in the [OpenAI](https://www.wikiprompt.org/wiki/openai) API, the `temperature` parameter accepts values from 0 to 2, with a default of 1.0. A value of 0 makes the model deterministic, always picking the highest-probability token, while higher values increase diversity. Similarly, [Anthropic](https://www.wikiprompt.org/wiki/anthropic)'s Claude models expose temperature in their API, and [Google DeepMind](https://www.wikiprompt.org/wiki/google-deepmind)'s Gemini models include it as a generation parameter. These providers also document that temperature interacts with other sampling parameters, such as top-p and top-k, and recommend adjusting them together rather than in isolation.

## Relationship to other sampling methods

Temperature is one of several techniques used to shape the output distribution. [Top-k sampling](https://www.wikiprompt.org/wiki/top-k-sampling) limits the next token to the k most probable options, which prevents very low-probability tokens from being chosen even at high temperatures. [Top-p sampling](https://www.wikiprompt.org/wiki/top-p-sampling) dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p, offering a more adaptive truncation than fixed k. Temperature is orthogonal to these truncation methods: it changes the shape of the distribution, while top-k and top-p change the support. In practice, they are often used together. For instance, a common recipe for creative tasks is temperature 0.7 with top-p 0.9, while for code generation, temperature 0.2 with top-p 0.1 is typical to minimize errors.

Another related concept is [temperature scaling](https://www.wikiprompt.org/wiki/temperature-scaling) in the context of model calibration, where a single temperature is learned on a validation set to improve confidence estimates. This is distinct from the sampling temperature used at generation time, though both share the same mathematical operation. The calibration temperature is usually close to 1.0 and is fixed after training, whereas the sampling temperature is a user-controlled hyperparameter.

## Practical guidelines and trade-offs

Choosing the right temperature depends on the application. For tasks requiring factual accuracy, such as question answering, summarization, or data extraction, a low temperature (0.1 to 0.3) is recommended to reduce hallucinations and produce consistent outputs. For creative writing, brainstorming, or generating multiple candidate solutions, a higher temperature (0.7 to 1.0) encourages novelty and variation. Extremely high temperatures (above 1.5) often lead to incoherent text, as the model loses grammatical structure and semantic coherence. Developers also need to consider that temperature affects reproducibility: setting temperature to 0 yields deterministic outputs, which is useful for testing and debugging, but stochastic sampling at higher temperatures means the same prompt can produce different results across runs, which may be undesirable in production systems that require stable outputs.

Temperature also interacts with the model's training distribution. Models trained on diverse data may tolerate higher temperatures better than those trained on narrow domains. For instance, a model fine-tuned on legal documents might produce nonsensical text at temperature 1.0, whereas a general-purpose model handles it gracefully. As of the early 2020s, most major language model providers have adopted temperature as a standard API parameter, and it is also implemented in open-source libraries like Hugging Face's Transformers, where it is passed to generation functions such as `generate()`.

## Implementation in software

In practice, temperature is implemented in the decoding loop of a language model. After the model forward pass produces logits, the code divides them by the temperature value, applies softmax, and then samples from the resulting distribution. Many frameworks also support a `do_sample` flag that, when set to `True`, enables stochastic sampling with temperature; when `False`, the model uses greedy decoding regardless of temperature. In the Hugging Face Transformers library, for example, the `temperature` argument is used only when `do_sample=True`. This design allows developers to switch between deterministic and stochastic modes easily.

Hardware accelerators like [AWS Trainium](https://www.wikiprompt.org/wiki/aws-trainium) and [Groq](https://www.wikiprompt.org/wiki/groq) often provide optimized inference paths, but temperature scaling is a simple arithmetic operation that adds negligible overhead. The main computational cost remains the forward pass of the model, not the sampling step. Consequently, temperature can be adjusted on the fly without significant latency impact, making it a practical tool for interactive applications.

## Limitations and criticisms

Temperature is a blunt instrument for controlling output quality. It does not guarantee coherence or factual correctness; it only changes the probability distribution. A high temperature can produce creative but false statements, while a low temperature can produce repetitive or overly conservative text. Researchers have noted that temperature is not a substitute for better decoding strategies like [beam search](https://www.wikiprompt.org/wiki/beam-search) or constrained decoding, which enforce structural constraints. Additionally, temperature is a global parameter applied uniformly across all tokens, but some contexts may require different levels of randomness - for example, being deterministic for code syntax but creative for comments. As of the mid-2020s, more advanced methods such as contrastive search or dynamic temperature adjustment have been proposed, but none have replaced temperature as the default control in commercial APIs.

## Future directions

Research continues on adaptive temperature schemes that adjust the value based on the token's predicted uncertainty or the task at hand. Some works have explored learning a temperature schedule during training, though this is not yet standard. In the broader field of [generative AI](https://www.wikiprompt.org/wiki/generative-ai), temperature remains a key user-facing parameter, and its simplicity is both its strength and weakness. As models grow larger and more capable, the need for finer-grained control may lead to new parameters, but temperature is likely to persist as a fundamental concept in language model inference.

## See also

- [top-k-sampling](https://www.wikiprompt.org/wiki/top-k-sampling)
- [top-p-sampling](https://www.wikiprompt.org/wiki/top-p-sampling)
- [beam-search](https://www.wikiprompt.org/wiki/beam-search)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [transformer](https://www.wikiprompt.org/wiki/transformer)

---
Source: https://www.wikiprompt.org/wiki/temperature
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-14T06:11:27.01016+00:00