In the context of large language models, temperature is a hyperparameter that governs the randomness of generated text. It is applied during the decoding phase, after the model computes a probability distribution over the next possible token but before the final token is selected. By scaling the logits (the raw, unnormalized scores) before they are passed through a softmax function, temperature reshapes the probability distribution, making it either more peaked (deterministic) or more flat (diverse). The parameter is typically a positive floating-point number, with a default value of 1.0 representing the model's native distribution. Values below 1.0 sharpen the distribution, favoring high-probability tokens, while values above 1.0 flatten it, giving lower-probability tokens a greater chance of being selected. Temperature is a core control in generative AI systems, widely used in APIs from providers such as OpenAI, Anthropic, and Google DeepMind to tune creative versus factual outputs.
Temperature is conceptually distinct from other sampling strategies, though it is often combined with them. While top-k sampling restricts the candidate pool to the k most likely tokens and top-p sampling (also called nucleus sampling) selects from the smallest set whose cumulative probability exceeds a threshold, temperature modifies the entire distribution before any truncation occurs. In practice, developers frequently set temperature alongside top-p to balance creativity and coherence. For instance, a low temperature (e.g., 0.2) with a moderate top-p (e.g., 0.9) is common for factual tasks like summarization, whereas a high temperature (e.g., 0.8) with a higher top-p (e.g., 0.95) is used for creative writing or brainstorming.
Mathematical formulation
The effect of temperature is defined through a scaling operation on the logits. Given a model's logits \( z_i \) for each token \( i \) in the vocabulary, the temperature-scaled probability \( p_i \) is computed as:
\[ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} \]
where \( T \) is the temperature value. When \( T = 1 \), the expression reduces to the standard softmax. As \( T \) approaches 0, the distribution converges to a point mass on the token with the highest logit, effectively making the model greedy and deterministic. As \( T \) increases toward infinity, the distribution approaches a uniform distribution over all tokens, yielding maximally random outputs. In practice, temperatures are rarely set above 2.0 because extreme values produce gibberish, and values below 0.1 are often treated as equivalent to greedy decoding. The scaling is applied before any sampling method, so the final selection can still use stochastic sampling from the modified distribution.
Historical origins
The concept of temperature in probabilistic models predates modern deep learning. It was introduced in statistical physics, where temperature controls the sharpness of the Boltzmann distribution. In machine learning, temperature scaling was popularized in the context of neural networks for calibrating confidence scores, notably in the 2017 paper "On Calibration of Modern Neural Networks" by Chuan Guo and colleagues, which used a single temperature parameter to rescale logits for better uncertainty estimates. The idea was quickly adopted in sequence generation, particularly for sequence-to-sequence models and transformers, as a practical knob for controlling output variability. Early adoption in machine learning libraries, such as TensorFlow and PyTorch, made temperature a standard argument in sampling functions, and it became a default parameter in many AI text generation interfaces.
Role in language model inference
During inference, a language model produces a probability distribution over its vocabulary for each position in the output sequence. Without temperature, the model would always choose the most likely token, leading to repetitive and often bland text. Temperature introduces stochasticity, which is essential for tasks that require variety, such as dialogue, story generation, or code completion where multiple valid continuations exist. In deep learning frameworks, temperature is applied at the decoding step, not during training, meaning it does not affect the model's learned weights. This makes it a lightweight, runtime-only adjustment that can be changed per request without retraining.
For example, in the OpenAI API, the temperature parameter accepts values from 0 to 2, with a default of 1.0. A value of 0 makes the model deterministic, always picking the highest-probability token, while higher values increase diversity. Similarly, Anthropic's Claude models expose temperature in their API, and Google DeepMind's Gemini models include it as a generation parameter. These providers also document that temperature interacts with other sampling parameters, such as top-p and top-k, and recommend adjusting them together rather than in isolation.
Relationship to other sampling methods
Temperature is one of several techniques used to shape the output distribution. Top-k sampling limits the next token to the k most probable options, which prevents very low-probability tokens from being chosen even at high temperatures. Top-p sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p, offering a more adaptive truncation than fixed k. Temperature is orthogonal to these truncation methods: it changes the shape of the distribution, while top-k and top-p change the support. In practice, they are often used together. For instance, a common recipe for creative tasks is temperature 0.7 with top-p 0.9, while for code generation, temperature 0.2 with top-p 0.1 is typical to minimize errors.
Another related concept is temperature scaling in the context of model calibration, where a single temperature is learned on a validation set to improve confidence estimates. This is distinct from the sampling temperature used at generation time, though both share the same mathematical operation. The calibration temperature is usually close to 1.0 and is fixed after training, whereas the sampling temperature is a user-controlled hyperparameter.
Practical guidelines and trade-offs
Choosing the right temperature depends on the application. For tasks requiring factual accuracy, such as question answering, summarization, or data extraction, a low temperature (0.1 to 0.3) is recommended to reduce hallucinations and produce consistent outputs. For creative writing, brainstorming, or generating multiple candidate solutions, a higher temperature (0.7 to 1.0) encourages novelty and variation. Extremely high temperatures (above 1.5) often lead to incoherent text, as the model loses grammatical structure and semantic coherence. Developers also need to consider that temperature affects reproducibility: setting temperature to 0 yields deterministic outputs, which is useful for testing and debugging, but stochastic sampling at higher temperatures means the same prompt can produce different results across runs, which may be undesirable in production systems that require stable outputs.
Temperature also interacts with the model's training distribution. Models trained on diverse data may tolerate higher temperatures better than those trained on narrow domains. For instance, a model fine-tuned on legal documents might produce nonsensical text at temperature 1.0, whereas a general-purpose model handles it gracefully. As of the early 2020s, most major language model providers have adopted temperature as a standard API parameter, and it is also implemented in open-source libraries like Hugging Face's Transformers, where it is passed to generation functions such as generate().
Implementation in software
In practice, temperature is implemented in the decoding loop of a language model. After the model forward pass produces logits, the code divides them by the temperature value, applies softmax, and then samples from the resulting distribution. Many frameworks also support a do_sample flag that, when set to True, enables stochastic sampling with temperature; when False, the model uses greedy decoding regardless of temperature. In the Hugging Face Transformers library, for example, the temperature argument is used only when do_sample=True. This design allows developers to switch between deterministic and stochastic modes easily.
Hardware accelerators like AWS Trainium and Groq often provide optimized inference paths, but temperature scaling is a simple arithmetic operation that adds negligible overhead. The main computational cost remains the forward pass of the model, not the sampling step. Consequently, temperature can be adjusted on the fly without significant latency impact, making it a practical tool for interactive applications.
Limitations and criticisms
Temperature is a blunt instrument for controlling output quality. It does not guarantee coherence or factual correctness; it only changes the probability distribution. A high temperature can produce creative but false statements, while a low temperature can produce repetitive or overly conservative text. Researchers have noted that temperature is not a substitute for better decoding strategies like beam search or constrained decoding, which enforce structural constraints. Additionally, temperature is a global parameter applied uniformly across all tokens, but some contexts may require different levels of randomness - for example, being deterministic for code syntax but creative for comments. As of the mid-2020s, more advanced methods such as contrastive search or dynamic temperature adjustment have been proposed, but none have replaced temperature as the default control in commercial APIs.
Future directions
Research continues on adaptive temperature schemes that adjust the value based on the token's predicted uncertainty or the task at hand. Some works have explored learning a temperature schedule during training, though this is not yet standard. In the broader field of generative AI, temperature remains a key user-facing parameter, and its simplicity is both its strength and weakness. As models grow larger and more capable, the need for finer-grained control may lead to new parameters, but temperature is likely to persist as a fundamental concept in language model inference.