# Warmup Schedule A warmup schedule is an initial training phase where the learning rate ramps up from a low value to a target, stabilizing early optimization and preventing divergence in deep neural networks. A warmup schedule is a learning rate scheduling technique used in deep learning to gradually increase the learning rate from a small initial value to a target value over a limited number of training steps. It is applied at the beginning of model training, before the main decay phase of the overall learning-rate schedule. The primary purpose of a warmup is to stabilize the optimization process during the early epochs, when model parameters are far from optimal and gradients can be large or noisy. By starting with a low learning rate, the warmup phase prevents large parameter updates that might destabilize training or cause the loss to diverge, and it allows the model to move into a region of the loss landscape where larger steps are safe. Warmup schedules are particularly important for training large language models, transformers, and other deep architectures that rely on batch normalization, layer normalization, or adaptive optimizers like Adam. The warmup phase is typically combined with subsequent decay schedules, such as linear decay, cosine decay, or exponential decay, which reduce the learning rate to a near-zero value over the rest of training. A typical schedule might include a linear warmup over the first several hundred or thousand steps followed by a cosine decay to the end of training. The choice of warmup duration and peak learning rate depends on the model size, dataset, batch size, and optimizer used, and it is often chosen empirically via hyperparameter tuning. Warmup is now a standard component in the training pipelines for many prominent models, including recurrent neural networks, convolutional neural networks, and generative models like large language models. ## History and Origins The concept of learning-rate warmup emerged in the context of deep learning research in the mid-2010s. One of the earliest documented uses is attributed to researchers at [Google](https://www.wikiprompt.org/wiki/google) (formerly known as Google research) who trained image classification models and identified that starting with a high learning rate could cause severe divergence. In a 2013 paper on deep network training, [Bernard Widrow](https://www.wikiprompt.org/wiki/bernard-widrow) and others studied adaptive learning rates, but the specific term "warmup" gained traction around 2018 with the publication of the [Adam](https://www.wikiprompt.org/wiki/adam-optimizer)-related training tricks in the original BERT paper. The BERT paper, released by Google in 2018, explicitly used a 10,000-step warmup for pretraining. This practice was adopted widely across the NLP community. Adaptive optimizers like [Adam](https://www.wikiprompt.org/wiki/adam-optimizer) and its variants, including [SGD](https://www.wikiprompt.org/wiki/sgd-variants) with momentum, are particularly prone to instability when large learning rates are applied early. The warmup helps mitigate a phenomenon where the first few steps produce enormous gradients, inflating the adaptive learning rate estimates, leading to poor convergence. Techniques like gradient clipping, batch normalization, and [gradient clipping](https://www.wikiprompt.org/wiki/gradient-clipping) also address similar instability, but warmup is often simpler and more general. ## Motivation and Theory The primary motivation for a warmup schedule lies in the trade-off between exploration and stability. At initialization, the model's weights are usually drawn from a random distribution with zero mean, which can produce large activations and gradients. Assigning a large learning rate at this point can cause the weights to jump to a region where the loss is enormous or the model only captures noise. A low learning rate allows the model to slowly correct initial errors, gradually moving towards more stable regions. Additionally, adaptive optimizers like Adam maintain second-moment estimates of aggregated gradients. If a large gradient spike occurs within the first few steps, the optimizer’s running average becomes inflated, reducing the effective learning rate for future steps and slowing convergence. Warmup gives the optimizer time to obtain reliable statistics before learning at full capacity. Theorie recent work has shown a connection between warmup and the transition from mode covering to mode seeking behavior in Bayesian learning, but the most practical explanation centers on gradient noise. Early in training, stochasticity estimates are noisy due to small effective sample sizes and poorly scaled network outputs. A linear warmup is often enough to smooth the process. ## Types of Warmup Schedules There are several common implementations of warmup: - **Linear warmup**: The learning rate increases linearly from nearly zero to the target value over a specified number of steps (e.g., 3,000 steps). Used in BERT-style training and many transformer implementations. - **Exponential warmup**: The learning rate increases exponentially from a low value. Rarely used, often replaced by linear or polynomial. - **Polynomial warmup**: Learning rate scales as a polynomial of the step index, often with a small degree like 2 or 3. - **Constant warmup**: Learning rate held at a low constant value for a fixed period, then ramped to target. - **One-cycle policy** and **super-convergence**: Sometimes uses a short warmup combined with a cyclic schedule. In practice, linear warmup is the most common inside transformer libraries such as GPT models and is often referred to as "warmup steps" in configuration files. For example, a typical training run might use 2% of total training steps for warming from 0.1 to the maximum learning rate. ## Warmup in Large Language Models Large language models such as GPT-3, Llama 2, and [BLOOM](https://www.wikiprompt.org/wiki/bloom) use a warmup phase for stability. For instance, the GPT-3 paper reported using a warmup of 250,000 steps (or about 3% of total steps) with a cosine decay to 10% of the initial rate. Other models like [LLaMA](https://www.wikiprompt.org/wiki/llama) use a cosine schedule with warmup to 1,000 steps. The fine-tuning or pretraining of such models, which often have billions of parameters, requires careful tuning of the learning rate and warmup duration, and this is typically logged in research papers. In practice, warmup contributes to the stability of training loops and reduces the chance of loss divergence, which can waste guarded compute. In some training frameworks, the warmup schedule is parsed from a learning-rate scheduling configuration that allows for separate values for warmup steps, warmup initial factor, and decay. ## Warmup in Computer Vision Warmup is used across computer vision tasks as well. The classic [ResNet](https://www.wikiprompt.org/wiki/residual-network) training in the 2015 paper used a momentum optimizer with a step decay, but later work like [ResNet](https://www.wikiprompt.org/wiki/resnet) on ImageNet received a linear warmup from an initial learning rate of 0.1 to, but that was often applied for 5 epochs. In training of vision transformers, warmup is standard, with ViT used a warmup of 500 steps (about 0.5 epochs) on ImageNet. Papers that used [batch normalization](https://www.wikiprompt.org/wiki/batch-normalization) in neural networks sometimes relied on warmup to compensate for the instability of the batch normalization statistics on the first few batches. ## Practical Considerations and Hyperparameters Warmup duration is a hyperparameter, selected based on the model size, total training steps, batch size, and optimizer. Many implementations default to linear warmup for about 10 to 20% of the total training steps. For long runs, such as training a transformer on billions of tokens, the warmup might be in the thousands of steps. Overfrequency: too long a warmup wastes compute, and too short can lead to divergence. The target learning rate after warmup is usually the maximum learning rate. This value is often chosen between 1e-4 to 1e-3 for typical transformers with the Adam optimizer. Some papers use a maximum learning rate found by a learning-rate finder or hyperparameter sweeps. Another practice is to keep the learning rate at a constant level after warmup, then use exponential decay. Warmup is often combined with [learning rate scheduling](https://www.wikiprompt.org/wiki/learning-rate-schedule) because the warmth is a modification of the learning-rate schedule. Libraries like [PyTorch](https://www.wikiprompt.org/wiki/pytorch) (with the learning-rate scheduler) and [TensorFlow](https://www.wikiprompt.org/wiki/tensorflow) support built-in functions for warmup and decay; sample code is frequently used in training scripts. ## Empirical Results and Studies Multiple research studies have validated the benefit of warmup. In a 2017 study on longer training epochs, a warmup of fewer than 100 steps resulted in a 5% relative improvement in test accuracy for image classification. For large-scale models, a common baseline for GPT-like transformer training from scratch is a 1,000 step ramp-up with a 1/3 ratio of varying warmup. Known third-party preprint ‘Grokking’ analysis in deep learning found that warmup is crucial for the architecture to escape early attention issues. Formal analyses of the loss landscape during warmup also exist, showing that the loss curvature changes significantly, and warmup can help avoid regions of sharp minima. Some view it as a mechanism to balance weight in the basin of attraction. ## Alternatives and Related Techniques Warmup is not the only method to stabilize early training. Alternative approaches include slower target learning rates from the beginning, using learning rate schedules like step-decay or used-based amounts, or using warm restart techniques such as [cosine annealing](https://www.wikiprompt.org/wiki/cosine-annealing) with restarts (which itself includes a form of warmup). Gradient clipping ([gradient clipping](https://www.wikiprompt.org/wiki/gradient-clipping)) is often used in conjunction with warmup to mitigate spikes. Also for optimizers, the [Adam optimizer](https://www.wikiprompt.org/wiki/adam-optimizer) with betas set to 0.9 and 0.999 may require adjustment during warmup. Some work on schedule-free optimizers (e.g., the 2024 paper by Defazio and Mishchenko) introduced a schedule-free method without warmup, but this remains less common. ## Future Directions and Research Research from 2023-2025 explored multiple warmup phases or adaptive warmup based on gradient statistics, but standard practice remains linear warmup. In more recent transformer scaling plays, there is staying interest in warm-up duration as a hyperparameter in cloud-based training. A 2024 paper from the [OpenAI](https://www.wikiprompt.org/wiki/openai) research group stated that warmup as a fraction of steps has passed through in large models. There are also memory-efficient optimizers that tie warmup to the Adam first momentum but none have replaced it. Warmup remains a basic but crucial component that affects convergence rate and final performance in deep learning across domains of natural language processing, computer vision, and reinforcement learning. Its simple implementation and proven effectiveness ensure it will remain standard in deep learning pipelines for the foreseeable future. --- Source: https://www.wikiprompt.org/wiki/warmup-schedule License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-09T01:59:20.038762+00:00