In Machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns". In the adaptive control literature, the learning rate is commonly referred to as gain.
In setting a learning rate, there is a trade-off between the rate of convergence and overshooting. While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction. Too high a learning rate will make the learning jump over minima, but too low a learning rate will either take too long to converge or get stuck in an undesirable local minimum.
In order to achieve faster convergence, prevent oscillations and getting stuck in undesirable local minima, the learning rate is often varied during training either in accordance to a learning rate schedule or by using an adaptive learning rate. The learning rate and its adjustments may also differ per parameter, in which case it is a diagonal matrix that can be interpreted as an approximation to the inverse of the Hessian matrix in Newton's method. The learning rate is related to the step length determined by inexact line search in quasi-Newton methods and related optimization algorithms.
Core Concepts: Decay and Momentum
A learning rate schedule changes the learning rate during learning and is most often changed between epochs or iterations. This is mainly done with two parameters: decay and momentum. Decay serves to settle the learning in a nice place and avoid oscillations, a situation that may arise when too high a constant learning rate makes the learning jump back and forth over a minimum, and is controlled by a hyperparameter.
Momentum is analogous to a ball rolling down a hill; the goal is for the ball to settle at the lowest point of the hill, corresponding to the lowest error. Momentum both speeds up the learning (increasing the effective learning rate) when the error cost gradient is heading in the same direction for a long time and also avoids local minima by 'rolling over' small bumps. Momentum is controlled by a hyperparameter analogous to a ball's mass which must be chosen manually - too high and the ball will roll over minima which we wish to find, too low and it will not fulfil its purpose. The formula for factoring in the momentum is more complex than for decay but is most often built in with deep learning libraries such as Keras.
Time-Based Schedules
Time-based learning schedules alter the learning rate depending on the learning rate of the previous time iteration. Factoring in the decay, the mathematical formula for the learning rate is:
\[ \eta_{n+1} = \frac{\eta_0}{1 + dn} \]
where \(\eta\) is the learning rate, \(\eta_0\) is the original learning rate, \(d\) is a decay parameter and \(n\) is the iteration step. This schedule reduces the learning rate smoothly and monotonically, with the reduction becoming less aggressive as training progresses. It is simple to implement and requires only the initial rate and a decay constant, making it a common choice for early Neural network experiments.
Step-Based Schedules
Step-based learning schedules change the learning rate according to some predefined steps. The decay application formula is here defined as:
\[ \eta_n = \eta_0 d^{\left\lfloor \frac{1+n}{r} \right\rfloor} \]
where \(\eta_n\) is the learning rate at iteration \(n\), \(\eta_0\) is the initial learning rate, \(d\) is how much the learning rate should change at each drop (0.5 corresponds to a halving) and \(r\) corresponds to the drop rate, or how often the rate should be dropped (10 corresponds to a drop every 10 iterations). The floor function (\(\lfloor \dots \rfloor\)) here drops the value of its input to 0 for all values smaller than 1. This approach is widely used in training large language models and transformers, where practitioners often reduce the rate by a factor of 0.1 or 0.5 at predetermined milestones, such as after a certain fraction of total training steps.
Exponential Schedules
Exponential learning schedules are similar to step-based, but instead of steps, a decreasing exponential function is used. The mathematical formula for factoring in the decay is:
\[ \eta_n = \eta_0 e^{-dn} \]
where \(d\) is a decay parameter. This schedule provides a smooth, continuous decay that is faster than time-based decay in the early stages and slower later. It is often preferred when the loss landscape is relatively smooth and a gentle reduction is desired. Variations include exponential decay with warm restarts, where the learning rate is periodically reset to a higher value to escape local minima, a technique explored in recent research on Curriculum Learning and Gradient Clipping.
Cosine and Warmup Schedules
Beyond the classical schedules, modern practice frequently employs cosine annealing, where the learning rate follows a cosine curve from an initial value down to near zero. This schedule is often combined with a linear warmup phase, during which the learning rate increases from a small value to the initial maximum over a few hundred or thousand steps. Warmup is particularly important for training very deep networks and large transformers, as it prevents early instability caused by large updates to randomly initialized weights. Cosine schedules with warmup have become a standard choice in many OpenAI and Google DeepMind research implementations, as they tend to yield better final performance than step-based decay for the same training budget.
Adaptive Learning Rate Methods
The issue with learning rate schedules is that they all depend on hyperparameters that must be manually chosen for each given learning session and may vary greatly depending on the problem at hand or the model used. To combat this, there are many different types of adaptive gradient descent algorithms such as Adagrad, Adadelta, RMSprop, and Adam which are generally built into deep learning libraries such as Keras. These methods adjust the learning rate per parameter based on historical gradient information, effectively removing the need for a manually tuned schedule in many cases. For example, Adam maintains per-parameter learning rates that are scaled by the inverse of the square root of the sum of squared past gradients, providing a form of automatic annealing. While adaptive methods often converge faster, they can sometimes generalize worse than well-tuned schedules on certain tasks, leading to hybrid approaches that use a fixed schedule on top of an adaptive optimizer.
Practical Considerations and Selection
Choosing a learning rate schedule involves balancing several factors: the architecture of the model, the size of the dataset, the optimization algorithm, and the available compute budget. For small models on simple datasets, a constant learning rate or a simple time-based decay may suffice. For large-scale training runs, such as those for Generative AI systems, practitioners often use a combination of warmup, cosine decay, and occasional restarts. The initial learning rate itself is typically chosen via a learning rate range test, where the rate is increased linearly over a small number of iterations and the value that yields the steepest loss decrease is selected. Many frameworks, including AWS and Google Cloud, provide built-in schedulers and automated tuning tools that reduce the manual burden.
Relationship to Other Techniques
Learning rate schedules interact closely with other training techniques. Batch Normalization and Layer Normalization can stabilize the loss landscape, allowing higher learning rates and simpler schedules. Weight Initialization affects the appropriate starting rate, as poorly scaled initial weights may require lower rates to avoid divergence. Gradient Clipping prevents exploding gradients, which is especially relevant when using aggressive schedules or recurrent architectures. In reinforcement learning, schedules are often tied to exploration strategies, and in RLHF for aligning language models, the learning rate is typically annealed carefully to preserve pretrained knowledge while adapting to human feedback.
Historical Context and Research
The concept of varying the learning rate dates back to early work in adaptive-control and stochastic gradient descent. Researchers at institutions like MIT CSAIL and Stanford AI Lab studied the theoretical properties of different decay functions, showing that a decaying rate is necessary for convergence in non-convex settings. The introduction of momentum by Bernard Widrow and others in the 1980s laid the groundwork for modern optimizers. Later, the development of adaptive methods like Adagrad and Adam, pioneered by researchers including David Kaplan and others, shifted the focus from manual schedules to automatic per-parameter adjustments. Today, the choice of schedule remains an active area of research, with studies comparing the generalization properties of various decay shapes across different model families, including residual networks and U-Nets.