In Machine learning, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. It influences how much newly acquired information overrides old information, metaphorically representing the speed at which a model learns. In adaptive control literature, it is often called gain. Setting a learning rate involves a trade-off: too high a rate causes the model to overshoot minima, while too low a rate slows convergence or risks getting stuck in undesirable local minima. Learning rate scheduling addresses this by varying the rate during training, either through predefined schedules or adaptive methods.
Role in Optimization
The learning rate determines the magnitude of the step taken in the descent direction, which is usually derived from the gradient of the loss function. In Deep learning, models are trained by iteratively adjusting weights to minimize a loss function, and the learning rate controls how aggressively those adjustments are made. A constant high rate can lead to oscillations around a minimum, while a constant low rate can make training impractically slow. Scheduling the rate over time helps achieve faster convergence, prevent oscillations, and avoid poor local minima. The learning rate can also differ per parameter, in which case it becomes a diagonal matrix approximating the inverse of the Hessian matrix, as in Newton's method. This relates to step length in quasi-Newton methods and inexact line search.
Time-Based Schedules
Time-based schedules adjust the learning rate at each iteration based on the previous rate and a decay parameter. The formula is \eta_{n+1} = \eta_0 / (1 + d n), where \eta_0 is the initial rate, d is the decay parameter, and n is the iteration step. This schedule reduces the rate gradually, allowing larger steps early in training and finer adjustments later. It is simple to implement and requires only one hyperparameter, the decay d. However, the rate decreases monotonically, which may not be optimal for all problems.
Step-Based Schedules
Step-based schedules change the learning rate at predefined intervals. The rate at iteration n is \eta_n = \eta_0 d^{\lfloor (1+n)/r \rfloor}, where d is the multiplicative factor (e.g., 0.5 for halving) and r is the drop rate (e.g., every 10 iterations). The floor function ensures the rate stays constant between drops. This schedule is common in practice because it is easy to tune and provides discrete reductions that can help the model settle into a minimum. The choice of d and r depends on the dataset and model architecture.
Exponential Schedules
Exponential schedules use a decreasing exponential function: \eta_n = \eta_0 e^{-d n}, where d is a decay parameter. This provides a smooth, continuous reduction in the learning rate, unlike the discrete drops of step-based schedules. Exponential decay is often used in Neural network training because it balances early exploration with later refinement. The decay parameter d controls how quickly the rate falls; a larger d leads to faster decay, which may cause premature convergence.
Warmup and Cosine Annealing
Modern training, especially for Large language models, often employs warmup, where the learning rate increases from a small value to a peak over the first few thousand steps. This prevents instability in early training when weights are random and gradients can be large. After warmup, the rate may decay using cosine annealing, which follows a cosine curve from the peak to a near-zero value. Cosine annealing has been shown to improve final performance in many Transformer (architecture)-based models. Some schedules combine warmup with cyclical patterns, where the rate oscillates between bounds, allowing the model to escape local minima and explore different regions of the loss landscape.
Momentum and Decay
Momentum is a technique often paired with learning rate schedules. It is analogous to a ball rolling down a hill, where the ball's velocity carries it over small bumps and speeds up when the gradient direction is consistent. Momentum is controlled by a hyperparameter similar to mass; too high a value causes the model to overshoot minima, while too low a value reduces its benefit. Decay, on the other hand, serves to settle learning and avoid oscillations by reducing the step size over time. Both are typically built into deep learning libraries such as Keras, which provide default implementations and tunable parameters.
Adaptive Learning Rates
A key limitation of fixed schedules is their dependence on manually chosen hyperparameters, which vary by problem and model. Adaptive gradient descent algorithms address this by adjusting the learning rate per parameter based on historical gradients. Common methods include Adagrad, Adadelta, RMSprop, and Adam. These algorithms are generally built into deep learning frameworks and have become standard for training Artificial intelligence models. Adam, introduced in 2014, combines momentum with per-parameter scaling and is widely used in Generative AI and other applications. Adaptive methods reduce the need for manual schedule tuning, though many practitioners still use a schedule on top of them for best results.
Practical Considerations
Choosing an initial learning rate is critical. Common techniques include using a default value, such as 0.01 or 0.001, or performing a learning rate range test, where the rate is increased linearly over a few epochs to find a suitable range. The optimal schedule often depends on the model size, dataset, and optimizer. For example, OpenAI and Google DeepMind have reported using warmup and cosine decay for training large models. In practice, schedules are often combined with early stopping, where training halts when validation performance stops improving. The choice of schedule can significantly affect final model quality, training time, and stability, making it a key hyperparameter in any training pipeline.