# Learning Rate Scheduling

Learning rate scheduling adjusts the step size in optimization algorithms during neural network training, balancing convergence speed and stability. Common schedules include time-based, step-based, exponential decay, and adaptive methods like Adam.

In machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns". In the adaptive control literature, the learning rate is commonly referred to as gain.

In setting a learning rate, there is a trade-off between the rate of convergence and overshooting. While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction. Too high a learning rate will make the learning jump over minima, but too low a learning rate will either take too long to converge or get stuck in an undesirable local minimum.

In order to achieve faster convergence, prevent oscillations and getting stuck in undesirable local minima, the learning rate is often varied during training either in accordance to a learning rate schedule or by using an adaptive learning rate. The learning rate and its adjustments may also differ per parameter, in which case it is a diagonal matrix that can be interpreted as an approximation to the inverse of the Hessian matrix in Newton's method. The learning rate is related to the step length determined by inexact line search in quasi-Newton methods and related optimization algorithms.

## Learning Rate Schedules

A learning rate schedule changes the learning rate during learning and is most often changed between epochs or iterations. This is mainly done with two parameters: decay and momentum. There are many different learning rate schedules, but the most common are time-based, step-based, and exponential.

Decay serves to settle the learning in a nice place and avoid oscillations, a situation that may arise when too high a constant learning rate makes the learning jump back and forth over a minimum. Decay is controlled by a hyperparameter.

Momentum is analogous to a ball rolling down a hill; we want the ball to settle at the lowest point of the hill (corresponding to the lowest error). Momentum both speeds up the learning (increasing the learning rate) when the error cost gradient is heading in the same direction for a long time and also avoids local minima by 'rolling over' small bumps. Momentum is controlled by a hyperparameter analogous to a ball's mass which must be chosen manually - too high and the ball will roll over minima which we wish to find, too low and it will not fulfil its purpose. The formula for factoring in the momentum is more complex than for decay but is most often built in with [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) libraries such as Keras.

### Time-Based Schedules

Time-based learning schedules alter the learning rate depending on the learning rate of the previous time iteration. Factoring in the decay, the mathematical formula for the learning rate is:

η_{n+1} = η₀ / (1 + d n)

where η is the learning rate, η₀ is the original learning rate, d is a decay parameter, and n is the iteration step.

### Step-Based Schedules

Step-based learning schedules change the learning rate according to some predefined steps. The decay application formula is here defined as:

η_n = η₀ d^(⌊(1+n)/r⌋)

where η_n is the learning rate at iteration n, η₀ is the initial learning rate, d is how much the learning rate should change at each drop (0.5 corresponds to a halving), and r corresponds to the drop rate, or how often the rate should be dropped (10 corresponds to a drop every 10 iterations). The floor function (⌊…⌋) here drops the value of its input to 0 for all values smaller than 1.

### Exponential Schedules

Exponential learning schedules are similar to step-based, but instead of steps, a decreasing exponential function is used. The mathematical formula for factoring in the decay is:

η_n = η₀ e^(−d n)

where d is a decay parameter.

## Adaptive Learning Rate

The issue with learning rate schedules is that they all depend on hyperparameters that must be manually chosen for each given learning session and may vary greatly depending on the problem at hand or the model used. To combat this, there are many different types of adaptive gradient descent algorithms such as Adagrad, Adadelta, RMSprop, and Adam, which are generally built into deep learning libraries such as Keras.

## Warmup and Cyclical Schedules

Beyond the classical decay schedules, modern training of large models often employs warmup and cyclical schedules. Warmup starts with a small learning rate and gradually increases it over a few epochs, which helps stabilize training in the early phase, especially for [transformer](https://www.wikiprompt.org/wiki/transformer)-based models. Cyclical schedules, such as cosine annealing, periodically vary the learning rate between a minimum and maximum value, which can help escape local minima and sometimes improve generalization.

## Practical Considerations

The choice of learning rate schedule can significantly affect the training of [neural-network](https://www.wikiprompt.org/wiki/neural-network) models. For instance, in training [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s, a common practice is to use a linear warmup followed by a cosine decay. This approach is used by organizations like [openai](https://www.wikiprompt.org/wiki/openai) and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) in their large-scale training runs. The initial learning rate is often set based on empirical rules or by running a learning rate finder, which tests a range of values over a few iterations.

## Relationship to Optimization Algorithms

Learning rate scheduling is closely tied to the choice of optimization algorithm. [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) frameworks such as [tensorflow](https://www.wikiprompt.org/wiki/tensorflow) and PyTorch provide built-in schedulers that can be combined with optimizers like SGD, Adam, or RMSprop. The interaction between the schedule and the optimizer's internal state (e.g., momentum buffers in Adam) is important; for example, reducing the learning rate too abruptly can cause the optimizer to overshoot.

## History and Development

The concept of adjusting the learning rate during training dates back to early work in [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and [machine-learning](https://www.wikiprompt.org/wiki/machine-learning). Researchers like [thomas-dietterich](https://www.wikiprompt.org/wiki/thomas-dietterich) and [michael-jordan](https://www.wikiprompt.org/wiki/michael-jordan) contributed to foundational understanding of optimization in learning systems. In the 2010s, the rise of deep learning brought renewed attention to scheduling, with papers on cyclical learning rates and warmup becoming widely cited.

## See Also

- [deep-learning](https://www.wikiprompt.org/wiki/deep-learning)
- [neural-network](https://www.wikiprompt.org/wiki/neural-network)
- [machine-learning](https://www.wikiprompt.org/wiki/machine-learning)
- [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)

---
Source: https://www.wikiprompt.org/wiki/learning-rate-scheduling
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:29:29.919879+00:00