In machine learning, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns." In the adaptive control literature, the learning rate is commonly referred to as gain.
In setting a learning rate, there is a trade-off between the rate of convergence and overshooting. While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction. Too high a learning rate will make the learning jump over minima, but too low a learning rate will either take too long to converge or get stuck in an undesirable local minimum.
Cyclical learning rates (CLR) is a training method that addresses this trade-off by periodically varying the learning rate between a lower bound and an upper bound, rather than monotonically decreasing it. This approach, introduced by Leslie N. Smith in 2015, is designed to improve convergence speed and model accuracy without requiring extensive manual tuning of the learning rate schedule.
Motivation and Core Idea
Traditional learning rate schedules, such as time-based, step-based, and exponential decay, reduce the learning rate over epochs to settle the model into a minimum. However, these schedules rely on hyperparameters that must be manually chosen for each problem, and a poorly chosen schedule can lead to slow convergence or poor final performance.
Cyclical learning rates exploit the observation that a moderately high learning rate can act as a regularizer, helping the model escape sharp local minima and find flatter, more generalizable minima. By cycling the learning rate, the model periodically explores regions of the loss landscape with larger steps, then refines with smaller steps. This approach can reduce the need for extensive hyperparameter tuning and often yields better accuracy than a fixed or monotonically decaying rate.
The method is particularly effective in Deep learning contexts, where loss landscapes are high-dimensional and non-convex. The cyclical pattern allows the optimizer to traverse saddle points and narrow valleys more effectively than a constant rate.
Triangular and Variant Schedules
The most basic CLR schedule is the triangular policy, where the learning rate linearly increases from a lower bound to an upper bound, then linearly decreases back to the lower bound, repeating this cycle. The cycle length is defined as the number of iterations for one complete increase-decrease period. A common variant, triangular2, halves the amplitude of the learning rate range after each cycle, gradually reducing the exploration range over time.
Another variant, the exponential triangular policy, applies an exponential decay to the amplitude of each cycle, combining the benefits of cyclical exploration with the stability of a decaying schedule. These variants allow practitioners to balance exploration and exploitation as training progresses.
The choice of bounds is critical. Smith recommended setting the lower bound to a value that allows learning to proceed, and the upper bound to a value that is high enough to cause some oscillation but not so high as to diverge. A practical heuristic is to run a learning rate range test, where the learning rate is increased linearly over a small number of iterations, and the loss is monitored to identify the range where the loss decreases most rapidly.
Learning Rate Range Test
The learning rate range test is a technique used to select appropriate bounds for CLR. In this test, the learning rate is increased linearly from a very small value to a large value over a few epochs, and the training loss is recorded. The practitioner then plots loss versus learning rate and selects the lower bound as the point where loss begins to decrease, and the upper bound as the point where loss stops improving or starts to increase.
This test provides a systematic way to set the cyclical bounds, reducing the guesswork involved in choosing hyperparameters. It is particularly useful in Artificial intelligence workflows where models are trained repeatedly with different architectures or datasets.
Relationship to Adaptive Methods
Cyclical learning rates are distinct from adaptive learning rate methods such as Adagrad, Adadelta, RMSprop, and Adam, which adjust the learning rate per parameter based on gradient history. These adaptive methods are built into many Deep learning libraries, such as Keras, and are often used as the base optimizer in conjunction with CLR. In practice, CLR can be applied on top of adaptive optimizers, cycling the global learning rate while the optimizer continues to adjust per-parameter scales.
Some research suggests that CLR can improve the performance of adaptive methods by preventing them from settling too quickly into sharp minima. The cyclical variation acts as a form of annealing that can help the model escape poor solutions.
Applications in Modern AI
Cyclical learning rates have been widely adopted in training Neural network models, including Large language model architectures. For instance, practitioners at organizations like OpenAI and Google DeepMind have used cyclical or similar warmup-and-decay schedules to train models on massive datasets. The technique is also common in computer vision and natural language processing tasks, where it has been shown to improve accuracy on benchmark datasets.
The method is particularly valuable in transfer learning and fine-tuning scenarios, where a pretrained model is adapted to a new task. A cyclical schedule can help the model adapt without catastrophic forgetting, as the periodic high learning rates allow the model to explore new feature spaces while the low rates preserve previously learned representations.
Practical Considerations
Implementing CLR requires specifying the cycle length and the bounds. The cycle length is often set to a multiple of the number of iterations per epoch, such as 2 to 10 epochs per cycle. Shorter cycles allow more frequent exploration, while longer cycles provide more stable training within each phase.
One common pitfall is setting the upper bound too high, which can cause the loss to diverge. The learning rate range test mitigates this risk. Additionally, CLR may not be beneficial for all problems; for very simple convex optimization tasks, a constant or decaying rate may suffice.
In distributed training environments, such as those using Amazon Web Services or Google Cloud, CLR can be implemented with minimal overhead, as it only requires adjusting the learning rate at each iteration. Many deep learning frameworks, including PyTorch and TensorFlow, provide built-in support for cyclical schedules.
See Also
References
- Smith, Leslie N. (2015). "Cyclical Learning Rates for Training Neural Networks." arXiv:1506.01186.
- Smith, Leslie N. (2017). "Cyclical Learning Rates for Training Neural Networks." IEEE Winter Conference on Applications of Computer Vision.
- Géron, Aurélien (2017). "Gradient Descent." Hands-On Machine Learning with Scikit-Learn and TensorFlow. O'Reilly. pp. 113–124. ISBN 978-1-4919-6229-9.
- Plagianakos, V. P.; Magoulas, G. D.; Vrahatis, M. N. (2001). "Learning Rate Adaptation in Stochastic Gradient Descent." Advances in Convex Analysis and Global Optimization. Kluwer. pp. 433–444. ISBN 0-7923-6942-4.