Here is the translation of the provided text into English, following all the specified rules:
Learning Rate Schedules
In machine learning, the learning rate is a tuning parameter that determines the step size at each iteration while moving toward a minimum of a loss function. It is a crucial hyperparameter in training neural networks. A learning rate that is too high can cause the model to converge too quickly to a suboptimal solution, while a learning rate that is too low can cause the training process to be very slow. A learning rate schedule is a predefined framework used by optimizers to adjust the learning rate. This is often implemented as a decaying function of the iteration number.
Common Schedules
Time-Based Decay
This schedule decays the learning rate according to the formula:
lr = lr0 / (1 + kt)
where lr0 is the initial learning rate, k is a decay constant, and t is the iteration number. This is a simple and common method.
Step Decay
This schedule reduces the learning rate by a factor (e.g., by half) every few epochs. For example, the learning rate might be reduced by 50% every 10 epochs. This allows the model to take large steps initially and then smaller steps as it gets closer to the optimum.
Exponential Decay
This schedule follows the mathematical form:
lr = lr0 * e^(-kt)
where k is a hyperparameter that controls the decay rate. This provides a smooth, continuous decrease in the learning rate.
Cosine Annealing
This schedule describes the learning rate as a cosine function, which slowly decreases the learning rate from an initial value to a near-zero value. It can be described by the formula:
lr = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(t / T * pi))
where lr_max is the initial learning rate, lr_min is the minimum learning rate, t is the current epoch, and T is the total number of epochs. This schedule has been shown to improve performance and is often used in modern training pipelines.
Warmup
This schedule starts with a very small learning rate and gradually increases it to the initial learning rate over a few epochs. This is often used in conjunction with other schedules to stabilize the training process in the early stages, especially for large models and batches.
Cyclical Learning Rates
This schedule varies the learning rate between a minimum and a maximum bound, often following a triangular or sinusoidal wave pattern. This can help the model escape local minima and find a better global minimum.