Adaptive Moment Estimation, commonly known as Adam, is an optimization algorithm for training machine learning models. It is designed to minimize an objective function, typically a loss function, by iteratively updating model parameters. Adam combines the advantages of two other stochastic gradient descent (SGD) extensions: momentum, which accelerates convergence by accumulating past gradients, and RMSprop, which adapts the learning rate per parameter based on the magnitude of recent gradients. This dual mechanism allows Adam to handle sparse gradients and noisy data effectively, making it a popular choice in Deep learning and Neural network training.
Adam was introduced in a 2014 paper by Diederik P. Kingma and Jimmy Ba, titled "Adam: A Method for Stochastic Optimization." The algorithm has since become a default optimizer for many Machine learning tasks, particularly in training large language models and other complex Transformer (architecture)-based architectures. Its popularity stems from its efficiency and relatively low memory requirements compared to full-matrix adaptation methods.
Algorithm Overview
Adam maintains two moving averages for each parameter: the first moment (mean) and the second moment (uncentered variance) of the gradients. These averages are updated at each iteration, and the parameter update is computed using bias-corrected estimates. The method combines the benefits of AdaGrad, which works well with sparse gradients, and RMSprop, which handles non-stationary objectives. The algorithm uses two hyperparameters, typically denoted as beta1 and beta2, which control the exponential decay rates of the moment estimates. In practice, default values of 0.9 for beta1 and 0.999 for beta2, with a small epsilon for numerical stability, are commonly used.
The update rule for Adam involves computing the biased first and second moment estimates, then correcting their initialization bias before performing the parameter update. The step size for each parameter is scaled by the inverse of the square root of the second moment estimate, allowing parameters with large gradients to take smaller steps and those with small gradients to take larger steps. This per-parameter adaptive learning rate often leads to faster and more stable convergence compared to standard stochastic gradient descent.
Relationship to Stochastic Gradient Descent
Adam is a variant of stochastic-gradient-descent (SGD), a foundational optimization method in machine learning. SGD updates model parameters by moving them in the direction opposite to the gradient of the objective function, computed on a small batch of training data. While effective, SGD can be sensitive to the choice of learning rate and can struggle with sparse or noisy gradients. Adam addresses these issues by maintaining two additional estimates: the first moment (the mean) and the second moment (the uncentered variance) of the gradients. These estimates are used to scale the parameter updates, providing per-parameter adaptive learning rates.
Algorithm Overview
Adam computes adaptive learning rates for each parameter from estimates of the first and second moments of the gradients. At each iteration, it maintains two exponentially decaying moving averages: one for the gradient (first moment) and one for the squared gradient (second moment). These are often referred to as m and v. To correct for a bias toward zero in the initial steps, the algorithm includes bias-correction terms.
A typical update step proceeds as follows: compute the gradient of the loss with respect to parameters, update the biased first and second moment estimates, correct for bias, and then update the parameters using these corrected moments. The algorithm has three main hyperparameters: the learning rate, and two decay rates for the moment estimates (commonly denoted as beta1 and beta2). The paper that introduced Adam recommended default values of 0.9 for beta1, 0.999 for beta2, and a small epsilon to avoid division by zero.
Applications in Deep Learning
Adam has become a standard optimizer in Deep learning due to its robustness and ease of use. It is particularly effective for training Transformer (architecture) models, including those used in large language models and Generative AI systems. Many frameworks and platforms, such as TensorFlow and PyTorch, include Adam implementations, making it a default choice for many practitioners. Its ability to handle sparse gradients and its relatively fast convergence have contributed to its widespread adoption.
Variants and Extensions
Several variants of Adam have been developed to address specific limitations. For example, AdamW decouples weight decay from the optimization steps, which has been shown to improve generalization in some Deep learning tasks. AMSGrad modifies the second moment update to ensure that the learning rate does not increase, which can occur in some settings. These variants retain the core adaptive moment mechanism while adjusting specific behaviors.
Applications and Impact
Adam is widely used in training Transformer (architecture)-based models, including those powering modern Generative AI systems and large language models. Its robustness to hyperparameter choices and ability to handle large-scale problems have made it a default choice in many Deep learning frameworks. The algorithm's efficiency is particularly valuable when training on hardware like NVIDIA GPUs or AWS Trainium chips, where computational resources are substantial but still finite. Research groups at institutions such as Stanford AI Lab, BAIR (Berkeley AI Research), and University of Toronto have contributed to its theoretical understanding and practical refinements.