# AMSGrad

AMSGrad is an adaptive learning rate optimization algorithm for training neural networks, introduced as a variant of Adam that uses the maximum of past squared gradients to address convergence issues.

AMSGrad is an optimization algorithm used in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) to train [neural networks](https://www.wikiprompt.org/wiki/neural-network). It was proposed in 2018 by Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar in a paper titled 'On the Convergence of Adam and Beyond'. AMSGrad is a variant of the popular [Adam](https://www.wikiprompt.org/wiki/adam-optimizer) optimizer, designed to fix a theoretical convergence problem in Adam by modifying how past gradients are aggregated. The key change is that AMSGrad keeps a running maximum of past squared gradients, rather than an exponential moving average, which ensures that the effective learning rate does not increase over time. This adjustment improves the algorithm's convergence guarantees in certain settings, particularly for convex and non-convex optimization problems. AMSGrad has been widely adopted in research and practice, though its practical benefits over Adam are often modest and problem-dependent.

The algorithm maintains two state variables for each parameter: a first moment estimate (the mean of gradients) and a second moment estimate (the maximum of squared gradients). At each iteration, the first moment is updated as an exponential moving average of the gradient, similar to Adam. The second moment is updated by taking the element-wise maximum of the current squared gradient and the previous second moment estimate. The parameter update then divides the first moment by the square root of the second moment, with a small epsilon term for numerical stability. This design prevents the second moment from decreasing, which in turn prevents the learning rate from increasing, a behavior that can occur in Adam when the gradient magnitude shrinks.

The motivation for AMSGrad arose from a counterexample showing that Adam can fail to converge to the optimal solution in certain simple convex problems. Reddi, Kale, and Kumar demonstrated that the exponential moving average of squared gradients in Adam can cause the effective learning rate to become too large, leading to oscillations and divergence. By using the maximum, AMSGrad ensures a monotonically non-increasing learning rate, which restores convergence guarantees. The paper also provided regret bounds for AMSGrad, showing that it achieves the same order of regret as Adam in stochastic settings.

## Background and the Adam Optimizer

[Adam](https://www.wikiprompt.org/wiki/adam-optimizer) (Adaptive Moment Estimation) was introduced by Diederik Kingma and Jimmy Ba in 2014 and has become one of the most widely used optimizers in deep learning. Adam combines the advantages of two other extensions of stochastic gradient descent: AdaGrad, which adapts learning rates per parameter based on the sum of squared gradients, and RMSProp, which uses an exponential moving average of squared gradients. Adam maintains both a first moment (mean) and a second moment (variance) of gradients, and applies bias correction to account for the initial zero initialization. The algorithm is known for its robustness to hyperparameter choices and its ability to handle sparse gradients and noisy data.

However, in 2018, Reddi, Kale, and Kumar identified a flaw in Adam's convergence proof. They constructed a simple convex optimization problem where Adam fails to converge to the global optimum, even with a constant learning rate. The issue stems from the fact that Adam's second moment estimate can decrease over time, which can cause the effective step size to increase, potentially leading to overshooting. This theoretical counterexample motivated the development of AMSGrad.

## The AMSGrad Algorithm

The AMSGrad algorithm is formally defined as follows. Let \(\theta_t\) denote the parameter vector at iteration \(t\), and \(g_t\) the gradient of the loss function with respect to \(\theta_t\). The algorithm uses hyperparameters \(\alpha\) (learning rate), \(\beta_1\), \(\beta_2\) (exponential decay rates for the first and second moments), and \(\epsilon\) (a small constant for numerical stability). The update rules are:

1. Compute the gradient \(g_t\).
2. Update the first moment estimate: \(m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t\).
3. Update the second moment estimate using the maximum: \(v_t = \max(v_{t-1}, \beta_2 v_{t-1} + (1 - \beta_2) g_t^2)\).
4. Compute the bias-corrected first moment: \(\hat{m}_t = m_t / (1 - \beta_1^t)\).
5. Update parameters: \(\theta_{t+1} = \theta_t - \alpha \hat{m}_t / (\sqrt{v_t} + \epsilon)\).

The key difference from Adam is in step 3, where Adam uses \(v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2\) (an exponential moving average), while AMSGrad takes the element-wise maximum of the previous \(v_{t-1}\) and the current moving average. This ensures that \(v_t\) is non-decreasing, so the effective learning rate \(\alpha / (\sqrt{v_t} + \epsilon)\) is non-increasing.

## Theoretical Properties

AMSGrad was designed to provide stronger convergence guarantees than Adam. The paper proved that AMSGrad achieves a regret bound of \(O(\sqrt{T})\) for convex optimization, which is optimal for online learning. In contrast, Adam was shown to have a regret bound that can be worse in certain cases. For non-convex problems, AMSGrad also provides convergence to a stationary point under standard assumptions. The use of the maximum ensures that the algorithm maintains a monotonically decreasing step size, which is a common requirement in convergence proofs for stochastic optimization.

However, some researchers have noted that the theoretical advantages of AMSGrad do not always translate to better practical performance. In many deep learning tasks, Adam and AMSGrad perform similarly, and sometimes Adam can outperform AMSGrad. The choice between the two often depends on the specific problem and hyperparameter tuning.

## Practical Usage and Impact

AMSGrad has been implemented in major deep learning frameworks, including TensorFlow, PyTorch, and Keras, often as an option within the Adam optimizer (e.g., `amsgrad=True` in PyTorch). It is used in training various models, from [ResNets](https://www.wikiprompt.org/wiki/residual-network) to [transformers](https://www.wikiprompt.org/wiki/transformer), although it is less commonly the default choice compared to Adam or SGD with momentum. In practice, AMSGrad is often tried when Adam exhibits unstable training or when convergence issues are suspected.

Research has shown that AMSGrad can be beneficial in certain scenarios, such as training with sparse gradients or when the loss landscape has sharp minima. However, a 2019 study by Lucas et al. found that AMSGrad does not consistently outperform Adam across a range of tasks, and its advantages are limited. Nevertheless, AMSGrad remains an important contribution to the family of [SGD variants](https://www.wikiprompt.org/wiki/sgd-variants) and has inspired further research into adaptive optimization methods.

## Relationship to Other Optimizers

AMSGrad is part of a broader family of adaptive learning rate methods that includes AdaGrad, RMSProp, and Adam. It is also related to later developments such as AdamW, which decouples weight decay from the adaptive learning rate, and Nadam, which incorporates Nesterov momentum. The idea of using a maximum of past gradients has also been explored in other contexts, such as in the RAdam optimizer, which rectifies the variance of the adaptive learning rate. AMSGrad's focus on ensuring a non-increasing learning rate has influenced the design of more stable optimizers.

## Criticisms and Limitations

Despite its theoretical appeal, AMSGrad has faced criticism. Some researchers argue that the counterexample used to motivate AMSGrad is contrived and does not reflect real-world optimization problems. Others have pointed out that the maximum operation can make the algorithm more sensitive to initial gradients and can lead to overly conservative updates, slowing convergence. Additionally, the memory and computational overhead of maintaining the maximum is negligible, but the practical gains are often marginal.

A notable critique came from a 2019 paper by Chen and Gu, which showed that AMSGrad's convergence guarantee relies on a specific choice of hyperparameters and that in practice, the algorithm can still fail to converge in some non-convex settings. This has led to ongoing research into adaptive optimizers that combine the strengths of Adam and AMSGrad.

## Legacy and Influence

AMSGrad has had a lasting impact on the field of optimization for deep learning. It highlighted the importance of theoretical analysis in understanding optimizer behavior and spurred a wave of research into the convergence properties of adaptive methods. The algorithm is often cited in papers that propose new optimizers, and it remains a standard baseline in optimization research. While it may not be the default choice in most applications, AMSGrad is a valuable tool in the optimizer toolbox, particularly for researchers and practitioners who encounter convergence issues with Adam.

## See Also

- [adam-optimizer](https://www.wikiprompt.org/wiki/adam-optimizer)
- [sgd-variants](https://www.wikiprompt.org/wiki/sgd-variants)
- [learning-rate-schedule](https://www.wikiprompt.org/wiki/learning-rate-schedule)
- [gradient-clipping](https://www.wikiprompt.org/wiki/gradient-clipping)
- [deep-learning](https://www.wikiprompt.org/wiki/deep-learning)

## References

- Reddi, S. J., Kale, S., & Kumar, S. (2018). On the Convergence of Adam and Beyond. International Conference on Learning Representations (ICLR).
- Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR.
- Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR.
- Lucas, J., et al. (2019). On the Convergence of Adam and Beyond: A Closer Look. arXiv preprint.

Note: The references are provided for completeness, but the article does not include external links as per guidelines.

---
Source: https://www.wikiprompt.org/wiki/amsgrad
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T22:26:40.264875+00:00