# AdamW

AdamW is an optimization algorithm for training neural networks, introduced as a variant of Adam with decoupled weight decay. It separates weight decay from the adaptive gradient updates, improving generalization and training stability.

AdamW is an optimization algorithm used in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) to train [neural networks](https://www.wikiprompt.org/wiki/neural-network). It is a variant of the Adam optimizer that decouples weight decay from the adaptive learning rate updates. The algorithm was introduced in 2017 by Ilya Loshchilov and Frank Hutter, and it has become a standard choice for training large models, including [transformers](https://www.wikiprompt.org/wiki/transformer) and [large language models](https://www.wikiprompt.org/wiki/large-language-model). By separating weight decay from the gradient-based parameter updates, AdamW addresses a known issue in the original Adam optimizer, where weight decay was applied in a way that interfered with the adaptive learning rates, leading to suboptimal convergence and generalization.

The primary motivation for AdamW came from the observation that in Adam, the L2 regularization term (often used as weight decay) is divided by the square root of the exponential moving average of squared gradients. This coupling causes the effective weight decay to vary across parameters and over time, which can hinder optimization. AdamW proposes a simple fix: apply weight decay directly to the parameters after the gradient update, independent of the adaptive learning rate scaling. This decoupling has been shown to improve training performance on a variety of tasks, including image classification and language modeling.

## Background: Optimization in Machine Learning

Training a neural network involves minimizing a loss function, typically by using variants of stochastic gradient descent (SGD). In SGD, the model parameters are updated iteratively by moving in the direction of the negative gradient of the loss, computed on a random subset of the training data. The step size, or learning rate, controls how large each update is. Over the years, many improvements have been proposed to accelerate convergence and improve final performance, such as momentum, adaptive learning rates, and weight decay.

Weight decay is a regularization technique that penalizes large parameter values by adding a term proportional to the sum of squared weights to the loss function. In standard SGD, weight decay is equivalent to L2 regularization, but this equivalence breaks down in adaptive methods like Adam. AdamW was designed to restore the intended behavior of weight decay in adaptive optimizers.

## The Adam Optimizer

Adam (Adaptive Moment Estimation) was introduced by Diederik Kingma and Jimmy Ba in 2014. It maintains per-parameter learning rates by keeping an exponentially decaying average of past gradients (first moment) and past squared gradients (second moment). The update rule for Adam is:

\[ \theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \]

where \(\hat{m}_t\) and \(\hat{v}_t\) are bias-corrected estimates of the first and second moments, \(\eta\) is the learning rate, and \(\epsilon\) is a small constant for numerical stability. Adam became popular due to its robustness to hyperparameters and fast convergence, especially for training deep networks.

However, in the original Adam implementation, weight decay was implemented as L2 regularization, which adds a term \(\frac{\lambda}{2} \|\theta\|^2\) to the loss. This term is then included in the gradient, and because Adam normalizes the gradient by the second moment, the effective weight decay becomes \(\lambda / \sqrt{\hat{v}_t}\). This means that parameters with large gradients receive less regularization, which can lead to overfitting and poor generalization.

## The AdamW Algorithm

AdamW modifies the update rule by removing the L2 regularization term from the gradient and instead applying weight decay directly to the parameters after the adaptive update. The update rule becomes:

\[ \theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} - \eta \lambda \theta_t \]

where \(\lambda\) is the weight decay coefficient. This decoupling ensures that weight decay is applied uniformly across all parameters, regardless of their gradient magnitudes. The authors argued that this leads to better generalization and more stable training, especially when using large learning rates.

In their paper, Loshchilov and Hutter demonstrated that AdamW outperforms Adam with L2 regularization on several benchmark tasks, including image classification on CIFAR-10 and language modeling on Penn Treebank. They also showed that AdamW is more robust to the choice of learning rate and weight decay hyperparameters.

## Impact on Deep Learning

AdamW has had a significant impact on the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence). It is now the default optimizer in many popular deep learning frameworks, such as PyTorch and TensorFlow, and is widely used in training [transformers](https://www.wikiprompt.org/wiki/transformer) and [large language models](https://www.wikiprompt.org/wiki/large-language-model). For example, many models developed by organizations like [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind) use AdamW as part of their training pipelines.

The decoupled weight decay in AdamW has been particularly beneficial for training large-scale models, where regularization is crucial to prevent overfitting on massive datasets. It has also been shown to improve convergence speed and final performance compared to other optimizers like SGD with momentum or Adam with L2 regularization.

## Comparison with Other Optimizers

AdamW is often compared with other adaptive optimizers such as AdaGrad, RMSProp, and Adam. While AdaGrad and RMSProp adjust learning rates based on historical gradients, Adam combines both momentum and adaptive learning rates. AdamW improves upon Adam by fixing the weight decay issue, making it more reliable for a wide range of tasks.

Another related optimizer is SGD with momentum, which is simpler but often requires careful tuning of the learning rate schedule. AdamW offers a good balance between ease of use and performance, which is why it has become a go-to choice for many practitioners. However, some studies have shown that SGD with momentum can achieve better generalization on certain tasks if properly tuned, but AdamW remains competitive and more robust to hyperparameter choices.

## Practical Considerations

When using AdamW, there are a few practical considerations. The weight decay coefficient \(\lambda\) is typically set to a small value, such as 0.01 or 0.1, but it may need to be tuned for specific tasks. The learning rate is often set to a value like 1e-4 or 3e-4 for training transformers. Additionally, AdamW often benefits from a learning rate warmup schedule, where the learning rate increases gradually from a small value to the target value over the first few thousand steps.

In practice, AdamW is implemented in most deep learning libraries, so users can simply specify the optimizer and set the weight decay parameter. For example, in PyTorch, one can use `torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)`. This simplicity has contributed to its widespread adoption.

## Extensions and Variants

Several extensions and variants of AdamW have been proposed. For instance, AdamW with decoupled weight decay has been combined with other techniques like Lookahead, which maintains a slower set of parameters for smoother updates. Another variant is AdamW with cosine annealing learning rate schedules, which has been shown to improve performance on image classification tasks.

In the context of large language models, AdamW is often used with mixed precision training and gradient accumulation to handle large batch sizes. Some frameworks also offer fused implementations of AdamW that reduce memory usage and improve computational efficiency, which is important when training models with billions of parameters.

## Conclusion

AdamW has become a fundamental tool in the machine learning toolkit. By decoupling weight decay from the adaptive gradient updates, it addresses a subtle but important flaw in the original Adam optimizer, leading to better generalization and more stable training. Its simplicity and effectiveness have made it the default choice for many deep learning applications, from image classification to natural language processing. As the field continues to evolve, AdamW remains a reliable and widely used optimization method.

## References

- Loshchilov, I., & Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv:1711.05101.
- Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv:1412.6980.

---
Source: https://www.wikiprompt.org/wiki/adamw
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:32:55.905867+00:00