Gradient descent is an iterative optimization algorithm used to minimize a Loss function by repeatedly adjusting a model's parameters in the direction opposite to the gradient, the vector of partial derivatives that points toward the steepest increase in loss. It is the fundamental mechanism by which most modern Machine learning and Deep learning systems learn from data, underlying the training of everything from simple linear regressions to models with hundreds of billions of parameters.
History
The method predates computing: French mathematician Augustin-Louis Cauchy described a version of gradient-based minimization in 1847. Its statistical form, stochastic approximation, was formalized by Herbert Robbins and Sutton Monro in 1951. Gradient descent became central to AI once it was paired with Backpropagation in the 1980s as the standard way to compute gradients efficiently through multi-layer Neural networks, and its practical dominance grew alongside the availability of large datasets and GPU (in AI) hardware capable of running the many small update steps the method requires.
Variants
Batch gradient descent computes the gradient over an entire dataset before each update, which is accurate but slow and memory-intensive for large datasets. Stochastic gradient descent (SGD) updates parameters using one example, or more commonly a small "mini-batch," at a time, trading some accuracy per step for far more frequent updates and better scalability. Momentum methods accumulate a running average of past gradients to smooth out noisy updates and accelerate convergence in consistent directions. Adaptive methods, most notably Adam, introduced by Diederik Kingma and Jimmy Ba in 2014, maintain per-parameter learning rates based on estimates of the gradient's first and second moments, and Adam or its variants are the default optimizer for training almost all contemporary Large language models.
Practical considerations
The learning rate, which scales the size of each update step, is the single most consequential hyperparameter in gradient descent: too high and training diverges or oscillates, too low and training is impractically slow or gets stuck in poor regions of the loss landscape. Modern training runs typically use a learning rate schedule that warms up gradually and then decays over the course of training. Because the loss surfaces of deep networks are extremely high-dimensional and non-convex, gradient descent in practice does not find a true global minimum; instead it relies on the empirical observation that many local minima found by gradient descent in overparameterized networks generalize well, a phenomenon still not fully explained theoretically. Gradient descent is also vulnerable to problems such as vanishing and exploding gradients in very deep or recurrent architectures, which motivated innovations like the LSTM's gating mechanisms and, later, the residual connections used throughout Transformer (architecture) architectures.
Relationship to other concepts
Gradient descent is what a network's Loss function is minimized with, but it is not by itself sufficient to produce a useful model: without techniques such as Regularization, gradient descent will happily drive training loss toward zero by memorizing the training set, a failure mode known as Overfitting. In Reinforcement learning, gradient-based methods are adapted to optimize an agent's expected reward rather than a fixed labeled loss, forming the basis of policy-gradient algorithms used in systems such as those trained with RLHF.