Backpropagation is the algorithm used to compute how a Neural network's loss changes with respect to each of its weights, by applying the chain rule of calculus backward through the network's layers. It is the mechanism that makes training deep networks with Gradient descent computationally feasible, because it calculates all the gradients needed for an update in roughly the same time as a single forward pass, rather than requiring a separate costly calculation for every individual weight.
History
The mathematical technique behind backpropagation, reverse-mode automatic differentiation, was described in various forms starting in the 1960s, and Paul Werbos applied it explicitly to neural networks in his 1974 Harvard doctoral thesis, though the work went largely unnoticed at the time. Backpropagation entered mainstream AI research after a 1986 Nature paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams, "Learning representations by back-propagating errors," which demonstrated that the method could train multi-layer networks to discover useful internal representations, directly answering the limitations that Marvin Minsky and Seymour Papert had identified in single-layer Perceptrons two decades earlier. The 1986 paper is widely credited with helping end the second AI winter and reviving interest in connectionist approaches to AI throughout the late 1980s and 1990s.
How it works
A network's forward pass computes an output and a Loss function value from the current inputs and weights. Backpropagation then works backward from that loss, using the chain rule to compute, layer by layer, how much each weight contributed to the error, producing a gradient for every parameter in the network in a single backward pass. These gradients are then used by an optimizer, typically a variant of Gradient descent, to nudge each weight in the direction that reduces the loss. The efficiency of this backward computation, formally an instance of reverse-mode automatic differentiation, is what makes it practical to train networks with billions of parameters; a naive numerical approach recalculating each gradient independently would be computationally intractable at that scale.
Adoption and limitations
Despite its 1986 popularization, backpropagation did not immediately dominate: through the 1990s and 2000s, deep networks trained with backpropagation suffered from vanishing and exploding gradients in networks with many layers, which limited practical depth and contributed to a preference for shallower models and alternative methods like support vector machines. Architectural fixes, including the gating mechanisms in LSTM networks, careful weight initialization, and later residual connections and normalization layers, gradually resolved these issues and enabled the very deep networks, including Convolutional neural networks and eventually Transformer (architecture)s, that define modern Deep learning. Backpropagation itself has also drawn scientific criticism, most prominently from Geoffrey Hinton in later years, for its biological implausibility, since the brain has no known mechanism for the precise, symmetric backward error signals the algorithm requires, motivating ongoing research into alternative, more biologically plausible learning rules that so far have not matched its practical performance.
Significance
Backpropagation remains the near-universal training method for Neural networks across domains, from Computer vision to Natural language processing, and every major deep learning framework, including PyTorch and TensorFlow, is built around automatic differentiation engines that implement it at scale.