# Deep Residual Learning (ResNet)

Deep Residual Learning (ResNet) is a deep learning architecture introduced in 2015 that uses residual connections to enable training of very deep neural networks, winning that year's ImageNet challenge.

Deep Residual Learning, commonly known as ResNet, is a [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) architecture introduced in 2015 for image recognition. It won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year, demonstrating that networks with hundreds of layers could be trained effectively. The key innovation is the residual connection, which allows layers to learn residual functions with reference to their inputs, stabilizing training and convergence in very deep [neural networks](https://www.wikiprompt.org/wiki/neural-network).

The residual connection motif, defined as \(x \mapsto f(x) + x\) where \(f\) is an arbitrary neural network module, had been used in earlier work, but ResNet's publication made it widely popular for feedforward networks. This motif now appears in many architectures seemingly unrelated to ResNet, including [transformer](https://www.wikiprompt.org/wiki/transformer) models such as BERT and GPT, as well as systems like AlphaGo Zero, AlphaStar, and AlphaFold.

## Mathematics of Residual Connections

In a multilayer neural network, consider a subnetwork with stacked layers (e.g., 2 or 3). Let \(H(x; \alpha)\) denote the subnetwork, where \(x\) is the input and \(\alpha\) is the set of parameters. If \(H^*\) is the desired optimal output, residual learning adds \(x\) directly to the output, so the optimal learned output becomes \(H^* - x\), interpreted as a "residual" with respect to \(x\). This is implemented via a skip connection that performs an identity mapping from the subnetwork's input to its output.

The function \(F(x; \alpha) = H(x; \alpha) + x\) is often represented by matrix multiplication interlaced with activation functions and normalization operations such as [batch normalization](https://www.wikiprompt.org/wiki/batch-normalization) or [layer normalization](https://www.wikiprompt.org/wiki/layer-normalization). A residual block is one such subnetwork, and a deep residual network is constructed by stacking these blocks.

Long short-term memory (LSTM) networks have a memory mechanism that serves as a residual connection. In an LSTM without a forget gate, an input \(x_t\) is processed by a function \(F\) and added to a memory cell \(c_t\), resulting in \(c_{t+1} = c_t + F(x_t)\). An LSTM with a forget gate essentially functions as a highway network.

To stabilize the variance of layer inputs, it is recommended to replace residual connections \(x + f(x)\) with \(x/L + f(x)\), where \(L\) is the total number of residual layers.

## Projection Connections

When the function \(F\) maps from \(\mathbb{R}^n\) to \(\mathbb{R}^m\) with \(n \neq m\), the expression \(F(x) + x\) is undefined. In this case, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection defined by \(P(x) = Mx\), with \(M\) an \(m \times n\) matrix. The matrix \(M\) is trained via [backpropagation](https://www.wikiprompt.org/wiki/backpropagation), like any other model parameter.

## Signal Propagation

The introduction of identity mappings facilitates signal propagation in both forward and backward paths. In the forward direction, if the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th block (assuming no activation between blocks), then the input to the next block is \(x_{\ell+1} = F(x_\ell) + x_\ell\). This additive structure allows gradients to flow directly through the network during backpropagation, mitigating the vanishing gradient problem that plagued earlier deep architectures.

## Impact and Legacy

ResNet's success at ILSVRC 2015 marked a turning point in [computer vision](https://www.wikiprompt.org/wiki/computer-vision) and [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence). It enabled the training of networks with hundreds of layers, which was previously infeasible due to degradation in training accuracy. The residual connection has since become a standard component in many deep learning models, including those used in [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) and [generative AI](https://www.wikiprompt.org/wiki/generative-ai).

The architecture's influence extends beyond image recognition. Residual connections are integral to modern [large language models](https://www.wikiprompt.org/wiki/large-language-model) and other sequence-based models, where they help maintain stable training over many layers. The principle of learning residuals rather than full transformations has also inspired variations such as [U-Net](https://www.wikiprompt.org/wiki/u-net) and other encoder-decoder designs.

ResNet remains a foundational reference in deep learning, and its design principles are taught in courses and applied in research across [machine learning](https://www.wikiprompt.org/wiki/machine-learning) domains. The residual connection's simplicity and effectiveness have made it one of the most enduring contributions to the field.

---
Source: https://www.wikiprompt.org/wiki/deep-residual-learning
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:28:39.988317+00:00