# Residual Network (ResNet)

A residual neural network (ResNet) is a deep learning architecture where layers learn residual functions via skip connections, introduced in 2015 for image recognition and widely adopted in modern neural networks.

A residual neural network, also known as a residual network or ResNet, is a deep learning architecture in which layers learn residual functions with reference to their inputs. Developed in 2015 for image recognition, it won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year. The key innovation is the residual connection, an architectural motif that adds the input of a subnetwork to its output, enabling the training of networks with hundreds of layers.

The residual connection stabilizes training and convergence in deep neural networks, making it a common motif in modern architectures such as [transformer](https://www.wikiprompt.org/wiki/transformer) models (e.g., BERT and GPT models like ChatGPT), as well as systems like AlphaGo Zero, AlphaStar, and AlphaFold. Its impact extends beyond image recognition, influencing [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) across domains.

## Mathematics of Residual Connections

In a multilayer neural network, consider a subnetwork with stacked layers (e.g., 2 or 3). Let \(H(x; \alpha)\) denote the subnetwork, where \(x\) is the input and \(\alpha\) is the parameter set. If \(H^*\) is the desired optimal output, residual learning adds \(x\) directly to the output, so the optimal learned output becomes \(H^* - x\), interpreted as a "residual" relative to \(x\). The operation of adding \(x\) is implemented via a skip connection that performs an identity mapping, connecting the subnetwork's input to its output. This connection is later termed a "residual connection."

The function \(F(x; \alpha) = H(x; \alpha) + x\) is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). Such a subnetwork is called a "residual block," and a deep residual network stacks these blocks.

Long short-term memory (LSTM) networks have a memory mechanism that serves as a residual connection. In an LSTM without a forget gate, an input \(x_t\) is processed by \(F\) and added to a memory cell \(c_t\), yielding \(c_{t+1} = c_t + F(x_t)\). An LSTM with a forget gate essentially functions as a highway network.

To stabilize variance of layer inputs, it is recommended to replace residual connections \(x + f(x)\) with \(x/L + f(x)\), where \(L\) is the total number of residual layers.

## Projection Connections

If the function \(F\) maps from \(\mathbb{R}^n\) to \(\mathbb{R}^m\) with \(n \neq m\), the expression \(F(x) + x\) is undefined. To handle this, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection defined by \(P(x) = Mx\), with \(M\) an \(m \times n\) matrix. The matrix is trained via backpropagation like any other parameter.

## Signal Propagation

The introduction of identity mappings facilitates signal propagation in both forward and backward paths. In forward propagation, if the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th block (assuming no activation between blocks), then the next input is \(x_{\ell+1} = F(x_\ell) + x_\ell\). This additive structure allows gradients to flow directly through the network, mitigating the vanishing gradient problem that plagued earlier deep networks.

## Historical Context and Development

The residual connection motif had been used prior to ResNet, for instance in LSTM networks and highway networks. However, the 2015 publication of ResNet made it widely popular for feedforward networks, appearing in architectures seemingly unrelated to image recognition. The original ResNet paper, authored by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, introduced deep residual learning for image recognition and achieved state-of-the-art results on ILSVRC 2015.

## Impact on Deep Learning

ResNet's success demonstrated that very deep networks could be trained effectively, leading to a shift in [neural-network](https://www.wikiprompt.org/wiki/neural-network) design. The residual connection became a standard component in subsequent architectures, including [transformer](https://www.wikiprompt.org/wiki/transformer) models used in [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s. For example, BERT and GPT models rely on residual connections to train deep transformers. Systems like AlphaGo Zero and AlphaFold also incorporate residual blocks, showing the motif's versatility.

## Variants and Extensions

Several variants of residual networks have been proposed. Pre-activation ResNets move batch normalization and activation before the weight layers, improving training. Wide ResNets increase width rather than depth, achieving better performance with fewer layers. DenseNet connects each layer to every subsequent layer, building on the residual idea. ResNeXt introduces cardinality, splitting convolutions into parallel paths. These variants highlight the flexibility of the residual concept.

## Practical Considerations

In practice, residual connections are implemented with skip connections that may include projection matrices when dimensions change. Normalization techniques like batch normalization are often applied within residual blocks. The choice of activation functions, such as ReLU, affects training dynamics. Residual connections also enable the use of higher learning rates and reduce sensitivity to initialization, as they keep the signal variance stable.

## Legacy and Future Directions

The residual network has become a foundational concept in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning), influencing both academic research and industrial applications. Its principles are embedded in modern frameworks and hardware optimizations. As of the early 2020s, residual connections remain a default choice in many architectures, including those for [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) and [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems. The idea of learning residuals has also inspired other areas, such as [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) theory and optimization.

## See Also

- [deep-learning](https://www.wikiprompt.org/wiki/deep-learning)
- [neural-network](https://www.wikiprompt.org/wiki/neural-network)
- [transformer](https://www.wikiprompt.org/wiki/transformer)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)

---
Source: https://www.wikiprompt.org/wiki/residual-network
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:29:17.062492+00:00