# ResNet Paper (2015)

ResNet is a deep learning architecture using residual connections, introduced in 2015 for image recognition, winning ILSVRC 2015.

Residual neural networks, commonly known as ResNet, are a class of deep learning architectures that enable the training of very deep neural networks by introducing skip connections that add the input of a layer to its output. The architecture was developed in 2015 for image recognition and won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year. The key innovation is the residual connection, which allows layers to learn residual functions with reference to the layer inputs, stabilizing training and enabling networks with hundreds of layers.

The residual connection motif, defined as \(x \mapsto f(x) + x\), had been used in earlier models, but the publication of ResNet made it widely popular for feedforward networks. It has since become a common component in many deep neural network architectures, including transformer models such as BERT and GPT, as well as systems like AlphaGo Zero, AlphaStar, and AlphaFold.

## Mathematics

### Residual Connection

In a multilayer neural network, consider a subnetwork with a few stacked layers, denoted as \(H(x; \alpha)\), where \(x\) is the input and \(\alpha\) are the parameters. In standard learning, the subnetwork aims to approximate a desired output \(H^*\). Residual learning modifies the subnetwork to output \(F(x; \alpha) = H(x; \alpha) + x\), so that the optimal learned function becomes \(H^* - x\), which is interpreted as the residual with respect to \(x\). The addition of \(x\) is implemented via a skip connection that performs an identity mapping, connecting the input of the subnetwork directly to its output. This skip connection is referred to as a residual connection. The subnetwork, together with the skip connection, forms a residual block. A deep residual network is constructed by stacking multiple such blocks.

The residual connection stabilizes training by facilitating gradient flow, which is particularly important for networks with hundreds of layers. Long short-term memory (LSTM) networks have a memory mechanism that acts as a residual connection: in an LSTM without a forget gate, the update rule is \(c_{t+1} = c_t + F(x_t)\), where \(c_t\) is the memory cell and \(F\) is a function of the input. An LSTM with a forget gate functions similarly to a highway network. To stabilize variance, it is recommended to scale the residual connection by \(1/L\), where \(L\) is the total number of residual layers, resulting in \(x/L + f(x)\).

### Projection Connection

When the function \(F\) maps from \(\mathbb{R}^n\) to \(\mathbb{R}^m\) with \(n \neq m\), the addition \(F(x) + x\) is undefined. In such cases, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection defined by \(P(x) = Mx\), with \(M\) being an \(m \times n\) matrix. The matrix \(M\) is trained via backpropagation along with the other parameters of the model.

### Signal Propagation

The identity mapping in residual connections facilitates signal propagation in both forward and backward directions. In the forward pass, if the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th block (assuming no activation between blocks), then the input to the next block is \(x_{\ell+1} = F(x_\ell) + x_\ell\). This additive structure allows information to flow directly through the network, mitigating the vanishing gradient problem.

## Impact on Deep Learning

The introduction of ResNet in 2015 marked a significant milestone in deep learning. Prior to ResNet, training very deep networks was challenging due to degradation in accuracy as depth increased. ResNet's residual learning framework addressed this by reformulating layers as learning residual functions, which are easier to optimize. The architecture achieved state-of-the-art results on the ILSVRC 2015 challenge, with an error rate of 3.57% on the ImageNet test set, surpassing human-level performance on some tasks.

ResNet's success spurred the development of even deeper architectures, such as ResNet-50, ResNet-101, and ResNet-152, which are widely used as backbone networks in computer vision tasks like object detection and segmentation. The residual connection concept has been adopted in various other domains, including natural language processing and reinforcement learning.

## Legacy and Influence

The residual connection has become a fundamental building block in modern neural network design. It is a key component in transformer architectures, which power large language models such as GPT and BERT. The technique is also used in systems like AlphaGo Zero, AlphaStar, and AlphaFold, demonstrating its versatility across different types of neural networks.

The publication of the ResNet paper, titled "Deep Residual Learning for Image Recognition," was authored by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. It has become one of the most cited papers in computer vision and deep learning, influencing subsequent research on network architecture and training stability.

## See Also

- [deep-learning](https://www.wikiprompt.org/wiki/deep-learning)
- [neural-network](https://www.wikiprompt.org/wiki/neural-network)
- [transformer](https://www.wikiprompt.org/wiki/transformer)
- [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence)

---
Source: https://www.wikiprompt.org/wiki/resnet-paper-2015
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T21:32:47.875156+00:00