ResNet, short for residual neural network, is a deep learning architecture in which layers learn residual functions with reference to their inputs. Developed in 2015 for image recognition, it won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year. The architecture's defining feature is the residual connection, a motif that adds the input of a subnetwork to its output, enabling the training of networks with hundreds of layers. This design has become a fundamental building block in modern Deep learning models, including Transformer (architecture) architectures and systems like AlphaGo Zero and AlphaFold.
The residual connection stabilizes training and convergence in deep neural networks, addressing the vanishing gradient problem that previously limited network depth. Its influence extends far beyond image recognition, appearing in neural networks across domains, from large language models to reinforcement learning systems.
Mathematics
The core idea of ResNet is to reformulate the learning target. In a multilayer network, consider a subnetwork with stacked layers that computes a function \(H(x;\alpha)\), where \(x\) is the input and \(\alpha\) are the parameters. Instead of learning \(H\) directly, residual learning makes the subnetwork learn \(F(x;\alpha) = H(x;\alpha) + x\). The optimal output \(H^\) is then achieved by learning the residual \(H^ - x\), which is often easier to optimize.
The addition of \(x\) is implemented via a skip connection that performs an identity mapping, connecting the subnetwork's input directly to its output. This skip connection is the "residual connection." A subnetwork with this connection is called a residual block, and a deep residual network is built by stacking such blocks.
Residual Connection
In a residual block, the function \(F\) typically consists of matrix multiplications interleaved with activation functions and normalization operations like batch normalization or layer normalization. The residual connection \(x + F(x)\) allows gradients to flow directly through the network during backpropagation, mitigating the degradation problem where deeper networks become harder to train.
The concept of residual connections has precedents. Long short-term memory (LSTM) networks, for instance, have a memory cell that acts as a residual connection: \(c_{t+1} = c_t + F(x_t)\), where \(c_t\) is the cell state and \(F\) processes the input \(x_t\). An LSTM with a forget gate essentially functions as a highway network, a related architecture that also uses gated skip connections.
To stabilize the variance of layer inputs in very deep networks, it is recommended to scale the residual connection by the total number of layers \(L\), replacing \(x + f(x)\) with \(x/L + f(x)\). This normalization helps maintain consistent signal magnitudes across hundreds of layers.
Projection Connection
When the function \(F\) changes the dimensionality of the input, such as \(F: \mathbb{R}^n \to \mathbb{R}^m\) with \(n \neq m\), the addition \(F(x) + x\) is undefined. In this case, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection \(P(x) = Mx\) with an \(m \times n\) matrix \(M\). This matrix is trained via backpropagation along with other model parameters, allowing the network to adapt the skip connection to the new dimensionality.
Signal Propagation
The introduction of identity mappings facilitates signal propagation in both forward and backward paths. In the forward pass, if the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th block (assuming no activation between blocks), then \(x_{\ell+1} = F(x_\ell) + x_\ell\). This recurrence allows information to flow directly through the network, preserving the input signal even if \(F\) produces small values.
In the backward pass, the gradient of the loss with respect to the input of a block includes a term that is the gradient with respect to the output, passed through the identity mapping. This ensures that gradients do not vanish as they propagate through many layers, a key factor in enabling the training of networks with hundreds of layers.
History and Impact
The residual connection motif had been used in earlier work, but the publication of ResNet in 2015 made it widely popular for feedforward networks. The architecture was developed by researchers at Microsoft, including Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Their paper, "Deep Residual Learning for Image Recognition," demonstrated that residual networks could be trained with depths of 152 layers or more, achieving state-of-the-art results on the ImageNet benchmark.
ResNet's success at ILSVRC 2015 marked a turning point in machine learning. It showed that very deep networks were not only feasible but also superior in performance, leading to a surge in research on deep architectures. The residual connection became a standard component in subsequent models, including Transformers used in natural language processing, such as BERT and GPT models, and in systems like AlphaGo Zero, AlphaStar, and AlphaFold.
Variants and Extensions
Several variants of ResNet have been developed to improve performance or efficiency. Pre-activation ResNet moves the activation functions before the weight layers, which improves regularization and allows even deeper networks. Wide ResNet increases the number of channels in each block, trading depth for width to achieve better accuracy with fewer layers. ResNeXt introduces a cardinality dimension, using parallel branches within each block to increase representational power without increasing depth.
Other extensions include DenseNet, which connects each layer to every other layer in a feedforward fashion, and ResNet with stochastic depth, which randomly drops layers during training to regularize the network. These variants have further pushed the boundaries of what is possible with deep residual learning.
Applications
ResNet has been applied to a wide range of tasks beyond image recognition, including object detection, semantic segmentation, and video analysis. It is also used in medical imaging, autonomous driving, and facial recognition systems. The architecture's versatility and robustness have made it a default choice for many computer vision applications.
In addition, the residual connection principle has been adopted in other domains, such as natural language processing and reinforcement learning. For example, Transformer models rely on residual connections to train deep networks for language understanding and generation, as seen in OpenAI's GPT series and Anthropic's models. The influence of ResNet is thus pervasive across the field of artificial intelligence.