A residual neural network, also known as a residual network or ResNet, is a deep learning architecture in which layers learn residual functions with reference to their inputs. Developed in 2015 for image recognition, it won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year. The key innovation is the residual connection, an architectural motif that adds the input of a subnetwork to its output, enabling the training of networks with hundreds of layers.
The residual connection stabilizes training and convergence in deep neural networks, making it a common motif in modern architectures such as Transformer (architecture) models (e.g., BERT and GPT models like ChatGPT), as well as systems like AlphaGo Zero, AlphaStar, and AlphaFold. Its impact extends beyond image recognition, influencing Deep learning across domains.
Mathematics of Residual Connections
In a multilayer neural network, consider a subnetwork with stacked layers (e.g., 2 or 3). Let \(H(x; \alpha)\) denote the subnetwork, where \(x\) is the input and \(\alpha\) is the parameter set. If \(H^\) is the desired optimal output, residual learning adds \(x\) directly to the output, so the optimal learned output becomes \(H^ - x\), interpreted as a "residual" relative to \(x\). The operation of adding \(x\) is implemented via a skip connection that performs an identity mapping, connecting the subnetwork's input to its output. This connection is later termed a "residual connection."
The function \(F(x; \alpha) = H(x; \alpha) + x\) is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). Such a subnetwork is called a "residual block," and a deep residual network stacks these blocks.
Long short-term memory (LSTM) networks have a memory mechanism that serves as a residual connection. In an LSTM without a forget gate, an input \(x_t\) is processed by \(F\) and added to a memory cell \(c_t\), yielding \(c_{t+1} = c_t + F(x_t)\). An LSTM with a forget gate essentially functions as a highway network.
To stabilize variance of layer inputs, it is recommended to replace residual connections \(x + f(x)\) with \(x/L + f(x)\), where \(L\) is the total number of residual layers.
Projection Connections
If the function \(F\) maps from \(\mathbb{R}^n\) to \(\mathbb{R}^m\) with \(n \neq m\), the expression \(F(x) + x\) is undefined. To handle this, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection defined by \(P(x) = Mx\), with \(M\) an \(m \times n\) matrix. The matrix is trained via backpropagation like any other parameter.
Signal Propagation
The introduction of identity mappings facilitates signal propagation in both forward and backward paths. In forward propagation, if the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th block (assuming no activation between blocks), then the next input is \(x_{\ell+1} = F(x_\ell) + x_\ell\). This additive structure allows gradients to flow directly through the network, mitigating the vanishing gradient problem that plagued earlier deep networks.
Historical Context and Development
The residual connection motif had been used prior to ResNet, for instance in LSTM networks and highway networks. However, the 2015 publication of ResNet made it widely popular for feedforward networks, appearing in architectures seemingly unrelated to image recognition. The original ResNet paper, authored by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, introduced deep residual learning for image recognition and achieved state-of-the-art results on ILSVRC 2015.
Impact on Deep Learning
ResNet's success demonstrated that very deep networks could be trained effectively, leading to a shift in Neural network design. The residual connection became a standard component in subsequent architectures, including Transformer (architecture) models used in Large language models. For example, BERT and GPT models rely on residual connections to train deep transformers. Systems like AlphaGo Zero and AlphaFold also incorporate residual blocks, showing the motif's versatility.
Variants and Extensions
Several variants of residual networks have been proposed. Pre-activation ResNets move batch normalization and activation before the weight layers, improving training. Wide ResNets increase width rather than depth, achieving better performance with fewer layers. DenseNet connects each layer to every subsequent layer, building on the residual idea. ResNeXt introduces cardinality, splitting convolutions into parallel paths. These variants highlight the flexibility of the residual concept.
Practical Considerations
In practice, residual connections are implemented with skip connections that may include projection matrices when dimensions change. Normalization techniques like batch normalization are often applied within residual blocks. The choice of activation functions, such as ReLU, affects training dynamics. Residual connections also enable the use of higher learning rates and reduce sensitivity to initialization, as they keep the signal variance stable.
Legacy and Future Directions
The residual network has become a foundational concept in Deep learning, influencing both academic research and industrial applications. Its principles are embedded in modern frameworks and hardware optimizations. As of the early 2020s, residual connections remain a default choice in many architectures, including those for Generative AI and Artificial intelligence systems. The idea of learning residuals has also inspired other areas, such as Machine learning theory and optimization.