# Residual Connection

A residual connection is a neural network motif that adds a layer's input to its output, enabling stable training of very deep networks. It was popularized by the ResNet architecture in 2015 and is now a standard component in models like transformers.

A residual connection is a fundamental architectural motif in deep learning where a layer's input is added directly to its output, allowing the network to learn residual functions rather than the full transformation. Formally, for an input \(x\) and a subnetwork \(f\), the residual connection computes \(f(x) + x\). This simple addition stabilizes training and enables the construction of neural networks with hundreds or even thousands of layers, which would otherwise suffer from degradation and vanishing gradients. The concept was popularized by the residual network (ResNet) architecture in 2015, which won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) that year, and has since become a ubiquitous component in modern [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) systems, including [transformer](https://www.wikiprompt.org/wiki/transformer) models and [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s.

The term "residual connection" specifically refers to the mapping \(x \mapsto f(x) + x\), where \(f\) is an arbitrary neural network module. While the motif had been used in earlier work, such as in long short-term memory (LSTM) networks, the publication of ResNet made it widely adopted for feedforward networks. Today, residual connections appear in architectures seemingly unrelated to ResNet, including [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) models, [neural-network](https://www.wikiprompt.org/wiki/neural-network) systems for image recognition, and [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) applications across various domains.

## Mathematics of Residual Connections

In a multilayer neural network, consider a subnetwork with stacked layers (e.g., 2 or 3 layers) that maps an input \(x\) to an output \(H(x; \alpha)\), where \(\alpha\) represents the parameters. Without residual connections, the subnetwork must learn the desired optimal output \(H^*\) directly. With a residual connection, the subnetwork learns the residual \(H^* - x\), and the final output is \(F(x; \alpha) = H(x; \alpha) + x\). This reparameterization makes it easier for the network to learn identity mappings, as the subnetwork only needs to adjust the deviation from the input.

The operation is implemented via a "skip connection" that performs an identity mapping, connecting the input of the subnetwork directly to its output. This skip connection is what is referred to as a residual connection. The function \(F\) is typically represented by matrix multiplications interlaced with activation functions and normalization operations, such as [batch-normalization](https://www.wikiprompt.org/wiki/batch-normalization) or [layer-normalization](https://www.wikiprompt.org/wiki/layer-normalization). A residual block is one such subnetwork with its skip connection, and a deep residual network is constructed by stacking these blocks.

## Projection Connections

When the subnetwork \(F\) changes the dimensionality of the input, such that \(F: \mathbb{R}^n \to \mathbb{R}^m\) with \(n \neq m\), the addition \(F(x) + x\) is undefined. In this case, a projection connection is used: \(y = F(x) + P(x)\), where \(P\) is typically a linear projection defined by \(P(x) = Mx\), with \(M\) being an \(m \times n\) matrix. This projection matrix is trained via backpropagation, just like any other parameter in the model. Projection connections allow residual architectures to be applied across layers with different widths or channel counts, which is common in convolutional networks like ResNet when transitioning between stages.

## Signal Propagation

The introduction of identity mappings in residual connections facilitates signal propagation in both forward and backward paths, which is key to their effectiveness in training deep networks.

### Forward Propagation

If the output of the \(\ell\)-th residual block is the input to the \((\ell+1)\)-th residual block (assuming no activation function between blocks), then the input to the next block is \(x_{\ell+1} = F(x_\ell) + x_\ell\). This recursive relationship means that the signal from earlier layers can flow directly to later layers through the identity mappings, avoiding the degradation of information that occurs in non-residual networks. Over \(L\) blocks, the output can be expressed as \(x_L = x_0 + \sum_{i=0}^{L-1} F(x_i)\), showing that the initial input is always present in the final output.

### Backward Propagation

During backpropagation, the gradient of the loss with respect to the input of a residual block is \(\frac{\partial \mathcal{L}}{\partial x_\ell} = \frac{\partial \mathcal{L}}{\partial x_{\ell+1}} \left(1 + \frac{\partial F}{\partial x_\ell}\right)\). The term \(1\) ensures that the gradient can flow backward through the identity mapping without vanishing, even if the subnetwork \(F\) has small or zero gradients. This prevents the vanishing gradient problem that plagues very deep non-residual networks, allowing effective training of architectures with hundreds of layers.

## Historical Development

The residual connection motif predates ResNet. The LSTM, introduced in 1997, has a memory cell mechanism that serves as a residual connection: an input \(x_t\) is processed by a function \(F\) and added to the memory cell \(c_t\), resulting in \(c_{t+1} = c_t + F(x_t)\). An LSTM with a forget gate essentially functions as a highway network, which is another early variant of gated residual connections. However, it was the 2015 ResNet paper by researchers at [microsoft](https://www.wikiprompt.org/wiki/microsoft) (including Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun) that demonstrated the power of residual connections for very deep feedforward networks. ResNet won the ILSVRC 2015 with an error rate of 3.57% on the ImageNet dataset, surpassing human-level performance for the first time.

## Impact on Deep Learning

Residual connections have had a transformative impact on [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning). They enabled the development of networks with unprecedented depth, such as ResNet-152 with 152 layers, which was previously impossible to train effectively. The success of ResNet in image recognition led to the rapid adoption of residual connections in other domains, including natural language processing and [reinforcement-learning](https://www.wikiprompt.org/wiki/reinforcement-learning).

In [transformer](https://www.wikiprompt.org/wiki/transformer) models, residual connections are a core component. The original transformer architecture, introduced in 2017, uses residual connections around each sublayer (e.g., multi-head attention and feedforward networks) followed by layer normalization. This design is used in models like BERT and the GPT series, including those developed by [openai](https://www.wikiprompt.org/wiki/openai). The stability provided by residual connections is essential for training these models, which can have hundreds of layers and billions of parameters.

Residual connections also appear in other notable systems. The AlphaGo Zero system, developed by [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), uses residual networks in its neural networks to master the game of Go. AlphaStar, which achieved grandmaster level in StarCraft II, and AlphaFold, which predicts protein structures, also incorporate residual connections. These applications demonstrate the versatility of the motif across different types of neural network architectures.

## Variants and Improvements

Several variants of residual connections have been proposed to further improve training stability and performance. One recommendation is to replace the residual connection \(x + f(x)\) with \(x/L + f(x)\), where \(L\) is the total number of residual layers. This scaling helps stabilize the variance of layer inputs, which is particularly important in very deep networks. This technique is used in some modern transformer implementations, such as GPT-2, where the residual stream is scaled by \(1/\sqrt{L}\).

Another variant is the pre-activation residual block, where the activation function (e.g., ReLU) is applied before the weight layers rather than after. This modification, introduced in 2016, improves gradient flow and has been shown to make training even deeper networks easier. Additionally, some architectures use gated residual connections, where a learned gate controls the contribution of the residual path, as seen in highway networks.

## Practical Considerations

When implementing residual connections, several practical considerations arise. The choice of where to place normalization (before or after the addition) can significantly affect training dynamics. In transformers, the "post-norm" (normalization after addition) was used in the original paper, but "pre-norm" (normalization before the sublayer) has become more common in later models due to improved stability. The scaling factor for the residual branch is another hyperparameter that can be tuned.

Residual connections also interact with other techniques like [dropout](https://www.wikiprompt.org/wiki/dropout) and [weight-initialization](https://www.wikiprompt.org/wiki/weight-initialization). In practice, residual connections are often used in conjunction with [gradient-clipping](https://www.wikiprompt.org/wiki/gradient-clipping) and careful [learning-rate-schedule](https://www.wikiprompt.org/wiki/learning-rate-schedule) management to train very deep models. The combination of these techniques has enabled the training of models with trillions of parameters, such as those used in modern [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s.

## Conclusion

Residual connections are a simple yet powerful idea that has revolutionized deep learning. By adding the input to the output of a subnetwork, they allow gradients to flow through deep networks, preventing degradation and enabling the training of architectures with hundreds of layers. From their origins in the 2015 ResNet paper to their ubiquitous presence in transformers and other modern models, residual connections have become a foundational tool in the field. Their impact extends across image recognition, natural language processing, and beyond, making them one of the most important architectural innovations in the history of [machine-learning](https://www.wikiprompt.org/wiki/machine-learning).

---
Source: https://www.wikiprompt.org/wiki/residual-connection
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T22:26:49.958807+00:00