The ResNet paper, formally titled "Deep Residual Learning for Image Recognition," was published in 2015 by researchers at Microsoft Research. It introduced the residual neural network (ResNet), a Deep learning architecture in which layers learn residual functions with reference to layer inputs. The paper won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of 2015, achieving a top-5 error rate of 3.57%, surpassing human-level performance on that benchmark. Its central innovation, the residual connection, became a foundational motif in subsequent Neural network designs.
A residual connection is the architectural operation \(x \mapsto f(x) + x\), where \(f\) is an arbitrary neural network module. Instead of learning an unreferenced mapping \(H(x)\), the network learns a residual \(F(x) = H(x) - x\), with the input \(x\) added via an identity skip connection. This stabilizes training and convergence for networks with hundreds of layers, a problem that had previously limited depth due to vanishing gradients and degradation. The idea had precedents in earlier work, such as long short-term memory (LSTM) cells, which use additive memory updates akin to residual paths, but ResNet popularized the motif for feedforward networks.
Architecture and Residual Blocks
A residual block consists of a subnetwork \(H(x; \alpha)\) with parameters \(\alpha\), typically two or three stacked convolutional or fully connected layers, interleaved with activation functions and normalization (e.g., batch normalization). The block output is \(F(x) = H(x; \alpha) + x\). Deep residual networks stack many such blocks, with variants like ResNet-34, ResNet-50, ResNet-101, and ResNet-152 denoting layer counts. The 152-layer version won ILSVRC 2015, and the paper demonstrated that deeper residual networks outperformed shallower non-residual counterparts, contrary to prior observations of performance saturation.
When input and output dimensions differ (\(F: \mathbb{R}^n \to \mathbb{R}^m\), \(n \neq m\)), a projection connection is used: \(y = F(x) + P(x)\), where \(P(x) = Mx\) is a learned linear projection matrix \(M\) of shape \(m \times n\), trained via backpropagation. This allows residual learning across feature map size changes, common in convolutional networks when downsampling.
Signal Propagation and Training Stability
The identity mapping in residual connections facilitates signal propagation in both forward and backward directions. In forward propagation, if the output of block \(\ell\) is the input to block \(\ell+1\) (without intervening activations), then \(x_{\ell+1} = F(x_\ell) + x_\ell\). This additive structure ensures that gradients can flow directly to earlier layers during backpropagation, mitigating the vanishing gradient problem. The paper showed that residual networks with hundreds of layers could be trained effectively, whereas plain deep networks suffered from increased training error as depth grew.
To further stabilize variance, later work recommended scaling residual connections as \(x/L + f(x)\), where \(L\) is the total number of residual layers, though the original ResNet paper used unscaled additions. This scaling became relevant in very deep transformers and other architectures.
Impact on Deep Learning
ResNet's publication marked a turning point in Artificial intelligence research, enabling the construction of much deeper models. The residual connection became a standard component in numerous architectures, including Transformer (architecture) models such as BERT and GPT (e.g., Large language model systems like ChatGPT), as well as AlphaGo Zero, AlphaStar, and AlphaFold from Google DeepMind. The motif appears in seemingly unrelated networks, from Computer vision to natural language processing, because it simplifies optimization and allows effective training of very deep stacks.
The paper also influenced hardware and software development, as training deeper models required more compute, contributing to the rise of specialized accelerators like AWS Trainium and cloud services such as Microsoft Azure and Google Cloud. Research groups at University of Toronto, Stanford AI Lab, and MIT CSAIL adopted residual networks in subsequent studies, cementing their role in modern Machine learning practice.
Legacy and Recognition
By 2025, the ResNet paper had become one of the most cited in computer science, with tens of thousands of citations. Its authors, including Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, received numerous awards, including the 2023 IEEE Frank Rosenblatt Award. The architecture remains a baseline for image classification tasks, and its principles underpin many state-of-the-art systems. The term "residual connection" is now standard terminology in deep learning textbooks and courses, and the paper's insights on identity mappings continue to inform research on network depth and trainability.
The paper's success also spurred interest in even deeper architectures, leading to innovations like DenseNet and ResNeXt, though none displaced the fundamental residual idea. As of 2025, residual connections are ubiquitous in production AI systems, from OpenAI's models to Anthropic's Claude, demonstrating the enduring impact of the 2015 work.