In machine learning, the Highway Network is a deep feedforward neural network architecture that was the first to successfully train networks with hundreds of layers, far exceeding the 20 to 30 layers typical of state-of-the-art models in 2014. It was introduced in May 2015 by researchers seeking to overcome the "degradation" problem, where stacking too many layers caused a steep reduction in training accuracy. The architecture uses learned gating mechanisms to regulate information flow across layers, inspired by long short-term memory (LSTM) recurrent neural networks. These gates create "information highways" that allow gradients to propagate more easily, mitigating the vanishing gradient problem and improving optimization.
The Highway Network was developed concurrently with the residual neural network (ResNet), published in December 2015. ResNet can be seen as a special case of a Highway Network where the gates are always open (activation 1.0). Highway Networks have found practical applications in text sequence labeling and speech recognition tasks.
Model and Gating Mechanism
The Highway Network extends a standard feedforward layer by adding two gates alongside the main transformation function \(H(W_H, x)\): a transform gate \(T(W_T, x)\) and a carry gate \(C(W_C, x)\). Both gates are typically sigmoid functions, outputting values between 0 and 1. The carry gate is defined as \(C(W_C, x) = 1 - T(W_T, x)\), ensuring that the transform and carry paths sum to one.
The output of a Highway layer is computed as:
\(y = H(x, W_H) \cdot T(x, W_T) + x \cdot C(x, W_C)\)
or equivalently:
\(y = H(x, W_H) \cdot T(x, W_T) + x \cdot (1 - T(x, W_T))\)
This formulation allows the layer to either pass the input through unchanged (when the transform gate is near 0) or apply a nonlinear transformation (when the gate is near 1). The gating is learned during training, enabling the network to dynamically control how much information flows through each layer.
Relationship to LSTM and ResNet
The Highway Network draws direct inspiration from LSTM recurrent neural networks. Sepp Hochreiter analyzed the vanishing gradient problem in 1991, attributing it to the difficulty of training deep networks. The original LSTM (1997) introduced a constant error carousel, a residual connection with a fixed weight of 1.0, allowing gradients to flow across long time steps. A later LSTM variant (2000) added learnable "forget gates" that modulate these identity connections, addressing the vanishing gradient problem more flexibly.
The Highway Network applies these principles to feedforward networks. It is analogous to an LSTM with forget gates unfolded in time, where the gates are learned. In contrast, ResNet, published later in December 2015, has no equivalent of forget gates; its skip connections are always open, resembling the original 1997 LSTM. If the gates in a Highway Network are kept open (activation 1.0), it becomes a ResNet.
The residual connection is a special case of the broader "short-cut connection" or "skip connection" concept, which dates back to Rosenblatt (1961) and Lang & Witbrock (1988). These connections take the form \(x \mapsto F(x) + Ax\), where \(A\) is a weight matrix. In a residual connection, \(A\) is the identity matrix, but in general skip connections, \(A\) can be arbitrary. Thus, every residual connection is a skip connection, but not all skip connections are residual.
Training and Performance
The original Highway Network paper reported experiments with networks of 20, 50, and 100 layers, and mentioned ongoing work with up to 900 layers. The gating mechanism allowed these very deep networks to be trained effectively, achieving better optimization than plain deep networks. By regulating information flow, the gates help prevent the vanishing gradient problem, where gradients become too small to update weights in earlier layers.
Compared to other deep learning architectures, Highway Networks offer the advantage of improved trainability for very deep models. However, they require additional parameters for the gates, increasing computational cost. The success of Highway Networks and ResNets demonstrated that very deep architectures could be trained effectively, paving the way for subsequent advances in deep learning.
Applications and Legacy
Highway Networks have been applied to text sequence labeling and speech recognition, where deep architectures are beneficial for capturing complex patterns. They also influenced the development of later architectures, such as ResNet, which became a foundational building block in computer vision. The concept of gated skip connections has been adopted in various forms in modern deep learning, including in transformers and large language models.
The ideas behind Highway Networks are part of the broader evolution of deep learning, which has been driven by contributions from institutions like the University of Toronto and researchers such as Sepp Hochreiter and Jürgen Schmidhuber. While Highway Networks themselves are less commonly used today, their principles remain relevant in understanding how to train very deep neural networks.