The gated recurrent unit (GRU) is a type of gating mechanism used in recurrent neural networks, introduced in 2014 by Kyunghyun Cho and colleagues. It is designed to address the vanishing gradient problem common in standard recurrent networks by using two gates - an update gate and a reset gate - to control how much of the past information is retained or forgotten. Compared to the long short-term memory (LSTM) unit, the GRU has a simpler architecture: it lacks a separate context vector and output gate, resulting in fewer parameters and often faster training. Despite this simplification, GRUs have shown performance comparable to LSTMs on tasks such as polyphonic music modeling, speech signal modeling, and natural language processing. Research by Yoshua Bengio's team found that gating is broadly beneficial, but did not reach a definitive conclusion on whether GRUs or LSTMs are superior in general.
Architecture
The GRU processes input sequences step by step, maintaining a hidden state that carries information across time steps. At each time step, the unit computes an update gate and a reset gate, both of which are functions of the current input and the previous hidden state. The update gate determines how much of the previous state to carry forward, while the reset gate decides how much of the past state to use when computing a candidate activation. The final hidden state is a convex combination of the previous state and the candidate activation, weighted by the update gate.
Mathematically, for an input vector \(x_t\) and previous hidden state \(h_{t-1}\), the gates and candidate activation are computed as:
- Update gate: \(z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)\)
- Reset gate: \(r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)\)
- Candidate activation: \(\hat{h}_t = \phi(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)\)
- Final hidden state: \(h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \hat{h}_t\)
Here, \(\sigma\) is the sigmoid activation function, \(\phi\) is typically the hyperbolic tangent, and \(\odot\) denotes the Hadamard (element-wise) product. The weight matrices \(W_z, W_r, W_h\) and \(U_z, U_r, U_h\) along with bias vectors \(b_z, b_r, b_h\) are learned during training. The initial hidden state is usually set to zero.
Variants
Several variations of the GRU exist, differing in how the gates are computed or combined. A notable simplified version is the minimal gated unit, which uses only one gate (often a combination of update and reset) to reduce computational complexity. Other variants may adjust the order of operations or the way the reset gate interacts with the previous state. These modifications aim to improve efficiency or performance on specific tasks, but the core principle of gated information flow remains consistent.
Applications
GRUs have been widely applied in sequence modeling tasks, including natural language processing (e.g., machine translation, language modeling), speech recognition, and music generation. They are also used in time series forecasting and in hybrid architectures where they are combined with convolutional layers or attention mechanisms. In many settings, GRUs serve as a lightweight alternative to LSTMs, especially when computational resources are limited or when the sequence length is moderate.
Comparison with LSTM
Both GRUs and LSTMs were developed to mitigate the vanishing gradient problem, but they differ in internal structure. An LSTM has three gates (input, forget, and output) and a separate cell state, whereas a GRU has two gates and no separate cell state. This makes the GRU parameter-efficient and often faster to train. Empirically, studies have found that GRUs perform similarly to LSTMs on many benchmark tasks, though results can vary depending on the dataset and task. Some research suggests that GRUs may generalize better on smaller datasets, while LSTMs might excel on larger ones, but no definitive advantage has been established.