A recurrent neural network, or RNN, is a Neural network architecture designed to process sequential data by maintaining an internal hidden state that is updated at each step and carried forward to influence how subsequent inputs are processed. Unlike feedforward networks, which map a fixed input to a fixed output with no memory of prior inputs, RNNs share the same weights across every step of a sequence and, in principle, can use information from arbitrarily far back in that sequence to inform each new prediction.
History
Early recurrent architectures included John Hopfield's 1982 Hopfield network, an associative memory model, and the simple recurrent network proposed by Jeffrey Elman in 1990, which showed that a network could learn to represent grammatical structure in a stream of words without being told the rules explicitly. Throughout the 1990s and 2000s, RNNs were the default architecture for sequence tasks in Natural language processing and Speech recognition, but plain RNNs proved difficult to train on long sequences because gradients computed through Backpropagation through time tend to shrink or blow up exponentially with sequence length, a problem formally analyzed by Sepp Hochreiter in 1991. This vanishing and exploding gradient problem motivated the LSTM architecture in 1997 and, later, the simpler gated recurrent unit, both of which added gating mechanisms that let gradients flow more stably across many steps.
Architecture and training
An RNN processes a sequence one element at a time, combining the current input with the hidden state carried over from the previous step to produce a new hidden state and, optionally, an output at that step. Training uses backpropagation through time, which unrolls the recurrent computation across all steps of a sequence and applies ordinary Gradient descent to the resulting, effectively very deep, unrolled network. This unrolling is also RNNs' main computational drawback: because each step depends on the previous one's output, the sequence must largely be processed step by step rather than in parallel, making RNNs slow to train on the long sequences and large datasets that became standard in the 2010s.
Peak and decline
RNNs, and particularly stacked LSTM variants, powered the first generation of practical Machine translation and speech systems through architectures like Seq2seq, introduced in 2014, which paired an encoder RNN that compressed an input sequence into a fixed vector with a decoder RNN that generated the output sequence from it. Attention mechanisms, added to seq2seq models around 2015 to let the decoder look back at all encoder states rather than relying on a single compressed vector, both improved translation quality and set the stage for the 2017 Transformer (architecture) architecture, which discarded recurrence entirely in favor of self-attention and could be trained fully in parallel. Because transformers matched or exceeded RNN performance while training far faster on the GPU (in AI) hardware available at the time, they displaced RNNs as the dominant sequence architecture for Large language models and most of Natural language processing within a few years.
Legacy
RNNs remain used in settings with strict memory or latency constraints and in some time-series and control applications, and their core idea, that a fixed-size hidden state can summarize an unboundedly long history, resurfaced in the mid-2020s in State space models such as Mamba, which aim to combine RNN-like linear-time inference with training efficiency closer to transformers.