# Recurrent neural network

A recurrent neural network is a neural network architecture that maintains a hidden state across sequence steps, allowing it to process sequential data such as text and speech.

A recurrent neural network, or RNN, is a [neural-network](https://www.wikiprompt.org/wiki/neural-network) architecture designed to process sequential data by maintaining an internal hidden state that is updated at each step and carried forward to influence how subsequent inputs are processed. Unlike feedforward networks, which map a fixed input to a fixed output with no memory of prior inputs, RNNs share the same weights across every step of a sequence and, in principle, can use information from arbitrarily far back in that sequence to inform each new prediction.

## History

Early recurrent architectures included [john-hopfield](https://www.wikiprompt.org/wiki/john-hopfield)'s 1982 Hopfield network, an associative memory model, and the simple recurrent network proposed by Jeffrey Elman in 1990, which showed that a network could learn to represent grammatical structure in a stream of words without being told the rules explicitly. Throughout the 1990s and 2000s, RNNs were the default architecture for sequence tasks in [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) and [speech-recognition](https://www.wikiprompt.org/wiki/speech-recognition), but plain RNNs proved difficult to train on long sequences because gradients computed through [backpropagation](https://www.wikiprompt.org/wiki/backpropagation) through time tend to shrink or blow up exponentially with sequence length, a problem formally analyzed by [sepp-hochreiter](https://www.wikiprompt.org/wiki/sepp-hochreiter) in 1991. This vanishing and exploding gradient problem motivated the [lstm](https://www.wikiprompt.org/wiki/lstm) architecture in 1997 and, later, the simpler gated recurrent unit, both of which added gating mechanisms that let gradients flow more stably across many steps.

## Architecture and training

An RNN processes a sequence one element at a time, combining the current input with the hidden state carried over from the previous step to produce a new hidden state and, optionally, an output at that step. Training uses backpropagation through time, which unrolls the recurrent computation across all steps of a sequence and applies ordinary [gradient-descent](https://www.wikiprompt.org/wiki/gradient-descent) to the resulting, effectively very deep, unrolled network. This unrolling is also RNNs' main computational drawback: because each step depends on the previous one's output, the sequence must largely be processed step by step rather than in parallel, making RNNs slow to train on the long sequences and large datasets that became standard in the 2010s.

## Peak and decline

RNNs, and particularly stacked LSTM variants, powered the first generation of practical [machine-translation](https://www.wikiprompt.org/wiki/machine-translation) and speech systems through architectures like [seq2seq](https://www.wikiprompt.org/wiki/seq2seq), introduced in 2014, which paired an encoder RNN that compressed an input sequence into a fixed vector with a decoder RNN that generated the output sequence from it. Attention mechanisms, added to seq2seq models around 2015 to let the decoder look back at all encoder states rather than relying on a single compressed vector, both improved translation quality and set the stage for the 2017 [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, which discarded recurrence entirely in favor of self-attention and could be trained fully in parallel. Because transformers matched or exceeded RNN performance while training far faster on the [gpu](https://www.wikiprompt.org/wiki/gpu) hardware available at the time, they displaced RNNs as the dominant sequence architecture for [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and most of [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) within a few years.

## Legacy

RNNs remain used in settings with strict memory or latency constraints and in some time-series and control applications, and their core idea, that a fixed-size hidden state can summarize an unboundedly long history, resurfaced in the mid-2020s in [state-space-model](https://www.wikiprompt.org/wiki/state-space-model)s such as Mamba, which aim to combine RNN-like linear-time inference with training efficiency closer to transformers.

---
Source: https://www.wikiprompt.org/wiki/recurrent-neural-network
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:29:52.197003+00:00