Long Short-Term Memory, or LSTM, is a Recurrent neural network architecture designed to learn long-range dependencies in sequential data by using gated units that regulate how information is added to, retained in, or removed from a persistent internal memory cell. It was introduced specifically to solve the vanishing gradient problem that made plain RNNs unable to learn relationships between events separated by many time steps.
History
LSTM was introduced in a 1997 paper by Sepp Hochreiter and Jürgen Schmidhuber, building on Hochreiter's 1991 diploma thesis analysis of why gradients vanish or explode in standard Recurrent neural networks trained with Backpropagation through time. The original architecture was refined over the following decade, most notably with the addition of a forget gate by Felix Gers and colleagues in 2000, which let the network learn to actively reset its memory cell rather than only accumulate information indefinitely. LSTM saw limited adoption for years after publication but became one of the most widely used architectures in Deep learning through the 2010s as larger datasets and GPU (in AI) training made its advantages over plain RNNs clearly measurable.
Architecture
An LSTM unit maintains a cell state, which acts as a conveyor belt carrying information across time steps largely unchanged, alongside three gates, the input, forget, and output gates, each a small neural layer that outputs values between zero and one to control how much information passes through. The forget gate decides what to discard from the cell state, the input gate decides what new information to add, and the output gate decides what part of the cell state to expose as the unit's output at that step. Because the cell state is updated through addition rather than repeated multiplication, gradients can flow backward through many time steps largely undiminished, which is the core mechanism that lets LSTMs learn dependencies spanning hundreds of steps where plain RNNs typically failed beyond just a handful.
Applications
Stacked and bidirectional LSTMs became the standard architecture for Machine translation, Speech recognition, and text generation throughout the 2010s, and LSTM-based Seq2seq models with attention were the direct predecessor to the Transformer (architecture) architecture. Google's neural machine translation system, deployed in 2016, used a deep LSTM stack and represented one of the largest production deployments of the architecture. LSTMs were also widely used outside language, including in time-series forecasting, handwriting recognition, and early Reinforcement learning agents that needed to remember information across long action sequences.
Decline and legacy
The 2017 introduction of the Transformer (architecture) architecture, which processes entire sequences in parallel using self-attention rather than sequentially through gated recurrence, offered both better training efficiency on GPU (in AI) hardware and, at scale, better performance, and LSTMs were largely displaced as the default choice for new Natural language processing and Large language model systems within a few years. Hochreiter's group revisited the architecture in 2024 with xLSTM, an attempt to modernize LSTM with techniques learned from transformers, arguing that recurrent architectures could still be competitive for very long sequences, though it has not displaced transformers or newer State space models in mainstream use as of 2025.