# LSTM

LSTM (Long Short-Term Memory) is a recurrent neural network architecture with gated memory cells that allows it to learn long-range dependencies in sequential data.

Long Short-Term Memory, or LSTM, is a [recurrent-neural-network](https://www.wikiprompt.org/wiki/recurrent-neural-network) architecture designed to learn long-range dependencies in sequential data by using gated units that regulate how information is added to, retained in, or removed from a persistent internal memory cell. It was introduced specifically to solve the vanishing gradient problem that made plain RNNs unable to learn relationships between events separated by many time steps.

## History

LSTM was introduced in a 1997 paper by [sepp-hochreiter](https://www.wikiprompt.org/wiki/sepp-hochreiter) and [juergen-schmidhuber](https://www.wikiprompt.org/wiki/juergen-schmidhuber), building on Hochreiter's 1991 diploma thesis analysis of why gradients vanish or explode in standard [recurrent-neural-network](https://www.wikiprompt.org/wiki/recurrent-neural-network)s trained with [backpropagation](https://www.wikiprompt.org/wiki/backpropagation) through time. The original architecture was refined over the following decade, most notably with the addition of a forget gate by Felix Gers and colleagues in 2000, which let the network learn to actively reset its memory cell rather than only accumulate information indefinitely. LSTM saw limited adoption for years after publication but became one of the most widely used architectures in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) through the 2010s as larger datasets and [gpu](https://www.wikiprompt.org/wiki/gpu) training made its advantages over plain RNNs clearly measurable.

## Architecture

An LSTM unit maintains a cell state, which acts as a conveyor belt carrying information across time steps largely unchanged, alongside three gates, the input, forget, and output gates, each a small neural layer that outputs values between zero and one to control how much information passes through. The forget gate decides what to discard from the cell state, the input gate decides what new information to add, and the output gate decides what part of the cell state to expose as the unit's output at that step. Because the cell state is updated through addition rather than repeated multiplication, gradients can flow backward through many time steps largely undiminished, which is the core mechanism that lets LSTMs learn dependencies spanning hundreds of steps where plain RNNs typically failed beyond just a handful.

## Applications

Stacked and bidirectional LSTMs became the standard architecture for [machine-translation](https://www.wikiprompt.org/wiki/machine-translation), [speech-recognition](https://www.wikiprompt.org/wiki/speech-recognition), and text generation throughout the 2010s, and LSTM-based [seq2seq](https://www.wikiprompt.org/wiki/seq2seq) models with attention were the direct predecessor to the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. Google's neural machine translation system, deployed in 2016, used a deep LSTM stack and represented one of the largest production deployments of the architecture. LSTMs were also widely used outside language, including in time-series forecasting, handwriting recognition, and early [reinforcement-learning](https://www.wikiprompt.org/wiki/reinforcement-learning) agents that needed to remember information across long action sequences.

## Decline and legacy

The 2017 introduction of the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, which processes entire sequences in parallel using self-attention rather than sequentially through gated recurrence, offered both better training efficiency on [gpu](https://www.wikiprompt.org/wiki/gpu) hardware and, at scale, better performance, and LSTMs were largely displaced as the default choice for new [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) and [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) systems within a few years. Hochreiter's group revisited the architecture in 2024 with xLSTM, an attempt to modernize LSTM with techniques learned from transformers, arguing that recurrent architectures could still be competitive for very long sequences, though it has not displaced transformers or newer [state-space-model](https://www.wikiprompt.org/wiki/state-space-model)s in mainstream use as of 2025.

---
Source: https://www.wikiprompt.org/wiki/lstm
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:29:53.357844+00:00