# Long Short‑Term Memory (LSTM)

Long short-term memory (LSTM) is a recurrent neural network architecture designed to handle long-range dependencies in sequential data by using gated cells that mitigate the vanishing gradient problem.

Long short-term memory (LSTM) is a type of recurrent neural network (RNN) introduced to address the vanishing gradient problem that limits traditional RNNs. Unlike standard RNNs, LSTM units incorporate a cell state and three gating mechanisms - an input gate, an output gate, and a forget gate - that regulate information flow over arbitrary time intervals. This design allows LSTM networks to maintain dependencies across thousands of timesteps, making them effective for sequence learning tasks such as speech recognition, machine translation, and time series forecasting. The name reflects an analogy to cognitive psychology's concepts of long-term and short-term memory, studied since the early 20th century.

LSTM's relative insensitivity to gap length gives it an advantage over other RNNs, hidden Markov models, and alternative sequence learning methods. The architecture was proposed by Sepp Hochreiter and Jürgen Schmidhuber in 1997, and it has since become a foundational component in deep learning, particularly before the rise of transformer-based models.

## Motivation

Classic RNNs theoretically can capture arbitrary long-term dependencies, but in practice they suffer from vanishing gradients during backpropagation. When training with backpropagation, gradients that propagate over many timesteps can shrink exponentially, causing the network to stop learning effectively. LSTM units mitigate this by allowing gradients to flow with little attenuation through the cell state, although they can still encounter exploding gradients.

The intuition behind LSTM is to create a module that learns when to remember and when to forget information. The network learns which pieces of information are likely needed later in the sequence and when they become irrelevant. For example, in natural language processing, an LSTM processing the sentence "Dave, as a result of his controversial claims, is now a pariah" can remember the grammatical gender and number of the subject "Dave" to correctly interpret the pronoun "his", and then discard that information after the verb "is".

## Architecture

An LSTM unit consists of a cell and three gates. The cell remembers values over arbitrary time intervals. The forget gate decides what information to discard from the previous state by mapping the previous state and current input to a value between 0 and 1, where 1 means retain and 0 means discard. The input gate determines which new information to store in the cell state, using a similar mechanism. The output gate controls which parts of the current cell state to output, again using a value between 0 and 1, considering both previous and current states.

Mathematically, the forward pass of an LSTM cell with a forget gate can be described using vector notation. Let \(x_t\) be the input at time step \(t\), \(h_{t-1}\) the previous hidden state, and \(c_{t-1}\) the previous cell state. The gates are computed as:

- Forget gate: \(f_t = \sigma_g(W_f x_t + U_f h_{t-1} + b_f)\)
- Input gate: \(i_t = \sigma_g(W_i x_t + U_i h_{t-1} + b_i)\)
- Output gate: \(o_t = \sigma_g(W_o x_t + U_o h_{t-1} + b_o)\)
- Candidate cell state: \(\tilde{c}_t = \tanh(W_c x_t + U_c h_{t-1} + b_c)\)
- Cell state update: \(c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t\)
- Hidden state: \(h_t = o_t \odot \tanh(c_t)\)

Here, \(\sigma_g\) is the sigmoid activation function, \(\tanh\) is the hyperbolic tangent, and \(\odot\) denotes element-wise multiplication. The weight matrices \(W_q\) and \(U_q\) (where \(q\) can be \(i\), \(o\), \(f\), or \(c\)) contain input and recurrent connections, respectively.

## Variants and Extensions

Several variants of LSTM have been developed to improve performance or adapt to specific tasks. The most common variant is the LSTM with a forget gate, which is now standard. Other notable variants include:

- **Gated Recurrent Unit (GRU)**: A simplified architecture that combines the input and forget gates into a single update gate, reducing computational complexity.
- **Bidirectional LSTM**: Processes sequences in both forward and backward directions, capturing context from both past and future.
- **Peephole connections**: Allow gates to access the cell state directly, improving timing and precision.

These variants have been widely adopted in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) applications, including [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and [neural-network](https://www.wikiprompt.org/wiki/neural-network)-based systems.

## Applications

LSTM networks have been applied to a wide range of tasks, including classification, data processing, time series analysis, speech recognition, machine translation, speech activity detection, robot control, video games, healthcare, and energy forecasting. In the early 2010s, LSTM-based models achieved state-of-the-art results in speech recognition and machine translation, paving the way for modern [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems.

With the advent of the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture in 2017, LSTM's dominance in sequence processing has diminished, particularly in large-scale [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) models. However, LSTM remains relevant for many applications, especially those with limited computational resources or where sequential processing is inherent, such as in embedded systems and real-time control.

## Limitations

Despite its advantages, LSTM has limitations. It is computationally more expensive than simple RNNs due to the additional gates. It can still suffer from exploding gradients, though techniques like gradient clipping help. Moreover, LSTM processes sequences sequentially, making it difficult to parallelize across timesteps, which is a key reason why transformers have become preferred for large-scale models. Nonetheless, LSTM's ability to model long-term dependencies with relative insensitivity to gap length continues to make it a valuable tool in the [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) toolbox.

---
Source: https://www.wikiprompt.org/wiki/long-short-term-memory
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:33:40.566594+00:00