Sequence to sequence (seq2seq) is a neural network architecture, introduced in 2014, that maps an input sequence to an output sequence of potentially different length using a pair of coupled networks: an encoder that compresses the input into a fixed-length representation, and a decoder that generates the output sequence from that representation one element at a time. Seq2seq established the encoder-decoder pattern that became foundational to neural Machine translation and directly motivated the development of the Attention mechanism and later the Transformer (architecture) architecture.
Architecture
In the original formulation, both the encoder and decoder are implemented as Recurrent neural networks, most commonly using LSTM units to mitigate the vanishing gradient problems of simpler recurrent designs. The encoder processes an input sequence, such as a sentence in a source language, one token at a time, updating an internal hidden state, and passes its final hidden state to the decoder as a summary of the entire input. The decoder then generates output tokens one at a time in an Autoregressive model fashion, using its own previous output as input to predict the next token, continuing until it produces a designated end-of-sequence marker.
Origins
The architecture was introduced in the 2014 paper "Sequence to Sequence Learning with Neural Networks" by Ilya Sutskever together with Oriol Vinyals and Quoc Le at Google, demonstrating strong results on English-to-French machine translation using a multi-layer LSTM encoder-decoder. A closely related formulation was published around the same time by Kyunghyun Cho and colleagues, who also introduced the gated recurrent unit as a simplified alternative to the LSTM within the same encoder-decoder framework.
Attention mechanism integration
A key limitation of the original seq2seq design was its reliance on a single fixed-length vector to summarize the entire input sequence, which degraded performance on longer sentences as information was compressed and lost. This bottleneck motivated Dzmitry Bahdanau and colleagues to introduce the Attention mechanism in 2015, allowing the decoder to look back at all encoder hidden states and dynamically weight which parts of the input were most relevant when generating each output token, substantially improving translation quality on long sequences and establishing attention as a standard component of sequence models.
Legacy
Seq2seq with attention became the dominant architecture for neural machine translation systems, including Google Translate's 2016 shift to a fully neural system, before recurrent components were ultimately removed entirely in the 2017 Transformer (architecture) architecture, whose "Attention Is All You Need" title directly references the mechanism that seq2seq research had shown to be so effective. The general encoder-decoder framing seq2seq established, mapping one sequence to another, remains conceptually central to modern Natural language processing, underlying tasks from summarization to code generation, even though the recurrent building blocks of the original architecture have been almost entirely superseded by attention-based Deep learning designs.