Sequence-to-sequence (often abbreviated as seq2seq) is a family of machine learning approaches used for natural language processing and other sequence transduction tasks. The framework transforms one sequence into another sequence, such as converting a sentence in English to French, generating a caption from an image, or producing a summary from a longer text. Originally developed by Lê Viết Quốc, a Vietnamese computer scientist and machine learning pioneer at Google Brain, seq2seq has become foundational in many modern AI systems, including large language models and conversational agents.
The core idea of seq2seq is to use two neural networks: an encoder that processes the input sequence and compresses its information into a fixed-length vector (or a set of context vectors), and a decoder that generates the output sequence step by step from that representation. This architecture was first proposed in 2014 and has since evolved, with the introduction of attention mechanisms and the transformer model, to address limitations such as the bottleneck problem and lack of parallelization.
Historical Roots
The conceptual roots of seq2seq lie in information theory and the noisy channel model of machine translation. As early as 1947, Warren Weaver, a pioneer in machine translation, noted: "One naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography. When I look at an article in Russian, I say: 'This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.'" This viewpoint framed translation as an encode-transmit-decode process, which directly inspired the encoder-decoder structure.
In the early 2010s, researchers began developing neural network-based encoder-decoder models for sequence transduction. The two papers most commonly cited as the originators of seq2seq were both published in 2014. One was by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, working at Google Brain, and the other by Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio, from the University of Montreal and Jacobs University. These papers proposed using recurrent neural networks (RNNs), specifically long short-term memory (LSTM) networks, for both the encoder and decoder.
The Bottleneck Problem and Attention
The original seq2seq models used a fixed-length encoding vector, which created a "bottleneck" problem: for long input sequences, information tended to be lost because it was difficult to compress all necessary details into a single vector. To address this, Bahdanau et al. introduced the attention mechanism in 2014. Their model, called RNNsearch, allowed the decoder to "search" through the source sentence during decoding, effectively emulating the process of looking back at relevant parts of the input. Attention computes a weighted sum of the encoder's hidden states, producing a context vector for each decoding step, which resolved the bottleneck by giving the decoder direct access to all input positions.
The Transformer Revolution
A significant limitation of RNN-based seq2seq models was their difficulty in parallelization, as recurrent processing is inherently sequential. The 2017 publication of the transformer architecture by Vaswani et al. (with key contributors including Jakob Uszkoreit and Lukasz Kaiser) resolved this problem by replacing the recurrent encoder with self-attention transformer blocks (called "encoder blocks") and the recurrent decoder with cross-attention causally-masked transformer blocks ("decoder blocks"). This allowed for much faster training on parallel hardware and led to the development of large-scale models such as large language models and generative AI systems.
Architecture
The seq2seq architecture consists of two main components: the encoder and the decoder.
Encoder
The encoder processes the input sequence, typically a sequence of tokens or words, and captures its essential information. In an RNN-based encoder, this information is stored in the hidden state of the network. With attention, the encoder also produces a set of hidden states for each input position, which are used to compute context vectors. The encoder can be bidirectional, meaning it reads the input from both directions to better capture context.
Decoder
The decoder takes the context vectors and hidden states from the encoder and generates the output sequence autoregressively, producing one element at a time. At each step, it considers previously generated elements, the context vector, and the input information to predict the next element. In models with attention, the context vector and the decoder's hidden state are concatenated to form an attention hidden vector, which is used as input to the next step. The decoder is often causally masked to prevent it from looking ahead at future tokens during training.
Training vs. Prediction
There is a subtle difference between training and prediction in seq2seq models. During training, both the input and output sequences are known, allowing the use of a technique called "teacher forcing." In teacher forcing, the decoder's input at each step is the reference output token from the training data, regardless of what the model predicted. This accelerates convergence and stabilizes training.
During prediction (inference), the reference output is unavailable, so the decoder must use its own previously generated tokens as input for the next step. This can lead to error accumulation, as a mistake early in the sequence can propagate. To mitigate this, techniques such as beam search are often used to explore multiple candidate sequences and select the most likely one.
Applications
Seq2seq models have been applied to a wide range of tasks. In machine translation, they enabled the overhaul of Google Translate into Google Neural Machine Translation in 2016. Other applications include image captioning, where the encoder processes an image (via a convolutional neural network) and the decoder generates a textual description; conversational models, such as chatbots; speech recognition, mapping audio sequences to text; and text summarization, compressing long documents into concise summaries. The architecture also underlies many modern transformer-based models used in artificial intelligence systems.
Priority Dispute
A notable priority dispute surrounds the invention of seq2seq. Tomáš Mikolov, known for developing word2vec and RNNLM, claims to have conceived the idea of using a neural language model on pairs of sentences and generating translation after seeing the first sentence, which he equates with seq2seq machine translation. He stated that he mentioned this idea to Ilya Sutskever and Quoc Le while at Google Brain, but they failed to acknowledge him in their 2014 paper. The paper by Sutskever et al. is widely credited as a key originator, but Mikolov's contributions remain a subject of debate in the research community.
Legacy and Impact
Seq2seq has had a profound impact on the field of machine learning. It introduced the encoder-decoder paradigm that is now standard in sequence transduction tasks. The attention mechanism, initially developed for seq2seq, became a fundamental component of the transformer architecture, which powers modern large language models such as those developed by OpenAI, Anthropic, and Google DeepMind. The principles of seq2seq are also applied in specialized hardware and cloud services, such as AWS Trainium and Azure, which are optimized for training and inference of such models. As of the early 2020s, seq2seq remains a core concept in deep learning curricula and research, though many production systems have moved to transformer-based architectures that build upon its foundations.