The transformer is a neural network architecture, introduced in 2017, that processes sequential data such as text using Attention mechanisms rather than the step-by-step recurrence used by earlier models, allowing an entire sequence to be processed in parallel. It has become the foundation of nearly all modern Large language models and has been adapted for images, audio, and other data types as well.
History
Before the transformer, sequence modeling for tasks such as machine translation relied primarily on Recurrent neural networks, including Seq2seq encoder-decoder models, which process input tokens one at a time and therefore struggle to parallelize and to retain information over long sequences. The transformer was introduced in the 2017 paper Attention Is All You Need by a team of researchers including Ashish Vaswani and Noam Shazeer, who proposed dispensing with recurrence entirely in favor of self-attention. All eight of the paper's authors subsequently left the company where the work was done to found or join other AI companies, an unusual outcome for a single research paper.
Architecture
A transformer processes an input sequence, first converted into vector representations through Tokenization and embedding, by repeatedly applying self-attention layers, which let each position in the sequence weigh the relevance of every other position, alongside simple feed-forward layers. Because self-attention has no inherent notion of sequence order, transformers add positional information to their input representations. Multiple attention "heads" running in parallel allow the model to capture different types of relationships simultaneously. The original architecture used a separate encoder and decoder; most modern large language models use a decoder-only variant, which generates text one token at a time based on all previously generated tokens.
Impact
The transformer's parallelizability allowed models to be trained on far larger datasets and with far more parameters than recurrent architectures permitted, directly enabling the scale-driven progress of models from BERT in 2018 through GPT-3 and beyond. Its self-attention mechanism, applicable in principle to any data that can be represented as a sequence, was also adapted for images, leading to vision transformers, and for combined text-and-image systems used in modern Vision-language models.
Variants and alternatives
The computational cost of self-attention grows quadratically with sequence length, motivating both efficiency-focused variants and entirely different architectures. Mixture of experts models activate only a subset of a transformer's parameters for any given input, reducing compute cost relative to model size. State space models, such as the Mamba architecture introduced in 2023, offer an alternative that scales linearly with sequence length rather than quadratically, though as of the mid-2020s they had not displaced the transformer as the dominant architecture for frontier language models.