# Transformer Architecture

The transformer is a deep learning architecture based on multi-head self-attention, introduced in 2017, that processes tokens in parallel and underpins modern large language models.

The transformer is a family of [artificial neural network](https://www.wikiprompt.org/wiki/neural-network) architectures used in [deep learning](https://www.wikiprompt.org/wiki/deep-learning), based on the multi-head attention mechanism. Input data such as text, images, or audio is converted into a sequence of numerical representations called tokens, each mapped to a vector via a word embedding table. At each layer, every token is contextualized within the context window using parallel multi-head attention, amplifying important tokens and diminishing less relevant ones. Because self-attention alone is permutation-invariant, transformers inject positional information through positional encodings or learned embeddings, so token order affects the output.

Transformers have no recurrent units, requiring less training time than earlier recurrent architectures like LSTM. Modern designs are grouped into encoder-only, decoder-only, and encoder-decoder variants, optimized for representation learning, autoregressive generation, or conditional sequence-to-sequence tasks. The original transformer was proposed in the 2017 paper "Attention Is All You Need" by researchers at [Google](https://www.wikiprompt.org/wiki/google-deepmind). It has since become the basis of [large language models](https://www.wikiprompt.org/wiki/large-language-model) (LLMs) and found applications in natural language processing, computer vision, reinforcement learning, audio, multimodal learning, robotics, and chess.

## History

### Predecessors
For many years, sequence modeling used plain recurrent neural networks (RNNs), such as the Elman network (1990). In theory, information from one token could propagate arbitrarily far, but the vanishing-gradient problem left long sequences without precise extractable information. A key breakthrough was LSTM, described in a 1995 technical report and formally published in 1997, which introduced gating mechanisms to mitigate vanishing gradients. LSTM used multiplicative gating units, conceptually distinct from additive attention. It became the standard for long sequence modeling until 2017, but RNNs process tokens sequentially, preventing parallelization. The linearly scaling fast weight controller (1992) learned to compute weight matrices based on input, equivalent to an unnormalized linear transformer.

### Attention with seq2seq
Encoder-decoder sequence transduction developed in the early 2010s, with two concurrent 2014 papers. A 380M-parameter model used two LSTMs for machine translation: an encoder LSTM turned a token sequence into a vector, and a decoder LSTM converted it to output. Another 130M-parameter model used gated recurrent units (GRUs), later shown to be comparable to LSTMs. These early seq2seq models lacked attention, relying on a fixed-size state vector after the last word, which poorly preserved long input information. Reversing the input sentence improved translation, highlighting the bottleneck. The RNN search model introduced attention to seq2seq, allowing better handling of long dependencies. In 2016, Google Translate was revamped to Google Neural Machine Translation, using an 8-layer bidirectional LSTM seq2seq model, outperforming statistical methods.

### Parallelizing attention
Seq2seq models with attention still suffered from recurrent networks' difficulty in parallelization, limiting GPU acceleration. In 2016, decomposable attention applied self-attention to feedforward networks, achieving state-of-the-art textual entailment with fewer parameters. Author [Jakob Uszkoreit](https://www.wikiprompt.org/wiki/jakob-uszkoreit) suspected attention without recurrence could suffice for translation, leading to the title "Attention Is All You Need."

## Architecture

The transformer architecture consists of an encoder and a decoder, each composed of layers with multi-head self-attention and position-wise feedforward networks. The encoder processes the input sequence in parallel, while the decoder generates output autoregressively, using masked self-attention to prevent future token leakage. [Multi-head attention](https://www.wikiprompt.org/wiki/multi-head-attention) runs multiple attention mechanisms in parallel, allowing the model to focus on different representation subspaces. [Positional encodings](https://www.wikiprompt.org/wiki/positional-encoding) are added to token embeddings to capture order. [Layer normalization](https://www.wikiprompt.org/wiki/layer-normalization) and residual connections stabilize training. The decoder also uses [cross-attention](https://www.wikiprompt.org/wiki/cross-attention) to attend to encoder outputs.

## Variants

### Encoder-only
Encoder-only models, like BERT, are optimized for representation learning, producing contextualized token embeddings useful for classification and understanding tasks. They use bidirectional attention, seeing both left and right context.

### Decoder-only
Decoder-only models, such as GPT, are designed for autoregressive generation, predicting the next token given previous ones. They use masked self-attention and are the basis of most modern LLMs, trained on large text corpora.

### Encoder-decoder
Encoder-decoder models, like the original transformer, handle conditional sequence-to-sequence tasks such as translation and summarization. The encoder processes the source, and the decoder generates the target.

## Applications

Transformers are used in [generative AI](https://www.wikiprompt.org/wiki/generative-ai) systems, including LLMs like GPT and BERT, which have been adopted by companies such as [OpenAI](https://www.wikiprompt.org/wiki/openai), [Anthropic](https://www.wikiprompt.org/wiki/anthropic), and [Google DeepMind](https://www.wikiprompt.org/wiki/google-deepmind). They power natural language processing, computer vision (vision transformers), reinforcement learning, audio processing, multimodal learning, robotics, and even chess-playing programs. Their scalability has driven advances in hardware, with specialized chips like [AWS Trainium](https://www.wikiprompt.org/wiki/aws-trainium) and [Groq](https://www.wikiprompt.org/wiki/groq)'s inference processors.

## Impact

Transformers have revolutionized [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence), enabling pre-trained systems that can be fine-tuned for diverse tasks. They have reduced training time compared to RNNs and allowed unprecedented model sizes, leading to breakthroughs in language understanding and generation. The architecture's flexibility has made it the default choice for many AI applications, influencing research and industry alike.

## Limitations and Future

Despite their success, transformers require quadratic computation in context length, making long sequences expensive. Researchers are exploring linear attention mechanisms and alternative architectures to address this. As of 2025, transformers remain dominant, but ongoing innovations may lead to more efficient designs.

---
Source: https://www.wikiprompt.org/wiki/transformer-architecture
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T22:27:03.890617+00:00