Self-attention is a mechanism in machine learning where each element in a sequence attends to all other elements, computing attention weights based on their relationships. Unlike earlier attention mechanisms that operated between encoder and decoder sequences, self-attention derives queries, keys, and values from the same input sequence, allowing the model to capture global dependencies directly. This concept is central to the Transformer (architecture) architecture, which replaced recurrence with attention mechanisms, and became the foundation for models like BERT, T5, and generative pre-trained transformers (GPT).
In self-attention, each token in the input sequence is transformed into three vectors: a query, a key, and a value. The attention weight between two tokens is computed as the dot product of the query of one token with the key of the other, typically scaled and passed through a softmax function. These weights, often called "soft" weights, exist only in the forward pass and change with every input step, unlike "hard" weights computed during training. The output for each token is a weighted sum of the values, where the weights reflect the relevance of other tokens to the current token.
History
Attention mechanisms were developed to address weaknesses in recurrent neural networks (RNNs), which favored information from later words in a sentence, attenuating earlier context. Early attention designs grafted an attention mechanism onto an encoder-decoder RNN for machine translation, as described by Bahdanau et al. in 2014. However, the major breakthrough came with self-attention, where each element in the input sequence attends to all others, enabling the model to capture global dependencies. This idea was central to the transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by Jakob Uszkoreit, Lukasz Kaiser, and colleagues at Google. Transformers removed the slower sequential RNN and relied on faster parallel attention, leading to models like BERT, T5, and GPT. Additional surveys of attention mechanisms in deep learning are provided by Niu et al. and Soydaner.
Mechanism
Self-attention operates on a sequence of token embeddings, each represented as a vector. For each token, the model computes a query vector, a key vector, and a value vector through learned linear transformations. The attention score between token i and token j is calculated as the dot product of query_i and key_j, often divided by the square root of the dimension to stabilize gradients. These scores are passed through a softmax to produce attention weights that sum to one for each query. The output for token i is the weighted sum of all value vectors, using these weights.
This mechanism allows each token to directly access any part of the sequence, regardless of distance, overcoming the sequential bottleneck of RNNs. In practice, transformers use multi-head attention, where multiple sets of query, key, and value projections run in parallel, allowing the model to attend to different aspects of the sequence simultaneously.
Interpreting Attention Weights
In translation, alignment refers to matching words from the source sentence to words in the translated sentence. Networks that perform verbatim translation without regard to word order would show highest attention scores along the diagonal of the alignment matrix. Off-diagonal dominance indicates more nuanced alignment. For example, translating "I love you" to French: on the first decoder pass, 94% of attention weight is on "I", producing "je"; on the second pass, 88% on "you", producing "t'"; on the third pass, 95% on "love", producing "aime". This yields an alignment matrix where "love" aligns with "aime".
Sometimes alignment is many-to-many, such as "look it up" corresponding to "cherchez-le". Soft attention weights, which produce a weighted sum of hidden vectors, work better than hard attention (setting one weight to 1 and others to 0), as there may not be a single best hidden vector.
Variants
Many variants of attention implement soft weights. Early forms include fast weight programmers or fast weight controllers (1992), where a "slow" neural network outputs the "fast" weights of another network through outer products, later renamed "linearized self-attention". Bahdanau-style attention, also known as additive attention, and Luong-style attention, or multiplicative attention, are common in RNN-based systems. Early attention mechanisms similar to modern self-attention were proposed using RNNs, but the highly parallelizable self-attention was introduced in 2017 and used in the transformer. Other variants include positional attention and factorized positional attention. For convolutional neural networks, attention can operate on spatial dimensions, channel dimensions, or combinations. These variants recombine encoder-side inputs to redistribute effects to each target output, often using a correlation-style matrix of dot products for re-weighting.
Optimizations
The size of the attention matrix grows quadratically with the number of tokens, requiring significant GPU memory for long sequences. Flash attention, introduced in 2022, reduces memory needs and increases efficiency without sacrificing accuracy by partitioning attention computation into blocks that fit into the GPU's faster on-chip memory, reducing intermediate storage. FlexAttention, developed by Meta, allows users to modify attention scores before softmax and dynamically chooses the optimal attention algorithm.
Applications
Attention is widely used in natural language processing, computer vision, and speech recognition. In NLP, it improves context understanding in tasks like question answering and summarization. In vision, visual attention helps models focus on relevant image regions, enhancing object detection and image captioning. Self-attention is a core component of large language models like GPT, which use it to process and generate text.
Attention Maps as Explanations for Vision Transformers
From the original vision transformer (ViT) paper, visualizing attention scores as heat maps, called saliency maps or attention maps, has become a routine way to inspect the decision-making process of ViT models. One can compute attention maps for any attention head at any layer, with deeper layers showing more semantically meaningful visualizations. Attention rollout is a recursive algorithm that combines attention scores across all layers by computing the dot product of successive attention maps. Because vision transformers are typically trained in a self-supervised manner, attention maps are generally not class-sensitive. When a classification head is attached, class-discriminative attention maps (CDAM) combine attention maps and gradients with respect to the class [CLS] token. Some class-sensitive interpretability methods originally developed for convolutional neural networks can also be adapted.