Cross-attention is a mechanism in Machine learning and Deep learning that computes attention weights between elements of two distinct sequences, typically a source and a target. It is a core component of the Transformer architecture, where it enables the decoder to focus on relevant parts of the encoder's output when generating each token. Unlike self-attention, which relates positions within a single sequence, cross-attention relates positions across sequences, making it essential for tasks such as machine translation, text summarization, and image captioning.
The concept emerged from earlier attention mechanisms developed for recurrent neural networks (RNNs), which suffered from difficulty in retaining information from distant parts of a sequence. Attention, inspired by human visual and cognitive focus, allowed models to weigh the importance of each input element directly. The breakthrough came with the introduction of the Transformer in 2017, which replaced recurrence with parallelizable attention, and cross-attention became a key ingredient in encoder-decoder models like BERT, T5, and generative pre-trained transformers (GPT).
History
The roots of cross-attention trace back to the early 1990s, when fast weight programmers, or fast weight controllers, proposed a mechanism where a "slow" neural network outputs the "fast" weights of another network through outer products. This idea, later renamed linearized self-attention, laid conceptual groundwork. In 2014, Bahdanau-style attention (additive attention) was introduced for RNN-based machine translation, allowing the decoder to align with encoder hidden states. Luong-style attention (multiplicative attention) followed in 2015, offering a more efficient scoring function.
The major breakthrough came with self-attention, where each element in a sequence attends to all others, enabling global dependency capture. This idea was central to the Transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by researchers at Google and the University of Toronto. The Transformer replaced recurrence with attention mechanisms, and cross-attention became a standard component in encoder-decoder models, forming the foundation for models like BERT, T5, and GPT.
Mechanism
In a typical encoder-decoder Transformer, the encoder processes the source sequence (e.g., a sentence in English) and produces a set of hidden representations. The decoder generates the target sequence (e.g., French) one token at a time. At each decoding step, the decoder uses cross-attention to compute a weighted sum of the encoder's outputs, where the weights reflect the relevance of each source token to the current target token.
The cross-attention operation involves three matrices: queries (Q) derived from the decoder's previous output, keys (K) and values (V) derived from the encoder's output. The attention weights are computed as the softmax of the scaled dot products between Q and K, which are then used to weight V. This allows the decoder to selectively focus on relevant source information, similar to how a human translator might look back at specific words in the source sentence.
Interpreting Attention Weights
Cross-attention weights can be visualized as an alignment matrix, showing which source tokens are most influential for each target token. In translation, alignment is the process of matching words from the source sentence to the translated sentence. Networks that perform verbatim translation without regard to word order would show highest scores along the diagonal of the matrix. Off-diagonal dominance indicates more nuanced alignment.
For example, translating "I love you" to French: on the first decoder pass, 94% of attention weight is on "I", producing "je"; on the second pass, 88% on "you", producing "t'"; on the third pass, 95% on "love", producing "aime". This yields an alignment matrix where "love" aligns with "aime". Sometimes alignment is many-to-many, such as "look it up" corresponding to "cherchez-le". Soft attention weights, which distribute weight across multiple tokens, work better than hard attention (setting one weight to 1 and others to 0), because a weighted sum of hidden vectors often provides a richer context than selecting a single best vector.
Variants
Cross-attention has several variants, differing in how the attention scores are computed:
- Additive attention (Bahdanau-style): Uses a feed-forward network to compute alignment scores.
- Multiplicative attention (Luong-style): Uses dot products between query and key vectors.
- Positional attention: Incorporates positional encodings to account for token order.
- Factorized positional attention: Reduces complexity by factoring attention across dimensions.
For convolutional neural networks, attention mechanisms can operate on spatial dimensions (spatial attention), channel dimensions (channel attention), or combinations. These variants recombine encoder-side inputs to redistribute effects to each target output, often using a correlation-style matrix of dot products for re-weighting.
Optimizations
Flash Attention
The size of the attention matrix grows quadratically with the number of input tokens, requiring significant GPU memory for long sequences. Flash attention, introduced in 2022, reduces memory needs and increases efficiency without sacrificing accuracy. It partitions the attention computation into smaller blocks that fit into the GPU's faster on-chip memory, reducing the need to store large intermediate matrices and lowering memory usage while improving computational efficiency.
FlexAttention
FlexAttention is an attention kernel developed by Meta that allows users to modify attention scores prior to softmax and dynamically chooses the optimal attention algorithm. This flexibility enables custom attention patterns, such as sparsity or masking, without sacrificing performance.
Applications
Cross-attention is widely used in natural language processing, computer vision, and speech recognition. In NLP, it improves context understanding in tasks like question answering and summarization. In vision, visual attention helps models focus on relevant image regions, enhancing object detection and image captioning. In speech recognition, cross-attention aligns acoustic features with text output.
Attention Maps as Explanations for Vision Transformers
From the original vision transformer (ViT) paper, visualizing attention scores as heat maps (saliency maps or attention maps) has become a routine way to inspect the decision-making process of ViT models. One can compute attention maps for any attention head at any layer, with deeper layers showing more semantically meaningful visualizations. Attention rollout is a recursive algorithm that combines attention scores across all layers by computing the dot product of successive attention maps.
Because vision transformers are typically trained in a self-supervised manner, attention maps are generally not class-sensitive. When a classification head is attached to the ViT backbone, class-discriminative attention maps (CDAM) combine attention maps and gradients with respect to the class token. Some class-sensitive interpretability methods originally developed for convolutional neural networks can be adapted to ViTs, providing insights into which image regions influence predictions.