An attention mechanism is a neural network component that computes a weighted combination of elements in its input based on their relevance to a given context, allowing a model to dynamically focus on the most pertinent parts of that input rather than treating every element equally. Attention is the central computational idea behind the Transformer (architecture) architecture and, by extension, most modern Large language models.
History
Attention was introduced in the context of Seq2seq machine translation models around 2014 and 2015, as a way to let a Recurrent neural network-based decoder look back at all of the encoder's hidden states when producing each output word, rather than relying on a single fixed-length summary vector of the entire input sentence. This addressed a specific weakness of earlier encoder-decoder models, which tended to lose information from long input sequences. The 2017 paper Attention Is All You Need, led by Ashish Vaswani and colleagues, showed that a model built entirely out of attention layers, without any recurrence at all, could outperform recurrent architectures while being far more parallelizable to train, giving rise to the transformer.
Self-attention
The key innovation used in transformers is self-attention, in which every element of a sequence attends to every other element of the same sequence, rather than a decoder attending to a separate encoder. For each position, the model computes three vectors, commonly called query, key, and value, derived from that position's Embedding. The similarity between a position's query and every other position's key determines how much weight that other position's value contributes to the output at the current position. This allows a model processing a sentence to directly relate words that are far apart, such as a pronoun and the noun it refers to, without the information having to pass through many intermediate steps as it would in a recurrent network.
Variants
Transformers typically use multi-head attention, running several attention computations in parallel with different learned projections, allowing the model to capture different kinds of relationships, such as syntactic structure and topical similarity, simultaneously. Cross-attention, in which one sequence attends to a separate sequence, is used in encoder-decoder transformers and in some multimodal systems, for example letting a decoder generating a caption attend to the encoded regions of an image.
Practical significance
Attention's computational cost grows quadratically with the length of the input sequence, since every position must be compared with every other position, which has made processing very long inputs expensive and motivated both efficiency-focused attention variants and alternative architectures such as State space models. Despite this cost, attention's ability to directly relate distant elements of an input, combined with its parallelizability compared to recurrence, made it the mechanism that unlocked the scale on which large language models are trained, including the growth of the Context window that determines how much text a model can attend to in a single interaction.