# Attention Is All You Need (2017)

Attention Is All You Need is a 2017 machine learning paper by eight Google researchers that introduced the Transformer architecture, a deep learning model based on attention mechanisms. The paper has become foundational for modern AI systems, including large language models, and has been cited over 250,000 times as of 2026.

"Attention Is All You Need" is a 2017 research paper on machine learning authored by eight scientists and engineers working at Google. The paper introduced a new deep learning architecture known as the transformer, based on the attention mechanism proposed in 2014 by Bahdanau et al. The transformer approach it describes has become the main architecture of a wide variety of artificial intelligence systems, including large language models. At the time, the focus of the research was on improving Seq2seq techniques for machine translation, but the authors went further in the paper, foreseeing the technique's potential for other tasks like question answering and what is now known as multimodal generative AI.

Some early examples on which the team tried their Transformer architecture included English-to-German translation, generating Wikipedia articles on "The Transformer", and parsing. These convinced the team that the Transformer is a general-purpose language model, and not just good for translation. As of 2026, the paper has been cited more than 250,000 times, placing it among the top ten most-cited papers of the 21st century. After the paper was published by Google, each of the eight authors left the company to join other companies or to found startups.

## Background

The authors of the paper are Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. All eight authors were "equal contributors" to the paper; the listed order was randomized (according to the paper itself). After the paper, each of the authors left Google to join other companies or to found startups.

The paper's title is a reference to the song "All You Need Is Love" by the Beatles. The name "Transformer" was picked because Jakob Uszkoreit, one of the paper's authors, liked the sound of that word. An early design document was titled "Transformers: Iterative Self-Attention and Processing for Various Tasks", and included an illustration of six characters from the Transformers franchise. The team was named Team Transformer.

## Methods discussed and introduced

The paper is best known for introducing the Transformer architecture, which underlies most modern large language models (LLMs). A key reason why the architecture is preferred by most modern LLMs is the parallelizability of the architecture over its predecessors. This ensures that the operations necessary for training can be accelerated on a GPU, allowing both faster training times and models of bigger sizes to be trained.

The paper introduced the following mechanisms as part of the development of the transformer architecture.

### Scaled dot-product attention and self-attention

The use of the scaled dot-product attention and self-attention mechanism instead of a recurrent neural network (RNN) or long short-term memory (which rely on recurrence) allows for better performance as described in the following paragraph. The paper described the scaled dot-product attention as follows:

Attention(Query, Key, Value) := softmax(Query × Key^T / sqrt(d_k)) × Value

where Query, Key, Value are respectively the query, key, value matrices, and d_k is the dimension of the values. Since the model relies on Query, Key, and Value matrices that come from the same source (i.e., the input sequence or context window), this eliminates the need for RNNs, completely ensuring parallelizability for the architecture. This differs from the original form of the Attention mechanism introduced in 2014. Additionally, the paper discusses the use of an additional scaling factor that was found to be most effective with respect to the dimension of the key vectors (represented as d_k and initially set to 64 within the paper) in the manner shown above. In the specific context of translation, which the paper focused on, the Query and Key matrices are usually represented in embeddings corresponding to the source language, while the Value matrix corresponds to the target language.

### Multi-head attention

In the self-attention mechanism, queries (Q), keys (K), and values (V) are dynamically generated for each input sequence (typically limited by the size of the context window), allowing the model to focus on different parts of the input sequence at different steps. Multi-head attention enhances this process by introducing multiple parallel attention heads. Each attention head learns different linear projections of the Q, K, and V matrices. This allows the model to capture different aspects of the relationships between words in the sequence simultaneously, rather than focusing on a single aspect. By doing so, the model can attend to information from different representation subspaces at different positions, which improves its ability to handle complex language patterns.

### Positional encoding

The paper introduced [positional-encoding](https://www.wikiprompt.org/wiki/positional-encoding) to inject information about the position of tokens in the sequence, since the architecture does not inherently process tokens in order. The authors proposed using sine and cosine functions of different frequencies to encode positions, allowing the model to learn relative positions. This mechanism has become standard in transformer-based models, though later variants have explored learned embeddings or other approaches.

### Other innovations

The paper also discussed the use of [layer-normalization](https://www.wikiprompt.org/wiki/layer-normalization) and [residual connections](https://www.wikiprompt.org/wiki/residual-network) to stabilize training, as well as [dropout](https://www.wikiprompt.org/wiki/dropout) for regularization. These components, combined with the attention mechanisms, formed the basis of the transformer block. The authors trained their models on machine translation tasks, achieving state-of-the-art results with faster training times compared to recurrent models.

## Impact and legacy

Following its publication, the Transformer architecture rapidly became the dominant approach in natural language processing. It underpins major systems developed by organizations such as [openai](https://www.wikiprompt.org/wiki/openai), [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind), and [anthropic](https://www.wikiprompt.org/wiki/anthropic), including large language models like GPT and BERT. The architecture has also been applied beyond text, influencing fields such as computer vision and audio processing. The paper's influence is reflected in its citation count, exceeding 250,000 as of 2026, making it one of the most cited works in computer science.

The departure of all eight authors from Google shortly after publication led to the founding of several AI startups, including Cohere and Inceptive, and contributed to the broader commercialization of transformer-based technologies. The paper's title, referencing the Beatles song, and the team's playful naming conventions have become part of AI lore.

## See also

- [transformer](https://www.wikiprompt.org/wiki/transformer)
- [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)
- [attention-mechanism](https://www.wikiprompt.org/wiki/attention-mechanism)
- [machine-learning](https://www.wikiprompt.org/wiki/machine-learning)


---
Source: https://www.wikiprompt.org/wiki/attention-is-all-you-need-2017
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T02:02:27.254846+00:00