"Attention Is All You Need" is a 2017 research paper on Machine learning authored by eight scientists and engineers working at Google. The paper introduced a new Deep learning architecture known as the Transformer (architecture), based on the attention mechanism proposed in 2014 by Bahdanau et al. The transformer approach has become the main architecture for a wide variety of Artificial intelligence systems, including Large language models. At the time, the research focused on improving sequence-to-sequence techniques for machine translation, but the authors foresaw the technique's potential for other tasks like question answering and what is now known as multimodal Generative AI.
Early experiments with the transformer architecture included English-to-German translation, generating Wikipedia articles on "The Transformer", and parsing. These tests convinced the team that the transformer is a general-purpose language model, not just for translation. As of 2026, the paper has been cited more than 250,000 times, placing it among the top ten most-cited papers of the 21st century. After publication, each of the eight authors left Google to join other companies or found startups.
Background
The authors are Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. All eight were equal contributors; the listed order was randomized. The title references the Beatles song "All You Need Is Love". The name "Transformer" was chosen because Jakob Uszkoreit liked the sound of the word. An early design document was titled "Transformers: Iterative Self-Attention and Processing for Various Tasks" and included an illustration of six characters from the Transformers franchise. The team was named Team Transformer.
Methods introduced
The paper is best known for introducing the transformer architecture, which underlies most modern LLMs. A key reason for its preference is the parallelizability of the architecture over its predecessors, enabling training on GPUs and allowing faster training times and larger models. The paper introduced several mechanisms.
Scaled dot-product attention and self-attention
The paper described scaled dot-product attention as: Attention(Query, Key, Value) = softmax(Query × Key^T / sqrt(d_k)) × Value, where Query, Key, and Value are matrices and d_k is the dimension of the keys (initially set to 64). Using self-attention instead of recurrent neural networks (RNNs) or long short-term memory (LSTM) eliminates recurrence, ensuring parallelizability. In translation, Query and Key matrices typically correspond to source-language embeddings, while Value corresponds to the target language.
Multi-head attention
In self-attention, queries, keys, and values are dynamically generated for each input sequence, allowing the model to focus on different parts. Multi-head attention introduces multiple parallel attention heads, each learning different linear projections of Q, K, and V. This allows the model to capture different aspects of word relationships simultaneously.
Positional encoding
Since the transformer has no recurrence, the paper introduced Positional Encoding to inject information about the position of tokens in the sequence. These encodings are added to the input embeddings, enabling the model to use order information.
Impact and legacy
The transformer architecture has become the foundation for most modern large language models, including those developed by OpenAI, Anthropic, and Google DeepMind. Its parallelizability and scalability have driven advances in Generative AI and other fields. The paper's influence is reflected in its citation count, making it one of the most cited papers of the 21st century.
Aftermath
Following the paper's publication, all eight authors left Google. Some founded startups, such as Aidan Gomez (Cohere) and Noam Shazeer (Character.AI, later returning to Google). Others joined companies like NVIDIA or Intel. Their departure contributed to the spread of transformer-based research across the industry.