# Transformer Paper (2017)

The 2017 paper 'Attention Is All You Need' by eight Google researchers introduced the transformer architecture, which relies on self-attention and has become the foundation for most modern large language models.

"Attention Is All You Need" is a 2017 research paper on [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) authored by eight scientists and engineers working at Google. The paper introduced a new [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) architecture known as the [transformer](https://www.wikiprompt.org/wiki/transformer), based on the attention mechanism proposed in 2014 by Bahdanau et al. The transformer approach has become the main architecture for a wide variety of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) systems, including [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s. At the time, the research focused on improving sequence-to-sequence techniques for machine translation, but the authors foresaw the technique's potential for other tasks like question answering and what is now known as multimodal [generative-ai](https://www.wikiprompt.org/wiki/generative-ai).

Early experiments with the transformer architecture included English-to-German translation, generating Wikipedia articles on "The Transformer", and parsing. These tests convinced the team that the transformer is a general-purpose language model, not just for translation. As of 2026, the paper has been cited more than 250,000 times, placing it among the top ten most-cited papers of the 21st century. After publication, each of the eight authors left Google to join other companies or found startups.

## Background

The authors are Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. All eight were equal contributors; the listed order was randomized. The title references the Beatles song "All You Need Is Love". The name "Transformer" was chosen because Jakob Uszkoreit liked the sound of the word. An early design document was titled "Transformers: Iterative Self-Attention and Processing for Various Tasks" and included an illustration of six characters from the Transformers franchise. The team was named Team Transformer.

## Methods introduced

The paper is best known for introducing the transformer architecture, which underlies most modern LLMs. A key reason for its preference is the parallelizability of the architecture over its predecessors, enabling training on GPUs and allowing faster training times and larger models. The paper introduced several mechanisms.

### Scaled dot-product attention and self-attention

The paper described scaled dot-product attention as: Attention(Query, Key, Value) = softmax(Query × Key^T / sqrt(d_k)) × Value, where Query, Key, and Value are matrices and d_k is the dimension of the keys (initially set to 64). Using self-attention instead of recurrent neural networks (RNNs) or long short-term memory (LSTM) eliminates recurrence, ensuring parallelizability. In translation, Query and Key matrices typically correspond to source-language embeddings, while Value corresponds to the target language.

### Multi-head attention

In self-attention, queries, keys, and values are dynamically generated for each input sequence, allowing the model to focus on different parts. Multi-head attention introduces multiple parallel attention heads, each learning different linear projections of Q, K, and V. This allows the model to capture different aspects of word relationships simultaneously.

### Positional encoding

Since the transformer has no recurrence, the paper introduced [positional-encoding](https://www.wikiprompt.org/wiki/positional-encoding) to inject information about the position of tokens in the sequence. These encodings are added to the input embeddings, enabling the model to use order information.

## Impact and legacy

The transformer architecture has become the foundation for most modern large language models, including those developed by [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). Its parallelizability and scalability have driven advances in [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) and other fields. The paper's influence is reflected in its citation count, making it one of the most cited papers of the 21st century.

## Aftermath

Following the paper's publication, all eight authors left Google. Some founded startups, such as Aidan Gomez (Cohere) and Noam Shazeer (Character.AI, later returning to Google). Others joined companies like [nvidia](https://www.wikiprompt.org/wiki/nvidia) or [intel](https://www.wikiprompt.org/wiki/intel). Their departure contributed to the spread of transformer-based research across the industry.

## See also

- [neural-network](https://www.wikiprompt.org/wiki/neural-network)
- [sequence-to-sequence](https://www.wikiprompt.org/wiki/sequence-to-sequence)
- [multi-head-attention](https://www.wikiprompt.org/wiki/multi-head-attention)

---
Source: https://www.wikiprompt.org/wiki/transformer-paper-2017
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:21:55.766085+00:00