# BERT Paper BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Google researchers in October 2018. It uses an encoder-only transformer architecture and self-supervised learning to create contextual text representations, significantly advancing natural language processing. BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning, employing an encoder-only [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. BERT dramatically improved the state of the art for large language models and, as of 2026, remains a common methodological component in natural language processing (NLP) research. BERT is trained through masked token prediction and next sentence prediction, learning contextual, latent representations of tokens in their context, similar to earlier models like ELMo and later GPT-2. It found applications for many NLP tasks, including coreference resolution and polysemy resolution. Its success over ELMo spawned the study of "BERTology," which attempts to interpret what the model learns. ## Architecture BERT is an encoder-only transformer architecture, consisting of four main modules: a tokenizer, an embedding layer, an encoder stack, and a task head. The tokenizer converts English text into a sequence of integers (tokens). The embedding layer converts these tokens into real-valued vectors. The encoder stack, composed of transformer blocks with self-attention but without causal masking, processes these vectors. The task head converts the final representation vectors back into a probability distribution over token types, functioning as an un-embedding layer. The task head is essential for pre-training but often unnecessary for downstream tasks like question answering or sentiment classification. In such cases, it is removed and replaced with a newly initialized module suited to the task, then fine-tuned. The latent vector representations feed directly into this new module, enabling sample-efficient transfer learning. ### Embedding BERT's tokenizer uses WordPiece, a sub-word strategy similar to byte-pair encoding, with a vocabulary size of 30,000. Any token not in the vocabulary is replaced by [UNK]. The embedding layer has three components: token type embeddings, position embeddings, and segment type embeddings. Token type embeddings translate one-hot vectors into dense vectors. Position embeddings use absolute positions, with each dimension being a sinusoidal function of the position. Segment type embeddings use a vocabulary of 0 or 1, indicating whether a token belongs to the first or second text segment, separated by the [SEP] token. These three embedding vectors are added together, then normalized using LayerNorm, producing a 768-dimensional vector for each token in BERTBASE. The vectors pass through 12 transformer encoder blocks and are decoded back to the 30,000-dimensional vocabulary space via a basic affine transformation. ### Architectural family The encoder stack has two free parameters: L (number of layers) and H (hidden size). There are always H/64 self-attention heads, and the feed-forward or filter size is always 4H. Varying L and H yields a family of models. Notation is L/H, so BERTBASE is 12L/768H, BERTLARGE is 24L/1024H, and BERTTINY is 2L/128H. ## Training BERT was pre-trained simultaneously on two tasks: masked language modeling (MLM) and next sentence prediction (NSP). In MLM, BERT ingests a sequence of words where one word may be randomly masked, and it predicts the original word. This teaches bidirectional context, understanding relationships between words from both directions at once. In NSP, BERT predicts whether one sentence logically follows another, which helps with tasks like question answering and document classification. ### Masked language modeling In MLM, 15% of tokens are randomly selected for the masked-prediction task. The selected token is replaced with a [MASK] token with 80% probability, with a random word token with 10% probability, or left unchanged with 10% probability. This partial masking avoids dataset shift, where the distribution of inputs during training differs from inference. If all selected tokens were masked, the model might not handle unmasked inputs well during inference. ## Model sizes and release BERT was originally implemented in English at two sizes: BERTBASE with 110 million parameters and BERTLARGE with 340 million parameters. Both were trained on the Toronto BookCorpus (800 million words) and English Wikipedia (2,500 million words). The weights were released on GitHub. On March 11, 2020, 24 smaller models were released, the smallest being BERTTINY with just 4 million parameters. These smaller models enabled broader experimentation and deployment in resource-constrained settings. ## Impact and legacy BERT's introduction marked a significant shift in NLP, moving from unidirectional models to bidirectional context understanding. Its pre-training and fine-tuning paradigm became a standard approach, influencing subsequent models like [GPT-2](https://www.wikiprompt.org/wiki/gpt-2) and the broader development of [large language model](https://www.wikiprompt.org/wiki/large-language-model)s. BERT's architecture and training methods have been widely adopted and adapted across the field, cementing its place as a foundational technique in modern [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence). --- Source: https://www.wikiprompt.org/wiki/bert-paper-2018 License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-12T16:21:41.050171+00:00