BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning, employing an encoder-only transformer architecture. BERT dramatically improved the state of the art for large language models and, as of 2026, remains a common methodological component in natural language processing (NLP) research.
BERT is trained using masked token prediction and next sentence prediction, enabling it to learn contextual, latent representations of tokens in their context, similar to ELMo and GPT-2. It found applications in many NLP tasks, such as coreference resolution and polysemy resolution. BERT improved on ELMo and spawned the study of "BERTology," which attempts to interpret what is learned by BERT.
Architecture
BERT is an "encoder-only" transformer architecture. At a high level, BERT consists of four modules: a tokenizer, an embedding module, an encoder stack, and a task head. The tokenizer converts English text into a sequence of integers (tokens). The embedding module converts tokens into real-valued vectors. The encoder stack consists of transformer blocks with self-attention but without causal masking. The task head converts final representation vectors into a probability distribution over token types, acting as an un-embedding layer.
The task head is necessary for pre-training but often unnecessary for downstream tasks such as question answering or sentiment classification. For such tasks, the task head is removed and replaced with a newly initialized module suited to the task, followed by fine-tuning. The latent vector representation is directly fed into this new module, enabling sample-efficient transfer learning.
Embedding
The tokenizer of BERT is WordPiece, a sub-word strategy similar to byte-pair encoding. Its vocabulary size is 30,000, and any token not appearing in the vocabulary is replaced by [UNK]. The embedding layer contains three components: token type embeddings, position embeddings, and segment type embeddings. Token type embeddings translate a one-hot vector into a dense vector based on token type. Position embeddings use absolute positions, with each dimension consisting of a sinusoidal function of the position. Segment type embeddings use a vocabulary of 0 or 1, indicating whether a token belongs to the first or second text segment (tokens after the [SEP] special token are type-1). The three embedding vectors are added together, then normalized using LayerNorm, outputting a 768-dimensional vector for each token. These vectors pass through 12 transformer encoder blocks and are decoded back to the 30,000-dimensional vocabulary space via an affine transformation.
Architectural family
The encoder stack has two free parameters: L (number of layers) and H (hidden size). There are always H/64 self-attention heads, and the feed-forward/filter size is always 4H. Varying these parameters yields a family of BERT models. Notation is L/H: BERTBASE is 12L/768H, BERTLARGE is 24L/1024H, and BERTTINY is 2L/128H.
Training
Pre-training
BERT was pre-trained simultaneously on two tasks: masked language modeling (MLM) and next sentence prediction (NSP). In MLM, BERT ingests a sequence of words where one word may be randomly masked, and it predicts the original word. This helps BERT learn bidirectional context, understanding relationships between words from both directions simultaneously. In NSP, BERT is trained to predict whether one sentence logically follows another, which is important for tasks like question answering or document classification.
#### Masked language modeling
In masked language modeling, 15% of tokens are randomly selected for the masked-prediction task. The selected token is replaced with a [MASK] token with probability 80%, replaced with a random word token with probability 10%, and left unchanged with probability 10%. This strategy avoids the dataset shift problem, where the distribution of inputs during training differs from that during inference.
Release and Impact
BERT was originally implemented in English at two model sizes: BERTBASE (110 million parameters) and BERTLARGE (340 million parameters). Both were trained on the Toronto BookCorpus (800 million words) and English Wikipedia (2,500 million words). The weights were released on GitHub, making the model widely accessible. On March 11, 2020, 24 smaller models were released, the smallest being BERTTINY with just 4 million parameters.
BERT's release had a profound impact on the field of natural language processing. It set new benchmarks on a wide range of tasks, including question answering, sentiment analysis, and named entity recognition. Its success popularized the pre-training and fine-tuning paradigm, influencing subsequent models like GPT-2 and RoBERTa. BERT also inspired research into interpretability, with "BERTology" emerging as a field to understand the internal representations learned by the model.
Applications and Legacy
BERT's contextual embeddings have been applied to numerous tasks, including coreference resolution, polysemy resolution, and machine translation. Its architecture has been adapted for various languages and domains, and it remains a baseline for many NLP systems. As of 2026, BERT continues to be a foundational tool in NLP research and industry applications, despite the rise of larger models like GPT-3 and T5. Its influence is evident in the widespread adoption of transformer-based models and the development of large language models.
BERT's release also spurred advances in efficient model variants, such as DistilBERT and ALBERT, which aim to reduce computational costs while retaining performance. The model's design principles, including masked language modeling and next sentence prediction, have been incorporated into many subsequent architectures, cementing BERT's legacy as a cornerstone of modern NLP.