BERT (Bidirectional Encoder Representations from Transformers) is a language representation model released by Google in October 2018. It was among the first widely influential applications of the Transformer (architecture) architecture to language understanding, and its release is often cited as a turning point that made transformer-based Pretraining the dominant paradigm in Natural language processing.
Architecture and training
BERT uses only the encoder half of the transformer architecture, in contrast to the decoder-based, autoregressive design later used by the GPT series. It is trained using two self-supervised objectives: masked language modeling, in which random words in a sentence are hidden and the model must predict them using context from both directions, and next-sentence prediction, in which the model learns whether one sentence plausibly follows another. This bidirectional training let BERT build representations informed by both preceding and following context simultaneously, an advantage over earlier unidirectional or shallowly bidirectional approaches such as word2vec and ELMo.
BERT was released in multiple sizes, most notably BERT-base (110 million parameters) and BERT-large (340 million parameters), modest by later standards but large for their time, and Google published the pretrained weights openly, allowing researchers to fine-tune the model for specific downstream tasks such as question answering, sentiment analysis, and named entity recognition with comparatively small amounts of task-specific data.
Impact
On release, BERT achieved state-of-the-art results across a wide range of NLP benchmarks, including the GLUE suite and the SQuAD question-answering dataset, often by a substantial margin over prior methods. Its "pretrain then fine-tune" paradigm was rapidly adopted across the field and directly influenced the design of subsequent encoder and encoder-decoder models. Google incorporated BERT into its core search ranking systems starting in 2019, describing it as one of the most significant improvements to search relevance in years, particularly for understanding the intent behind longer, more conversational queries.
Legacy relative to generative models
BERT predates the large language model era defined by GPT-2, GPT-3, and their successors, and its encoder-only, non-generative design means it cannot produce open-ended text the way autoregressive models can. Despite this, BERT and its many derivatives (including RoBERTa, ALBERT, and DistilBERT) remained widely used well into the 2020s for classification, retrieval, and embedding tasks where generation is unnecessary and efficiency matters, and BERT-style encoders continued to serve as components within larger retrieval and semantic search systems, including as building blocks in some retrieval-augmented generation pipelines. Researchers such as Christopher Manning and others in academic NLP have pointed to BERT's release as a pivotal moment demonstrating that large-scale self-supervised pretraining on unlabeled text could outperform methods relying primarily on labeled datasets, a lesson that carried directly into the scaling philosophy behind later generative language models.