Word2vec is a set of related neural network models, introduced by researchers at Google in 2013, that learn dense vector representations of words from large unlabeled text corpora, capturing aspects of word meaning and usage through vector arithmetic rather than hand-crafted linguistic rules. It was among the first widely adopted Embedding techniques to demonstrate that semantic relationships between words could be recovered directly from word co-occurrence patterns in text.
Method
Word2vec was introduced through two related architectures described by Tomas Mikolov and colleagues at Google, including Jeff Dean and Ilya Sutskever, in papers published in 2013: continuous bag-of-words (CBOW), which predicts a target word from its surrounding context words, and skip-gram, which predicts surrounding context words given a target word. Both are shallow, two-layer Neural network models trained via Self-supervised learning, since the training signal is derived automatically from raw text without human-labeled data, using nearby words as an implicit supervisory signal. The resulting weight matrix from the hidden layer serves as the learned word vectors, typically with a few hundred dimensions.
The "king minus man plus woman" example
Word2vec's most widely cited demonstration is its capacity for vector arithmetic that mirrors semantic and syntactic relationships: subtracting the vector for "man" from "king" and adding the vector for "woman" yields a vector closest to "queen" in the learned embedding space. Similar analogical structure appeared for relationships such as country-capital pairs and verb tense, suggesting that the geometry of the embedding space encoded meaningful, systematic relationships rather than arbitrary clustering, a result that generated substantial excitement in the Natural language processing research community.
Impact on NLP
Word2vec, along with contemporaneous methods such as GloVe, made pretrained word embeddings a standard input layer for downstream NLP models, replacing sparse one-hot word representations across tasks including Machine translation, sentiment analysis, and named entity recognition. Its efficiency, training on billions of words in hours on standard hardware at the time, made distributed word representations practical at a scale that had not been feasible with earlier neural language models.
Successors
Word2vec's context-independent representations, which assign every word a single fixed vector regardless of surrounding context, were superseded by contextual embedding models. The 2018 BERT model and the broader family of models built on the Transformer (architecture) architecture generate different vectors for the same word depending on its sentence context, addressing word2vec's inability to distinguish, for example, "bank" as a riverbank from "bank" as a financial institution. Despite being superseded for most state-of-the-art applications, word2vec remains widely used as a lightweight embedding baseline and as a pedagogical entry point for the broader concept of learned vector representations that now underpin Semantic search and Vector database retrieval systems.