In natural language processing, a word embedding is a representation of a word as a real-valued vector that encodes its meaning. The core idea is that words closer in the vector space are expected to be similar in meaning, enabling quantitative analysis of semantic relationships. Word embeddings are typically obtained through language modeling and feature learning techniques, mapping words or phrases from a vocabulary to vectors of numbers. These representations have been shown to boost performance in tasks such as syntactic parsing and sentiment analysis when used as input features.
Methods to generate word embeddings include neural networks, dimensionality reduction on word co-occurrence matrices, probabilistic models, and explicit representation of contexts. The approach has evolved from early semantic space models to modern deep learning techniques, becoming a cornerstone of natural language understanding systems.
Development and History
The underlying idea that "a word is characterized by the company it keeps" was proposed in a 1957 article by John Rupert Firth, rooted in distributional semantics. Early semantic space models, such as the vector space model for information retrieval, represented words in sparse, high-dimensional spaces. Dimensionality reduction via singular value decomposition led to latent semantic analysis in the late 1980s. In 2000, Yoshua Bengio and colleagues introduced neural probabilistic language models that learned distributed representations for words, reducing dimensionality. A 2002 NeurIPS study applied kernel CCA to bilingual corpora, an early example of self-supervised learning for word embeddings.
After foundational work by Bengio and others, most new techniques after about 2005 relied on neural network architectures. In 2013, a team at Google led by Tomas Mikolov created word2vec, a toolkit that trained vector space models faster than previous approaches, widely popularizing word embeddings. Later work included fastText, which represented words partly through character n-grams.
Polysemy and Homonymy
A key limitation of static word embeddings is that words with multiple meanings are conflated into a single vector, failing to handle polysemy and homonymy. For example, "club" could refer to a sandwich, a clubhouse, or a golf club. Multi-sense embeddings address this by assigning different vectors to different senses. Approaches include Multi-Sense Skip-Gram (MSSG), which performs word-sense discrimination and embedding simultaneously, and Non-Parametric Multi-Sense Skip-Gram (NP-MSSG), which allows a variable number of senses per word. Most Suitable Sense Annotation (MSSA) combines lexical databases like WordNet with word sense disambiguation in a self-improving manner.
Multi-sense embeddings improve performance in tasks such as part-of-speech tagging, semantic relation identification, and sentiment analysis. As of the late 2010s, contextually meaningful embeddings like ELMo and BERT provide token-level representations, where each occurrence of a word has its own embedding, better reflecting multi-sense nature.
Biological Sequences
Word embeddings have been extended to biological sequences, such as DNA, RNA, and proteins, for bioinformatics applications. Asgari and colleagues proposed BioVectors, which apply embedding techniques to n-grams in biological sequences, enabling analysis of functional and structural similarities.
Applications and Impact
Word embeddings are widely used in Machine learning and Deep learning models, serving as input representations for tasks like text classification, machine translation, and question answering. They are integral to Large language models and Transformer (architecture) architectures, which build on token-level embeddings. The technique has influenced research at institutions like University of Toronto and Stanford AI Lab, and is implemented in tools from companies such as Google DeepMind and OpenAI.
Future Directions
Research continues on improving embeddings for rare words, cross-lingual alignment, and domain-specific applications. The shift toward contextual embeddings has largely superseded static ones in many applications, but static embeddings remain useful for efficiency and interpretability.