# Subword Tokenization Subword tokenization is a text preprocessing technique that splits words into smaller units, balancing vocabulary size and meaning. It underpins modern language models, with methods like Byte-Pair Encoding (BPE) and WordPiece. Subword tokenization is a technique in natural language processing that converts raw text into a sequence of tokens, where each token can be a whole word, a part of a word, or even a single character. It sits between word-level tokenization, which treats each word as an indivisible unit, and character-level tokenization, which processes text as individual characters. By using subword units, tokenizers can handle large vocabularies efficiently, represent rare or unseen words, and preserve meaningful morphological information. This approach is fundamental to modern [large language models](https://www.wikiprompt.org/wiki/large-language-model) and other [neural network](https://www.wikiprompt.org/wiki/neural-network) architectures that process text. The core idea is to break words into smaller pieces that appear frequently in the training corpus. For example, the word "unhappiness" might be split into "un", "happi", and "ness", or "unhappiness" could be kept as a single token if it is common enough. This flexibility allows the model to understand novel words by combining known subword units, while also keeping the vocabulary size manageable. Subword tokenization has been widely adopted since the mid-2010s and is a key component of [transformer](https://www.wikiprompt.org/wiki/transformer)-based models. ## History and Development The concept of subword tokenization emerged from the need to handle open-vocabulary problems in machine translation and language modeling. Early approaches in the 1990s and 2000s used morphological analysis, but these were often language-specific and computationally expensive. A major breakthrough came with the introduction of Byte-Pair Encoding (BPE) for text compression, adapted for tokenization by Rico Sennrich and colleagues in 2016. Their paper "Neural Machine Translation of Rare Words with Subword Units" demonstrated that BPE could effectively handle rare words in [sequence-to-sequence](https://www.wikiprompt.org/wiki/sequence-to-sequence) models. Around the same time, the WordPiece model was developed at Google, initially for speech recognition and later popularized by [BERT](https://www.wikiprompt.org/wiki/bert). Another variant, Unigram Language Model, was introduced by Taku Kudo in 2018 and used in models like [ALBERT](https://www.wikiprompt.org/wiki/albert) and T5. These methods differ in how they select the subword units, but all share the goal of finding an optimal segmentation of text. ## Methods and Algorithms ### Byte-Pair Encoding (BPE) BPE is a data compression algorithm that iteratively merges the most frequent pair of consecutive symbols (initially characters) into a new symbol. For tokenization, the process starts with a corpus of text split into characters, then repeatedly counts adjacent pairs and merges the most frequent pair until a target vocabulary size is reached. The resulting vocabulary contains characters, subwords, and whole words. BPE is used in many models, including GPT series from [openai](https://www.wikiprompt.org/wiki/openai) and [LLaMA](https://www.wikiprompt.org/wiki/llama) from Meta. ### WordPiece WordPiece is similar to BPE but uses a likelihood-based criterion for merging. It builds a vocabulary by starting with individual characters and then adding new tokens that maximize the likelihood of the training data. This approach tends to produce more linguistically meaningful subwords. WordPiece is used in [BERT](https://www.wikiprompt.org/wiki/bert), [ELECTRA](https://www.wikiprompt.org/wiki/electra), and other models from Google. ### Unigram Language Model Unigram tokenization treats each subword as an independent unit and uses a probabilistic model to choose the best segmentation for each word. It starts with a large vocabulary of possible subwords and then prunes it based on how much each token contributes to the overall likelihood. This method allows for multiple segmentations and is used in [SentencePiece](https://www.wikiprompt.org/wiki/sentencepiece) implementations. ## Role in Modern Language Models Subword tokenization is a critical preprocessing step for almost all contemporary [large language models](https://www.wikiprompt.org/wiki/large-language-model). Models like [GPT-4](https://www.wikiprompt.org/wiki/gpt-4), [Claude](https://www.wikiprompt.org/wiki/claude), and [Gemini](https://www.wikiprompt.org/wiki/gemini) rely on tokenizers that convert text into subword tokens before feeding them into the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. The choice of tokenizer affects model performance, computational efficiency, and the ability to handle multiple languages. For example, a tokenizer with a larger vocabulary can represent more words as single tokens, reducing the sequence length and computational cost, but it may also increase the model's memory footprint. Conversely, a smaller vocabulary forces more words to be split, which can lead to longer sequences and potential loss of meaning. Tokenizers are typically trained on large corpora and are kept fixed during model training. Subword tokenization also enables cross-lingual transfer. By sharing subword units across languages, models can better handle code-switching and rare words in low-resource languages. This is particularly important for multilingual models like mBERT and [XLM-RoBERTa](https://www.wikiprompt.org/wiki/xlm-roberta). ## Challenges and Future Directions Despite its widespread use, subword tokenization has limitations. One issue is that it can produce arbitrary segmentations that do not align with morphological boundaries, making it harder for models to learn word structure. Another challenge is the handling of out-of-vocabulary words, especially in specialized domains like medicine or law, where new terms appear frequently. Some researchers have proposed character-level or byte-level tokenization to avoid these issues, but these approaches often require longer sequences and more computation. Recent developments include adaptive tokenization, where the tokenizer can be adjusted during training, and end-to-end models that learn to segment text without a fixed tokenizer. However, as of 2025, subword tokenization remains the standard in most production systems, and ongoing research aims to improve its efficiency and linguistic quality. ## See Also - [tokenization](https://www.wikiprompt.org/wiki/tokenization) - [byte-pair-encoding](https://www.wikiprompt.org/wiki/byte-pair-encoding) - [wordpiece](https://www.wikiprompt.org/wiki/wordpiece) - [sentencepiece](https://www.wikiprompt.org/wiki/sentencepiece) - [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) --- Source: https://www.wikiprompt.org/wiki/subword-tokenization License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-13T04:00:07.74947+00:00