Tokenization is the process of splitting text into discrete units called tokens, which serve as the basic vocabulary that a Large language model reads as input and produces as output. Tokens are typically subword pieces rather than whole words or individual characters, a design choice that balances vocabulary size against sequence length.
Why not words or characters
Splitting text into whole words would require a vocabulary large enough to cover every word form in every language a model supports, including rare words, misspellings, and names, while still failing whenever a genuinely new word appears. Splitting text into individual characters avoids that vocabulary problem but produces much longer sequences for a model to process, which is costly given that Transformer (architecture) Attention mechanisms scale poorly with sequence length. Subword tokenization is a compromise: common words are typically represented as a single token, while rarer words are broken into smaller, reusable pieces, and even entirely unseen character combinations can still be represented as a sequence of known subword units.
Methods
The most widely used subword tokenization method is byte-pair encoding, adapted for language models from a 1994 data compression algorithm and building on earlier statistical Natural language processing techniques for handling words not seen during training. It works by starting with individual characters or bytes and iteratively merging the most frequently co-occurring pairs into new tokens, building up a fixed-size vocabulary of the most useful subword units observed in a large training corpus. Variants such as SentencePiece and WordPiece follow similar merging logic with different preprocessing and handling of whitespace. The resulting vocabulary, typically tens of thousands of tokens, is fixed once a model is trained and cannot be changed without retraining.
From tokens to numbers
Once text is split into tokens, each token is mapped to an integer index and then to a learned vector representation called an Embedding, which is what the neural network actually processes. GPT-2 and GPT-3 both used byte-level byte-pair encoding tokenizers, and most subsequent large language models have followed similar approaches, sometimes with expanded vocabularies to better represent non-English languages and code.
Practical implications
Tokenization has several consequences that are visible to users of language models even without any technical background. A model's Context window, the maximum amount of text it can process at once, is measured in tokens rather than words or characters, and commercial APIs typically charge per token, making tokenization directly relevant to cost. Tokenization schemes trained predominantly on English text tend to split non-English text, especially languages with different scripts, into a larger number of tokens per word than English, which can make using a model more expensive and can effectively shrink the usable context window for speakers of those languages. Tokenization can also produce surprising behavior on tasks that require reasoning about individual characters, such as counting letters within a word, since a model may never see the word broken into its component characters at all.