A token is the fundamental unit of text that a language model reads and writes. In the context of large language models, tokens are not necessarily whole words; they can be subwords, characters, or even byte-level representations. The process of converting raw text into tokens is called tokenization, and it is a critical preprocessing step that determines how a model interprets and generates language. All operational metrics of a language model, including pricing, context window size, and processing speed, are measured in tokens, making them the currency of modern artificial intelligence systems.
Tokenization allows models to handle a vast vocabulary efficiently. Instead of storing every possible word, a tokenizer breaks text into smaller pieces based on a learned vocabulary, typically ranging from 30,000 to 100,000 tokens. For example, the word "unhappiness" might be split into "un", "happi", and "ness", while common words like "the" remain as single tokens. This approach balances computational efficiency with linguistic coverage, enabling models to process diverse languages and technical jargon without an impractical dictionary size.
Tokenization Methods
Several tokenization algorithms have been developed over the years. Byte-Pair Encoding (BPE), introduced by Rico Sennrich and colleagues in 2016, is one of the most widely used methods. BPE iteratively merges the most frequent pairs of characters or subwords in a training corpus until a target vocabulary size is reached. Another popular method is WordPiece, developed at Google, which uses a likelihood-based approach to select merges. More recently, SentencePiece, created by Google, treats the entire text as a sequence of characters and can handle languages without clear word boundaries, such as Japanese or Chinese.
Unigram language model tokenization, also implemented in SentencePiece, takes a probabilistic approach by selecting the most likely segmentation of a text based on a trained unigram model. Each method has trade-offs in terms of compression rate, out-of-vocabulary handling, and downstream model performance. The choice of tokenizer significantly affects how well a model understands rare words, morphological variations, and multilingual text.
Role in Language Models
In transformer-based models, tokens are converted into numerical embeddings that serve as input to the neural network. Each token is assigned a unique integer ID from the vocabulary, and these IDs are mapped to high-dimensional vectors that capture semantic and syntactic information. The model processes sequences of these embeddings through multiple layers of attention and feed-forward networks, learning patterns and relationships between tokens.
During generation, the model outputs a probability distribution over the entire token vocabulary for the next token, given the preceding context. This autoregressive process continues until a special end-of-sequence token is produced or a maximum length is reached. The granularity of tokens affects the model's ability to generate coherent text; subword tokens allow for flexible composition of new words, while character-level tokens would require much longer sequences and more computation.
Context Window and Limits
Context window size, measured in tokens, defines how much text a model can consider at once. Early models like GPT-2 had context windows of 1,024 tokens, while modern models such as GPT-4 Turbo support up to 128,000 tokens, and some models like Claude 3 offer 200,000-token contexts. These limits directly impact the types of tasks a model can perform, such as analyzing long documents, summarizing entire books, or maintaining coherent conversations over extended interactions.
When input exceeds the context window, models typically truncate the oldest tokens or use techniques like sliding windows or retrieval augmentation to manage longer texts. The context window also influences memory usage and computational cost; longer contexts require more attention computations, which scale quadratically with sequence length in standard transformers. This has motivated research into efficient attention mechanisms, such as sparse attention and linear attention, to extend context lengths without prohibitive resource demands.
Pricing and Cost
API providers charge based on token usage, typically differentiating between input and output tokens. For example, OpenAI's GPT-4 Turbo costs $0.01 per 1,000 input tokens and $0.03 per 1,000 output tokens, while Anthropic's Claude 3 Opus charges $0.015 for input and $0.075 for output per 1,000 tokens. These prices reflect the computational cost of processing tokens, with output tokens being more expensive due to the sequential generation process.
Token count also determines the cost of fine-tuning and training. Training a model involves processing billions of tokens, and the total cost scales linearly with the number of tokens. For instance, training a 70-billion-parameter model like Llama 2 required approximately 2 trillion tokens, costing millions of dollars in compute. Understanding token pricing is essential for budgeting AI applications, especially those involving large-scale text processing or real-time interactions.
Speed and Throughput
Processing speed is measured in tokens per second (TPS). Inference speed varies widely depending on hardware and model size. On high-end GPUs like NVIDIA A100, a 7-billion-parameter model might generate 20-30 tokens per second, while smaller models on specialized hardware can achieve hundreds of tokens per second. Companies like Groq have developed custom chips that achieve extremely high throughput, with their LPU (Language Processing Unit) delivering over 500 tokens per second for some models.
Speed is critical for real-time applications such as chatbots, code completion, and live translation. Latency, the time to first token, is also measured in milliseconds and depends on the model's size and the efficiency of the inference stack. Techniques like quantization, model pruning, and speculative decoding are used to improve token throughput, enabling more responsive and cost-effective deployments.
Token Efficiency and Optimization
Researchers and practitioners continuously seek to improve token efficiency. This includes developing tokenizers that compress text more effectively, reducing the number of tokens needed to represent the same information. For example, some models use byte-level tokenization to handle arbitrary text, while others employ specialized vocabularies for code or scientific notation.
Prompt engineering also focuses on token efficiency, as shorter prompts reduce cost and latency. Techniques like instruction tuning and few-shot learning aim to achieve desired outputs with minimal token usage. Additionally, model distillation and quantization can reduce the computational cost per token, making models faster and cheaper to run without significant performance loss.
Future Directions
As language models evolve, tokenization remains an active area of research. Multimodal models now tokenize not only text but also images, audio, and video, using separate or unified tokenizers. For instance, models like GPT-4V process image patches as tokens, while audio models convert waveforms into discrete representations. This expansion of token types enables more versatile AI systems that can understand and generate across modalities.
There is also growing interest in dynamic tokenization, where the segmentation adapts based on context or task. Some research explores using larger token units, such as entire phrases or sentences, to improve efficiency and coherence. However, these approaches face challenges in maintaining flexibility and handling novel inputs. The ongoing development of tokenization techniques will continue to shape the capabilities and economics of language models.