The context window is the maximum amount of text, measured in tokens, that a language model can take into account at once, spanning the prompt supplied by a user, any system instructions, retrieved documents, and the model's own output so far. It functions as the model's working memory: information outside the context window is invisible to the model at generation time unless it is explicitly re-supplied.
Technical basis
The size of the context window is bounded by the attention mechanism at the core of the Transformer (architecture) architecture, which computes relationships between every pair of tokens in the input. Because that computation scales quadratically with sequence length in the standard formulation, doubling the context window roughly quadruples the compute and memory required to process it, which historically made long context windows expensive to serve. Later engineering techniques, including sparse and windowed attention variants, key-value cache compression, and specialized memory management, reduced this cost enough to make much larger windows commercially practical.
Growth over time
Early transformer-based language models supported only a few hundred to a couple thousand tokens; GPT-3, released by OpenAI in 2020, supported roughly 2,048 tokens. Context windows expanded steadily over the following years: GPT-4 launched in 2023 with variants supporting up to 32,000 tokens, and by 2024 several labs offered context windows in the hundreds of thousands of tokens. Gemini, from Google DeepMind, became notable for supporting context windows of one million tokens or more, enough to process an entire novel, a large codebase, or hours of video transcript in a single prompt. Claude (AI model family) models from Anthropic similarly expanded from an initial 9,000-token window to context lengths in the hundreds of thousands of tokens over subsequent releases.
Tradeoffs and long-context limitations
A larger context window does not guarantee a model uses all of it well. Research on long-context performance has repeatedly found a "lost in the middle" effect, in which models are more accurate at retrieving information placed near the beginning or end of a long context than information buried in the middle, even when nothing in the architecture formally privileges those positions. Long-context "needle in a haystack" tests, in which a specific fact is hidden inside a large volume of irrelevant text, became a standard way to measure whether a model's effective context matches its advertised maximum. Serving very long contexts is also computationally expensive, so systems often face a practical choice between stuffing a large amount of raw text into the context window and using Retrieval-augmented generation to select only the most relevant passages, trading completeness for cost and, in principle, accuracy.
Relation to reasoning
The context window also constrains techniques like Chain-of-thought prompting and reasoning models, which generate substantial intermediate text before producing a final answer; a model that reasons at length consumes part of its own context window with that reasoning, which is one reason longer context windows became more important as those techniques became widespread.