Overfitting is a failure mode in Machine learning in which a model learns the training data too closely, including its noise and idiosyncrasies, at the expense of learning patterns that generalize to new, unseen data. An overfit model typically shows very low error on the data it was trained on but substantially higher error on held-out data, the opposite of the intended goal of building a model that performs well on examples it has never seen.
Causes
Overfitting tends to occur when a model has enough capacity, meaning enough parameters or effective flexibility, to memorize idiosyncratic details of a finite training set rather than being forced to find the simpler, more general patterns underlying it. It becomes more likely as the ratio of model capacity to training data size increases, which is why overfitting was historically a bigger concern for small datasets and comparatively simple models than it might seem for today's Large language models, whose training sets are enormous. Overfitting can also happen more locally, for example when a model is fine-tuned on a small, narrow dataset after Pretraining, where it can lose some of its broader capabilities while over-adapting to the fine-tuning set, a related phenomenon sometimes called catastrophic forgetting. Training for too many passes over the same data, known as too many epochs, is a classic direct cause, since a model with sufficient capacity will gradually shift from learning general patterns to memorizing specific examples the longer it trains on them.
Detection
The standard way to detect overfitting is to monitor a Loss function on a validation set, a portion of data held out from training, alongside the training loss. As long as both losses fall together, the model is generalizing; once the training loss keeps falling while the validation loss stalls or rises, the model has begun overfitting. This validation curve is the basis for early stopping, a technique that halts training near the point where validation performance is best, and for choosing between competing architectures or hyperparameter settings using a validation set the model was never trained on.
Mitigation
The general family of techniques for preventing or reducing overfitting is known as Regularization, which includes weight penalties, dropout, data augmentation, and simply gathering more or more diverse Training data so that memorizable idiosyncrasies matter less relative to genuine patterns. Cross-validation, which repeatedly splits data into different training and validation partitions, gives a more robust estimate of how well a model generalizes than a single train-validation split, and is standard practice for smaller datasets where a single held-out set would itself be noisy.
Overfitting at the benchmark level
A related, higher-level version of the same problem affects how the entire field measures progress: when researchers repeatedly tune models against the same public AI benchmark over years, results can improve on that specific benchmark without corresponding real-world gains, and popular evaluation sets risk leaking into Training data scraped from the web, a problem called benchmark or data contamination. This has made held-out, contamination-resistant evaluations, and benchmarks like ARC-AGI that are explicitly designed to resist memorization, increasingly important for honestly measuring whether Large language models are improving at genuine reasoning rather than overfitting to familiar test distributions.