Regularization refers to a broad family of techniques used in Machine learning to reduce Overfitting by discouraging a model from fitting the training data too precisely, so that it generalizes better to new data. Rather than changing what a model is trying to learn, most regularization methods change how it is allowed to learn, either by penalizing complexity directly, by injecting controlled noise during training, or by stopping training before the model has a chance to memorize idiosyncrasies of the training set.
Weight-based methods
The oldest and most direct approach adds a penalty term to the Loss function based on the size of the model's weights, encouraging the optimizer to prefer smaller, simpler weight values unless the data strongly justifies larger ones. L2 regularization, also called weight decay or ridge regularization, penalizes the sum of squared weights and tends to shrink all weights smoothly toward zero. L1 regularization penalizes the sum of absolute weight values and tends to push some weights to exactly zero, effectively performing feature selection. In modern deep learning, weight decay is typically applied directly inside the optimizer, for instance as a modification to Adam known as AdamW, rather than as an explicit term added to the loss.
Structural and stochastic methods
Dropout, introduced by Geoffrey Hinton's group around 2012 and formalized in a widely cited 2014 paper, randomly disables a fraction of a neural network's units during each training step, forcing the network to avoid relying too heavily on any single unit or fixed combination of units and effectively training an implicit ensemble of subnetworks that share weights. Batch normalization and its successors, though introduced primarily to stabilize and speed up training, also have a regularizing side effect by adding noise to each layer's inputs during training. Data augmentation, which creates modified copies of training examples, such as rotated or cropped images, or paraphrased text, artificially expands the effective diversity of Training data and reduces the chance a model latches onto superficial, non-generalizable patterns.
Early stopping and model selection
Early stopping halts training once performance on a held-out validation set stops improving, even if training loss continues to fall, directly targeting the divergence between training and validation performance that defines Overfitting. This requires monitoring validation loss throughout training rather than only at the end, and is one of the simplest and most widely used regularization techniques precisely because it requires no change to the model or loss function itself, only a stopping rule.
Regularization in large models
Somewhat counterintuitively, the largest Large language models and Deep learning systems often use lighter explicit regularization than smaller models, because training on enormous, diverse Training data with a limited number of passes provides a natural form of regularization: the model rarely if ever sees the same example twice, leaving little opportunity to memorize it. Dropout in particular is frequently reduced or removed entirely in large-scale Transformer (architecture) pretraining, though it remains common during smaller-scale Fine-tuning stages, where overfitting to a narrower dataset is a more realistic risk.