# Regularization

Regularization is a set of techniques, including weight penalties, dropout, and early stopping, used to reduce overfitting and improve how well a machine learning model generalizes to new data.

Regularization refers to a broad family of techniques used in [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) to reduce [overfitting](https://www.wikiprompt.org/wiki/overfitting) by discouraging a model from fitting the training data too precisely, so that it generalizes better to new data. Rather than changing what a model is trying to learn, most regularization methods change how it is allowed to learn, either by penalizing complexity directly, by injecting controlled noise during training, or by stopping training before the model has a chance to memorize idiosyncrasies of the training set.

## Weight-based methods

The oldest and most direct approach adds a penalty term to the [loss-function](https://www.wikiprompt.org/wiki/loss-function) based on the size of the model's weights, encouraging the optimizer to prefer smaller, simpler weight values unless the data strongly justifies larger ones. L2 regularization, also called weight decay or ridge regularization, penalizes the sum of squared weights and tends to shrink all weights smoothly toward zero. L1 regularization penalizes the sum of absolute weight values and tends to push some weights to exactly zero, effectively performing feature selection. In modern deep learning, weight decay is typically applied directly inside the optimizer, for instance as a modification to Adam known as AdamW, rather than as an explicit term added to the loss.

## Structural and stochastic methods

Dropout, introduced by Geoffrey Hinton's group around 2012 and formalized in a widely cited 2014 paper, randomly disables a fraction of a neural network's units during each training step, forcing the network to avoid relying too heavily on any single unit or fixed combination of units and effectively training an implicit ensemble of subnetworks that share weights. Batch normalization and its successors, though introduced primarily to stabilize and speed up training, also have a regularizing side effect by adding noise to each layer's inputs during training. Data augmentation, which creates modified copies of training examples, such as rotated or cropped images, or paraphrased text, artificially expands the effective diversity of [training-data](https://www.wikiprompt.org/wiki/training-data) and reduces the chance a model latches onto superficial, non-generalizable patterns.

## Early stopping and model selection

Early stopping halts training once performance on a held-out validation set stops improving, even if training loss continues to fall, directly targeting the divergence between training and validation performance that defines [overfitting](https://www.wikiprompt.org/wiki/overfitting). This requires monitoring validation loss throughout training rather than only at the end, and is one of the simplest and most widely used regularization techniques precisely because it requires no change to the model or loss function itself, only a stopping rule.

## Regularization in large models

Somewhat counterintuitively, the largest [large-language-model](https://www.wikiprompt.org/wiki/large-language-model)s and [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) systems often use lighter explicit regularization than smaller models, because training on enormous, diverse [training-data](https://www.wikiprompt.org/wiki/training-data) with a limited number of passes provides a natural form of regularization: the model rarely if ever sees the same example twice, leaving little opportunity to memorize it. Dropout in particular is frequently reduced or removed entirely in large-scale [transformer](https://www.wikiprompt.org/wiki/transformer) pretraining, though it remains common during smaller-scale [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning) stages, where overfitting to a narrower dataset is a more realistic risk.

---
Source: https://www.wikiprompt.org/wiki/regularization
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:29:49.624791+00:00