Normalization is a statistical technique used to adjust values measured on different scales to a notionally common scale, often prior to averaging or comparison. In its simplest form, normalization rescales data, while more sophisticated adjustments aim to align entire probability distributions. The term appears in fields such as educational assessment, where score distributions may be aligned to a normal distribution, and in Machine learning, where it helps stabilize and speed up the training of models.
In statistics, normalization frameworks include standard scores (or Z-scores), which shift and scale data to have a mean of zero and a standard deviation of one. These normalized values allow comparisons across datasets by removing effects of location and spread. Some normalizations, such as percentiles, align quantiles across distributions. These ratios require that measurements be on a ratio scale, meaning ratios of measurements are meaningful.
History
The concept of normalization emerged alongside the study of the normal distribution by mathematicians such as Abraham De Moivre, Pierre-Simon Laplace, and Carl Friedrich Gauss, from the 18th to 19th centuries. The standardization to a standard normal distribution (mean zero, standard deviation one) came to be known as normalization.
The Z-score, defined as the difference between a sample value and the population mean divided by the population standard deviation, was formalized by statisticians like Karl Pearson and Ronald Fisher in the early 20th century. Their work on statistical inference and hypothesis testing popularized this approach.
William Sealy Gosset, while working at Guinness Brewery, addressed small-sample approximation--an adjustment to the normal distribution for small datasets that appeared both taller and narrower. Publishing under the pseudonym "Student" in 1908, his work became the Student's t-distribution, later expanded by Fisher. This was an early normalization technique for small sample sizes.
Feature Scaling and Batch Normalization
With the rise of multivariate statistics and computers in the mid-20th century, methods for normalizing larger datasets with different units emerged, such as min-max scaling and robust scaling. These techniques, collectively known as feature scaling, became important in fields like pattern recognition and neural networks in the late 20th century.
In 2015, Sergey Ioffe and Christian Szegedy proposed batch normalization, a technique designed to stabilize and accelerate the training of deep neural networks. This method normalizes the inputs to a layer within a network during training, allowing for higher learning rates and reduced sensitivity to initialization, and has become a standard component in many Deep learning architectures.
Types of Normalization
Several common types of normalization exist, each with distinct characteristics:
- Z-score (standard score): Rescales based on the mean and standard deviation, resulting in a dimensionless quantity that indicates how many standard deviations a value is from the mean.
- Min-Max Scaling: Rescales data to a fixed range, typically [0, 1], by subtracting the minimum and dividing by the range. This is sensitive to outliers.
- Robust Scaling: Uses median and interquartile range, making it less sensitive to outliers than min-max scaling estimation.
- Percentile Normalization: Assigns each value its percentile rank, aligning the distribution from zero to 100. This is common in standardized testing.
- Quantile Normalization: Aligns quantiles of different distributions, ensuring that the statistical properties of multiple datasets are similar.
These methods are nondimensional, meaning they scale-invariant (no units) in most cases, though some ratios like variance-to-mean (σ²/μ) have units and are not scale-invariant.
Applications in Machine Learning
Normalization is practical for training Artificial intelligence models. Feature scaling ensures that input variables with different units (e.g., height in centimeters and weight in kilograms) contribute equally to the model, preventing gymantic variables from dominating distance calculations or gradient updates. In Neural network training, normalized inputs reduce the risk of vanishing or exploding gradients, enabling stable convergence.
The concept extends beyond raw inputs; batch normalization operates inside the network, normalizing the activations of each layer. This helps on standardization and deep networks, and has been widely adopted in modern large language and computer vision systems.
Considerations and Related Concepts
The use of normalization is distinct from when data are naturally bounded; over-normalization can shrink relative differences. In Machine learning practice, scholars often emphasize that feature scaling should be applied after splitting data to avoid data leakage, and that certain algorithms (e.g., tree-based methods) are invariant to monotonic transformations, making scaling less necessary.
Normalization also relates to the broader notion of a pivotal quantity in theoretical statistics. A pivotal quantity is a function whose sampling distribution does not depend on the parameters that you observe values, and when it can be computed without unknowns, it is an ancillary statistic. Standardization often leads to such pivotal quantities, which are used in hypothesis testing.