Principal component analysis (PCA) is a linear dimensionality reduction technique that transforms a dataset of possibly correlated variables into a smaller number of uncorrelated variables called principal components. The method identifies the directions of maximum variance in the data and projects the original observations onto these directions, producing a lower-dimensional representation that retains as much information as possible. PCA is one of the most widely used tools in Machine learning for exploratory data analysis, feature engineering, and preprocessing, and it also serves as a foundational concept in fields ranging from genetics to finance.
The technique was first introduced by Karl Pearson in 1901 and later developed independently by Harold Hotelling in 1933. It is mathematically equivalent to performing singular value decomposition (SVD) on the centered data matrix, which makes it computationally efficient and numerically stable. PCA does not require labeled data, making it an unsupervised method, and it assumes that the directions of greatest variance are the most informative, which holds for many real-world datasets.
Mathematical formulation
Given a data matrix X with n observations and p variables, PCA begins by centering each variable to have zero mean. The covariance matrix of the centered data is computed, and its eigenvectors and eigenvalues are obtained. The eigenvectors, sorted by decreasing eigenvalue, define the principal component axes, while the eigenvalues indicate the amount of variance explained by each component. The first principal component captures the largest possible variance, the second captures the next largest under the constraint of being orthogonal to the first, and so on.
The projection of the data onto the first k principal components yields a k-dimensional representation. The proportion of total variance explained by the first k components is the sum of their eigenvalues divided by the total sum of all eigenvalues. This quantity guides the choice of k, often using a scree plot or a threshold such as 95% variance retention.
Applications in machine learning
In Machine learning pipelines, PCA is frequently used to reduce the number of features before training a model. High-dimensional datasets can suffer from the curse of dimensionality, leading to overfitting and increased computational cost. By projecting data onto a lower-dimensional subspace, PCA can improve model generalization and speed up training. It is especially common in fields like image processing, where raw pixel intensities are highly correlated, and in genomics, where expression levels of thousands of genes are measured simultaneously.
PCA also serves as a visualization tool. Projecting data onto the first two or three principal components allows researchers to plot high-dimensional data in two or three dimensions, revealing clusters, outliers, or other structures. This is often used in exploratory analysis before applying more complex methods such as Neural network classifiers.
Another application is noise reduction. By discarding the components with the smallest eigenvalues, PCA filters out directions that are assumed to contain mostly noise, retaining the signal-dominant subspace. This principle underlies techniques like eigenfaces for facial recognition, where PCA was famously applied in the 1990s.
Relationship to other techniques
PCA is closely related to factor analysis, but the two differ in their goals: PCA aims to explain variance in the data, while factor analysis models the covariance structure using latent factors. PCA is also a special case of multidimensional scaling (MDS) when the dissimilarity measure is Euclidean distance. In the context of Deep learning, PCA is sometimes used as a preprocessing step for Neural network inputs, although modern methods often rely on learned representations.
Kernel PCA extends the method to nonlinear manifolds by applying a kernel function before computing principal components, allowing the capture of nonlinear structures. Sparse PCA and robust PCA are variants that impose sparsity or handle outliers, respectively. These extensions have found use in signal processing and computer vision.
Limitations and considerations
PCA assumes linearity, meaning it can only capture linear relationships among variables. For data lying on nonlinear manifolds, PCA may produce misleading results, and methods like t-SNE or UMAP are often preferred for visualization. PCA is also sensitive to the scaling of variables; without standardization, variables with larger variances dominate the components. Therefore, it is common practice to standardize each variable to unit variance before applying PCA.
The interpretability of principal components can be challenging, as each component is a linear combination of all original variables. This makes it difficult to assign physical meaning to the components in some applications. Additionally, PCA is an unsupervised method and does not consider class labels, so it may discard directions that are important for classification but have low variance.
Historical and modern context
PCA has been independently rediscovered in various fields, including meteorology (as empirical orthogonal functions) and ecology. Its mathematical foundations were laid in the early 20th century, and the advent of computers in the 1950s enabled its practical application to large datasets. Today, PCA remains a standard tool in the toolkit of data scientists and is implemented in all major statistical and Machine learning libraries, including scikit-learn, R, and MATLAB.
In the era of large-scale Artificial intelligence, PCA is still used for tasks such as whitening in Deep learning models and for compressing embeddings in Large language model pipelines. Its simplicity, interpretability, and computational efficiency ensure its continued relevance despite the availability of more complex nonlinear methods.