Kernel principal component analysis (kernel PCA) is a technique in machine learning for nonlinear dimensionality reduction. It extends classical principal component analysis (PCA) by applying the kernel trick, which allows the algorithm to operate in a high-dimensional, implicit feature space without explicitly computing the coordinates of data in that space. This enables kernel PCA to capture nonlinear patterns that linear PCA would miss, making it useful for tasks such as data visualization, noise reduction, and preprocessing for other algorithms.
The method was introduced in 1998 by Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Their formulation showed that principal components in the feature space can be obtained by solving an eigenvalue problem on the kernel matrix, which contains pairwise similarities between data points computed via a kernel function. Common kernels include the polynomial kernel, the radial basis function (RBF) kernel, and the sigmoid kernel. The choice of kernel and its parameters significantly influences the resulting embedding.
Mathematical formulation
Given a set of n data points x_i in the input space, kernel PCA first maps each point to a feature space via a function φ, such that φ(x_i) is a vector in a possibly infinite-dimensional Hilbert space. The kernel function k(x_i, x_j) computes the inner product ⟨φ(x_i), φ(x_j)⟩. The algorithm centers the data in the feature space by modifying the kernel matrix, then computes the eigenvectors and eigenvalues of the centered kernel matrix. The projection of a new point onto a principal component is expressed as a weighted sum of kernel evaluations between the new point and the training points, with weights derived from the eigenvectors.
Unlike linear PCA, which finds directions of maximum variance in the original space, kernel PCA finds directions of maximum variance in the feature space. The number of principal components is at most the number of training points, and the method can produce more components than the input dimensionality. In practice, one typically selects the top k components based on the magnitude of the associated eigenvalues.
Relation to other methods
Kernel PCA is closely related to other kernel-based methods such as support vector machines (SVMs) and Gaussian process regression, all of which rely on the kernel trick. It can be viewed as a nonlinear generalization of PCA, and it is also connected to spectral clustering and multidimensional scaling. In the case of a linear kernel, kernel PCA reduces exactly to standard PCA. The method is also a precursor to more advanced nonlinear embedding techniques like t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP), though those methods optimize different objectives.
Applications and limitations
Kernel PCA has been applied in various domains, including face recognition, image denoising, and anomaly detection. In face recognition, it can extract nonlinear features that improve classification accuracy compared to linear PCA. For anomaly detection, the reconstruction error in the feature space can indicate outliers. However, kernel PCA has notable limitations: the computational cost scales quadratically with the number of training samples, making it impractical for very large datasets without approximations. The choice of kernel and hyperparameters often requires cross-validation, and the interpretability of the resulting components is lower than in linear PCA. Additionally, the method assumes that the data lies on a low-dimensional manifold in the feature space, which may not hold in all cases.
Practical considerations
Implementing kernel PCA involves selecting a kernel function and its parameters, such as the bandwidth in an RBF kernel. The kernel matrix must be centered, and the eigenvalue decomposition can be performed using standard linear algebra libraries. For large-scale problems, approximate methods such as the Nyström method or random feature maps can reduce computational burden. In modern machine learning pipelines, kernel PCA is often used as a preprocessing step before applying clustering or classification algorithms. It has been implemented in popular libraries such as scikit-learn, which provides a KernelPCA class with options for different kernels and solvers.