Cross-validation, sometimes called rotation estimation or out-of-sample testing, is a family of model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. It is a resampling method that uses different portions of the data to train and test a model on different iterations. Cross-validation is commonly used in prediction settings to estimate how accurately a predictive model will perform in practice, and it can also assess the quality of a fitted model and the stability of its parameters.
In a typical prediction problem, a model is trained on a known dataset (the training set) and then tested on a dataset of unseen data (the validation or testing set). The goal is to measure the model's ability to predict new data that was not used in estimation, thereby flagging problems such as overfitting or selection bias and providing insight into how the model will generalize to an independent dataset. One round of cross-validation involves partitioning the sample into complementary subsets: the model is fit on one subset (training set) and validated on the other (validation set). To reduce variability, most methods perform multiple rounds with different partitions, and the validation results are averaged over the rounds to yield a more accurate estimate of predictive performance.
Motivation
Consider a model with one or more unknown parameters and a dataset to which the model can be fit. The fitting process optimizes the parameters to make the model fit the training data as well as possible. If an independent sample of validation data is taken from the same population, the model will generally not fit the validation data as well as it fits the training data. This discrepancy is especially large when the training set is small or when the number of parameters is large. Cross-validation provides a way to estimate the size of this effect, which is crucial for understanding model performance.
Example: Linear Regression
In linear regression, suppose there are real response values \(y_1, \ldots, y_n\) and \(n\) p-dimensional covariate vectors \(\mathbf{x}_1, \ldots, \mathbf{x}_n\), where each \(\mathbf{x}_i\) has components \(x_{i1}, \ldots, x_{ip}\). Using least squares to fit a hyperplane \(\hat{y} = a + \boldsymbol{\beta}^T \mathbf{x}\) to the data, the mean squared error (MSE) on the training set is defined as:
\[
\text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \frac{1}{n} \sum_{i=1}^n (y_i - a - \boldsymbol{\beta}^T \mathbf{x}_i)^2.
\]
If the model is correctly specified, the training MSE tends to underestimate the true prediction error because the model has been optimized to fit the training data. Cross-validation helps correct this optimism by evaluating the model on held-out data.
Common Methods
Several cross-validation techniques exist, differing in how they partition the data.
k-Fold Cross-Validation
In k-fold cross-validation, the original sample is randomly partitioned into \(k\) equal-sized subsamples. Of these, \(k-1\) subsamples are used as training data, and the remaining single subsample is used as validation data. The process is repeated \(k\) times, with each subsample used exactly once as the validation data. The \(k\) results are then averaged to produce a single estimate. Typical choices for \(k\) are 5 or 10, balancing bias and variance. A special case is leave-one-out cross-validation (LOOCV), where \(k\) equals the number of data points, meaning each observation is used once as the validation set.
Leave-One-Out Cross-Validation (LOOCV)
LOOCV is an extreme form of k-fold cross-validation where each training set consists of all but one observation, and the validation set is that single observation. This method is computationally expensive for large datasets but provides a nearly unbiased estimate of prediction error, though it can have high variance.
Repeated Random Subsampling
Also known as Monte Carlo cross-validation, this method randomly splits the data into training and validation sets multiple times, with replacement. The prediction error is averaged over the splits. This approach does not guarantee that each observation is used for validation exactly once, but it can be useful when the data size is large.
Stratified Cross-Validation
In classification problems with imbalanced classes, stratified cross-validation ensures that each fold maintains the same class proportions as the original dataset. This prevents a fold from lacking a particular class, which could lead to misleading performance estimates.
Applications
Cross-validation is widely used in Machine learning and Artificial intelligence for model selection and hyperparameter tuning. For instance, in training neural networks or large language models, practitioners use cross-validation to choose the number of layers, learning rates, or regularization parameters. It is also used in Deep learning to compare architectures and in Transformer (architecture)-based models to evaluate generalization before deployment.
Beyond prediction, cross-validation helps assess the stability of model parameters. If a model's parameters vary drastically across different training folds, that indicates instability and potential overfitting. This is particularly important in fields like Generative AI, where models must generalize to diverse real-world inputs.
Limitations and Considerations
Cross-validation has several limitations. It can be computationally expensive, especially with large datasets and complex models, as it requires repeated training. For time-series data, standard cross-validation is inappropriate because random splitting violates temporal order; instead, techniques like forward chaining are used. Additionally, cross-validation estimates the expected prediction error for a given training set size, but it does not fully account for the variability due to the random split, which can lead to high variance in the estimate when the dataset is small.
Another consideration is that cross-validation assumes the data are independent and identically distributed (i.i.d.). In practice, if there is clustering or correlation within the data, the validation results may be overly optimistic. Methods like grouped cross-validation can address this by ensuring that all observations from a group are kept together in either the training or validation set.
Relationship to Overfitting
Overfitting occurs when a model learns the noise in the training data rather than the underlying pattern, leading to poor performance on new data. Cross-validation is a primary tool for detecting overfitting: if the validation error is significantly higher than the training error, the model is likely overfitting. By averaging over multiple splits, cross-validation provides a more reliable estimate of generalization error than a single train-test split, helping practitioners avoid selecting overly complex models.
Historical Context
The concept of cross-validation has roots in statistics dating back to the 1930s, with early work by Seymour Geisser and others. It became more prominent with the rise of computational statistics and Machine learning in the late 20th century. Researchers such as Thomas G. Dietterich and Michael I. Jordan have contributed to its theoretical understanding and practical applications. Today, cross-validation is a standard tool in the toolbox of data scientists and is implemented in most statistical and machine learning libraries.
See Also
- Machine learning
- Artificial intelligence
- Deep learning
- Neural network
- Large language model
- Transformer (architecture)
- Generative AI
- Overfitting (if available)
References
- Geisser, S. (1975). The predictive sample reuse method with applications. Journal of the American Statistical Association.
- Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society.
- Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence.