A confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of a person or an algorithm on a specific task. In machine learning, these matrices show the success of a learning system in both supervised and unsupervised learning, where they are sometimes called a matching matrix. The name stems from the fact that the matrix makes it easy to identify whether the system is confusing two classes, that is, commonly mislabeling one class as another.
The confusion matrix has its origins in human perceptual studies of auditory stimuli. It was later adapted for machine learning studies and used by early researchers such as Frank Rosenblatt to compare human and machine classifications of visual and auditory stimuli. It is a special kind of contingency table with two dimensions, "actual" and "predicted", and identical sets of classes in both dimensions. Each row of the matrix represents instances in an actual class, while each column represents instances in a predicted class, or vice versa, depending on the convention used. The diagonal of the matrix therefore represents all instances that are correctly predicted.
Binary Classification Example
Consider a sample of 12 individuals, 8 diagnosed with cancer (class 1, positive) and 4 cancer-free (class 0, negative). A classifier distinguishes between individuals with and without cancer. After running the 12 individuals through the classifier, it makes 9 accurate predictions and misses 3: 2 individuals with cancer are wrongly predicted as cancer-free, and 1 person without cancer is wrongly predicted as having cancer.
Comparing actual and predicted classifications yields four possible outcomes:
- True positive (TP): Actual positive and predicted positive.
- False negative (FN): Actual positive and predicted negative.
- False positive (FP): Actual negative and predicted positive.
- True negative (TN): Actual negative and predicted negative.
These outcomes can be arranged in a 2×2 confusion matrix. For the example, the matrix would show:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | 6 (TP) | 2 (FN) |
| Actual Negative | 1 (FP) | 3 (TN) |
All correct predictions lie on the diagonal, making it easy to spot errors, which appear off-diagonal. Summing rows gives the total number of positive (P = TP + FN) and negative (N = FP + TN) samples in the dataset.
Table of Confusion
In predictive analytics, a table of confusion (sometimes also called a confusion matrix) is a two-by-two table reporting counts of true positives, false negatives, false positives, and true negatives. This allows more detailed analysis than simply observing accuracy, which can be misleading if the dataset is unbalanced. For example, with 95 cancer samples and 5 non-cancer samples, a classifier that labels everything as cancer would have 95% accuracy but 0% sensitivity for the non-cancer class. Metrics like F1 score can also be biased; informedness removes such bias, yielding 0 for any form of guessing. According to Davide Chicco and Giuseppe Jurman, the most informative metric to evaluate a confusion matrix is the Matthews correlation coefficient (MCC).
Some researchers argue that the confusion matrix and derived metrics do not fully reflect a model's knowledge. The matrix cannot show whether correct predictions were reached through sound reasoning or by chance (epistemic luck), nor does it capture situations where facts used for prediction later change (defeasibility). Thus, while useful, the confusion matrix may give an incomplete picture of a model's true reliability.
Multi-Class Confusion Matrices
Confusion matrices are not limited to binary classification; they can be used in multi-class classifiers. For a classifier with k classes, the matrix has k rows and k columns, where each cell (i, j) counts instances of actual class i predicted as class j. The diagonal represents correct predictions, and off-diagonal cells show misclassifications. This layout helps identify which classes are commonly confused with each other.
Derived Metrics
Several performance metrics can be computed from a confusion matrix:
- Accuracy: (TP + TN) / (TP + TN + FP + FN)
- Precision: TP / (TP + FP)
- Recall (Sensitivity): TP / (TP + FN)
- Specificity: TN / (TN + FP)
- F1 score: Harmonic mean of precision and recall
- Matthews correlation coefficient (MCC): A balanced measure that accounts for all four cells.
These metrics provide different perspectives on classifier performance, especially in imbalanced datasets.
Applications in Machine Learning
Confusion matrices are widely used in Machine learning to evaluate classifiers, including Neural network models and Deep learning systems. They are essential in fields like medical diagnosis, spam detection, and image recognition. For instance, in Artificial intelligence research, confusion matrices help compare model performance across different classes, guiding improvements in model architecture and training data.
Limitations and Criticisms
Despite their utility, confusion matrices have limitations. They do not capture the reasoning behind predictions, and they assume static ground truth. In dynamic environments, such as those involving Large language model outputs, the matrix may not reflect evolving correctness. Additionally, for highly imbalanced datasets, accuracy and F1 score can be misleading, prompting reliance on metrics like MCC or informedness.
Historical Context
The concept of the confusion matrix originated in human perceptual studies, particularly in audiology, where matrices measure how a person hears specific words or sounds. Frank Rosenblatt, a pioneer in early neural networks, used confusion matrices to compare human and machine classifications. This historical link underscores the matrix's role in bridging human and machine performance evaluation.
See Also
- Machine learning
- Neural network
- Deep learning
- Artificial intelligence
- Supervised learning
- Unsupervised learning
- classification
- Precision and Recall
- matthews-correlation-coefficient