A precision-recall curve is a graphical representation used in classification to evaluate the performance of a model by plotting precision (positive predictive value) on the y-axis against recall (sensitivity) on the x-axis for various decision thresholds. Each point on the curve corresponds to a specific threshold setting, showing how precision and recall change as the threshold is adjusted. The curve is particularly useful for imbalanced datasets where one class is rare, as it focuses on the positive class performance rather than the true negatives.
Precision is defined as the fraction of relevant instances among the retrieved instances, or equivalently, the number of true positives divided by the sum of true positives and false positives. Recall is the fraction of relevant instances that were retrieved, or the number of true positives divided by the sum of true positives and false negatives. In a classification task, a perfect classifier would have both precision and recall equal to 1.0, yielding a curve that passes through the top-right corner. However, in practice, there is often an inverse relationship: increasing recall tends to decrease precision, and vice versa. A precision-recall curve typically slopes downward from a high precision at low recall to a lower precision at high recall, though the exact shape depends on the model and data.
Relationship to Type I and Type II Errors
Precision and recall are closely tied to hypothesis testing concepts. Recall is the complement of the type II error rate (false negative rate), meaning that high recall corresponds to few missed positive instances. Precision is related to the type I error rate (false positive rate) but also depends on the prior distribution of positive and negative instances. For example, in a dataset with ten cats and twelve dogs, a dog recognition program that identifies eight dogs, of which five are actual dogs, has a precision of 5/8 and a recall of 5/12. The type I error rate is 3/10 (three false positives among ten cats), and the type II error rate is 7/12 (seven false negatives among twelve dogs). Precision can be seen as a measure of quality (how many selected items are relevant), while recall is a measure of quantity (how many relevant items are selected).
Trade-offs and Practical Examples
The trade-off between precision and recall is often context-dependent. For instance, a smoke detector is designed to prioritize recall, because the cost of missing a fire (a false negative) is catastrophic, even if it means many false alarms (low precision). Conversely, the criminal justice system, guided by Blackstone's ratio - "It is better that ten guilty persons escape than that one innocent suffer" - prioritizes precision, accepting lower recall to avoid convicting innocent people. A brain surgeon removing a tumor faces a similar dilemma: removing a larger area increases recall (ensuring all cancer cells are removed) but reduces precision (removing healthy tissue), while a conservative approach increases precision but risks leaving cancer cells behind.
Construction and Interpretation
To construct a precision-recall curve, a model outputs a score or probability for each instance. By varying the threshold above which an instance is classified as positive, one obtains different pairs of precision and recall values. These pairs are plotted as a curve. The area under the precision-recall curve (PR-AUC) is a single-number summary of the curve, with higher values indicating better overall performance. A random classifier typically produces a curve that is a horizontal line at the proportion of positive instances in the dataset, so the PR-AUC for a useful model should be above this baseline. The curve is especially informative when the positive class is rare, as the confusion matrix alone may be misleading.
Use in Machine Learning
Precision-recall curves are widely used in Machine learning and Artificial intelligence to evaluate binary classifiers, particularly in domains such as information retrieval, medical diagnosis, and fraud detection. They are often compared to receiver operating characteristic (ROC) curves, which plot the true positive rate against the false positive rate. Unlike ROC curves, precision-recall curves do not include true negatives, making them more sensitive to performance on the positive class. Many Neural network and Deep learning frameworks provide built-in functions to compute and plot precision-recall curves, facilitating model selection and threshold tuning. For example, in Natural language processing tasks like document classification, a precision-recall curve can help balance the retrieval of relevant documents against the inclusion of irrelevant ones.
Limitations and Considerations
Precision-recall curves have limitations. They are sensitive to class imbalance, and a model with high precision and recall on a balanced dataset may perform poorly on an imbalanced one. Additionally, the curve does not convey the absolute number of instances, so comparing curves across different datasets can be misleading. It is also possible to achieve perfect recall by retrieving every instance, but such a model would have low precision. Conversely, perfect precision can be achieved by selecting only the most confident instances, but recall would be low. Therefore, precision and recall are rarely discussed in isolation; instead, the curve provides a comprehensive view of the trade-off, allowing practitioners to choose a threshold that aligns with the application's priorities.