Precision and recall are two fundamental performance metrics used in classification and information retrieval to evaluate the quality of a model's predictions. Precision, also called positive predictive value, measures the fraction of relevant instances among all retrieved instances. Recall, also known as sensitivity, measures the fraction of relevant instances that were successfully retrieved. Both metrics are based on the concept of relevance, which is typically defined by the specific task, such as identifying a particular class in a dataset.
In a classification task, precision for a given class is the number of true positives (items correctly labeled as belonging to the class) divided by the total number of items labeled as belonging to that class, which is the sum of true positives and false positives. Recall is the number of true positives divided by the total number of items that actually belong to the class, which is the sum of true positives and false negatives. These definitions align with the general formulas: precision equals relevant retrieved instances divided by all retrieved instances, and recall equals relevant retrieved instances divided by all relevant instances.
Consider a computer program designed to recognize dogs in digital photographs. If a picture contains ten cats and twelve dogs, and the program identifies eight elements as dogs, of which five are actually dogs (true positives) and three are cats (false positives), then seven dogs are missed (false negatives) and seven cats are correctly excluded (true negatives). The program's precision is 5/8, since only five of the eight selected elements are relevant. Its recall is 5/12, because only five of the twelve relevant dogs are retrieved. This example illustrates how precision and recall capture different aspects of performance.
Relationship to Hypothesis Testing
Precision and recall can be understood through the lens of hypothesis testing. In a typical setup, the null hypothesis is that a given item is irrelevant (e.g., not a dog). A type I error occurs when a relevant item is incorrectly rejected, which corresponds to a false positive. A type II error occurs when an irrelevant item is incorrectly accepted, corresponding to a false negative. Perfect precision (no false positives) is equivalent to having no type I errors, while perfect recall (no false negatives) is equivalent to having no type II errors.
More generally, recall is the complement of the type II error rate, meaning recall equals one minus the false negative rate. Precision is related to the type I error rate but in a more complex way, as it also depends on the prior distribution of relevant versus irrelevant items in the population. In the dog example, there are three type I errors (false positives) out of ten total cats, giving a type I error rate of 3/10. There are seven type II errors (false negatives) out of twelve dogs, giving a type II error rate of 7/12.
Precision as Quality, Recall as Quantity
Precision is often viewed as a measure of quality, indicating how many of the retrieved items are actually relevant. High precision means that an algorithm returns more relevant results than irrelevant ones. Recall, on the other hand, is a measure of quantity, indicating how many of the relevant items are retrieved. High recall means that an algorithm returns most of the relevant results, whether or not irrelevant ones are also returned.
These metrics are not particularly useful in isolation. For instance, it is possible to achieve perfect recall by simply retrieving every single item in the dataset. Conversely, perfect precision can be achieved by selecting only a very small number of extremely likely items, even if many relevant items are missed. Therefore, precision and recall are typically evaluated together.
In a classification task, a precision score of 1.0 for a class C means that every item labeled as belonging to class C indeed belongs to that class, but it says nothing about how many items from class C were not labeled correctly. A recall of 1.0 means that every item from class C was labeled as belonging to class C, but it says nothing about how many items from other classes were incorrectly labeled as class C.
The Precision-Recall Tradeoff
Often, there is an inverse relationship between precision and recall. Increasing one typically reduces the other, but the context of the application may dictate which metric is more valued. For example, a smoke detector is generally designed to commit many type I errors, alerting in situations where there is no danger, because the cost of a type II error (failing to sound an alarm during a major fire) is prohibitively high. Smoke detectors are therefore designed with recall in mind, catching all real dangers even at the cost of many false alarms.
In contrast, the criminal justice system often emphasizes precision, as reflected in Blackstone's ratio: "It is better that ten guilty persons escape than that one innocent suffer." This principle highlights the cost of a type I error (convicting an innocent person). As such, the system is geared toward precision, avoiding wrongful convictions even at the cost of letting more guilty people go free, which reduces recall.
A brain surgeon removing a cancerous tumor illustrates the tradeoff as well. The surgeon must remove all tumor cells to prevent regeneration, but must also avoid removing healthy brain cells to preserve brain function. If the surgeon is more liberal in the area of the brain they remove, they increase recall (removing all cancer cells) but reduce precision (removing healthy cells). If the surgeon is more conservative, they increase precision but reduce recall. Greater recall increases the chances of removing all cancer cells but also increases the risk of removing healthy cells. Greater precision decreases the risk of removing healthy cells but also decreases the chances of removing all cancer cells.
Precision-Recall Curves and Combined Metrics
In practice, precision and recall scores are not discussed in isolation. A precision-recall curve plots precision as a function of recall, typically showing that precision decreases as recall increases. This curve provides a comprehensive view of a model's performance across different thresholds. Alternatively, values for one measure can be compared for a fixed level of the other, such as precision at a recall level of 0.75.
Several combined metrics incorporate both precision and recall. The F1 score, the harmonic mean of precision and recall, is widely used to summarize a model's performance into a single number. The F1 score is particularly useful when the class distribution is imbalanced, as it balances the tradeoff between precision and recall. Other metrics, such as the area under the precision-recall curve, are also used to evaluate models, especially in Machine learning tasks where positive classes are rare.
Applications in Machine Learning
Precision and recall are essential in many Machine learning applications, particularly in Artificial intelligence systems where classification errors have different consequences. For example, in Large language model evaluation, precision and recall can be used to assess the relevance of generated responses or retrieved documents. In Neural network based classifiers, these metrics guide the selection of decision thresholds and the tuning of model parameters.
In imbalanced classification problems, such as fraud detection or medical diagnosis, accuracy is often misleading because the majority class dominates. Precision and recall provide a more nuanced view, allowing practitioners to optimize for the specific costs of false positives and false negatives. Techniques such as Data Augmentation and Loss Functions are often employed to improve precision and recall in such scenarios.
Limitations and Considerations
While precision and recall are powerful, they have limitations. They do not account for true negatives, which can be important in some contexts. For example, in a binary classification problem with a large number of negative instances, a model with high precision and recall might still have many false positives, which could be problematic. Additionally, precision and recall are sensitive to the choice of the positive class, and the definitions can be ambiguous in multi-class settings.
Furthermore, precision and recall are not directly comparable across different datasets or tasks, as they depend on the base rate of the positive class. A model with high recall on a rare disease dataset may not perform well on a common condition. Therefore, it is important to consider the context and the costs associated with different types of errors when interpreting these metrics.