Choosing the Metric: Precision, Recall and the Cost of a Wrong Prediction

Quick take. Pick the metric before you pick the model. In quality and operations work the classes are usually imbalanced, and the two kinds of error cost very different amounts, so accuracy is the wrong scoreboard. Write the costs down, check that your probabilities are calibrated, and set the threshold from the costs.

Why accuracy misleads

Take an illustrative inspection station that sees 10,000 parts a week, of which 100 (1%) are defective. A model that labels every part “good” is 99% accurate. It also catches zero defects.

That is the whole problem in one line. When one class is rare, accuracy mostly measures how well you predict the common class. Even the receiver operating characteristic (ROC) curve, the standard plot of true positive rate against false positive rate, can look reassuring on imbalanced data. Saito and Rehmsmeier showed that precision-recall plots give a more informative picture in that setting [1].

I build predictive models for decisions such as quality optimization and real-time operations, and in that work the metric question comes first. Here is how to answer it, step by step.

Four counts and the metrics built on them

Fix a threshold, call “defective” the positive class, and every prediction lands in one of four cells.

The metrics are ratios of these counts:

• precision = TP / (TP + FP). Of the parts you flagged, what share were really defective?
• recall = TP / (TP + FN). Of the real defects, what share did you catch?
• false positive rate = FP / (FP + TN). Of the good parts, what share did you wrongly flag?
• F1 = 2 × precision × recall / (precision + recall), the harmonic mean of precision and recall.

Precision answers the inspector’s question: is this alarm worth walking over for? Recall answers the quality manager’s question: how many defects escaped?

A worked example

Same illustrative week: 10,000 parts, 100 of them defective. The model outputs a probability of defect for each part, and you choose the threshold above which a part is pulled for re-inspection.

Lowering the threshold buys recall with precision. F1 is highest at 0.50 (0.70, against 0.68 at 0.25 and 0.60 at 0.10). Does that make 0.50 the right choice? The table cannot tell you. Only the costs can.

Put a price on each mistake

Suppose, again with illustrative numbers, that:

• a false positive costs $50, because a good part is pulled and an inspector spends time on it;
• a false negative costs $450, because a defect reaches assembly or the customer and triggers rework, scrap or a warranty claim;
• correct decisions cost nothing extra, because cost is measured relative to doing the right thing.

Weekly cost = FP × 50 + FN × 450.

The threshold F1 preferred is the most expensive one. The cheapest is 0.10.

The cost-weighted threshold

You do not have to sweep thresholds by hand. Elkan derived the optimal two-class decision rule from a cost matrix [2]. Here is the mechanical version for one part with predicted defect probability p:

1. Expected cost if you pass the part = p × 450, since it is defective with probability p.
2. Expected cost if you pull the part = (1 - p) × 50, since it is good with probability 1 - p.
3. Pull it when the second cost is smaller: (1 - p) × 50 < p × 450, which simplifies to p > 50 / 500 = 0.10.

In general, when correct decisions cost nothing:

threshold = cost of a false positive / (cost of a false positive + cost of a false negative)

Elkan’s general form also handles non-zero costs for correct decisions [2]. His practical advice is worth repeating: train the classifier on the data as given, then make cost-sensitive decisions explicitly from its probability estimates [2].

The catch: calibration

The formula assumes the probabilities mean what they say. Among parts scored at 0.10, about 10% should turn out defective. Many models do not deliver that out of the box:

• Niculescu-Mizil and Caruana found that boosted trees push predicted probabilities away from 0 and 1, and that Platt scaling or isotonic regression corrects the distortion [3].
• Guo and colleagues found that modern deep neural networks are often poorly calibrated, and that temperature scaling, a one-parameter form of Platt scaling, works well on many datasets [4].

To check, build a reliability table on held-out data. Bin the scores (0-0.05, 0.05-0.10 and so on), then compare the mean score in each bin with the observed defect rate. If they diverge, recalibrate before you apply the cost threshold.

Compare models on the right curve

When positives are rare, compare candidate models on precision-recall curves, then choose the operating point with costs. Two results from Davis and Goadrich are useful here [5]:

• one curve dominates another in ROC space if and only if it dominates in precision-recall space;
• an algorithm that optimizes area under the ROC curve is not guaranteed to optimize area under the precision-recall curve.

So the model with the better ROC area is not automatically the better model for a rare-defect problem.

Thresholds go stale

A threshold is a decision, not a property of the model. Sculley and colleagues warn that manually set thresholds can become invalid when a model is updated on new data, and suggest learning thresholds by evaluation on held-out validation data [6]. Costs move too: a new customer contract can change the price of an escaped defect overnight.

Checklist before you ship a classifier

1. Name the positive class and write down what each error costs, in dollars or hours, with the people who pay those costs.
2. Report the four counts plus precision and recall, not accuracy alone.
3. Compare models on precision-recall curves when positives are rare.
4. Check calibration with a reliability table, and recalibrate if needed.
5. Set the threshold from costs, then confirm the expected cost on held-out data.
6. Re-derive the threshold whenever costs, base rates or the model change.

The metric is where the business and the model meet. Get it right first, and choosing the model becomes a much easier conversation.

References

1. T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS ONE, vol. 10, no. 3, e0118432, 2015.
2. C. Elkan, “The foundations of cost-sensitive learning,” in Proc. 17th Int. Joint Conf. Artificial Intelligence (IJCAI), 2001, pp. 973-978.
3. A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” in Proc. 22nd Int. Conf. Machine Learning (ICML), 2005, pp. 625-632.
4. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Machine Learning (ICML), PMLR vol. 70, 2017, pp. 1321-1330.
5. J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves,” in Proc. 23rd Int. Conf. Machine Learning (ICML), 2006, pp. 233-240.
6. D. Sculley et al., “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015, pp. 2503-2511.