Exam-style
A payments team ships a fraud classifier and reports 99.2% accuracy on a held-out set in which 0.8% of transactions are fraudulent. Six weeks later the investigators say almost nothing reaches their queue, and the frauds they find themselves were never flagged. The accuracy figure has not moved. What is that number hiding, and what belongs on the report instead?
Reveal the answer
C. A model that answered 'not fraud' on every transaction would score 99.2% by base rate alone, so report recall and precision together, with the F1 score as the balance figure
Work out what a model that has learned nothing would score. Answer ‘not fraud’ every time and 99.2% of those answers are right, because 99.2% of the transactions are not fraud. The shipped model matches that, so the headline says only that it has reached the base rate, and it can do that while catching no fraud at all. Accuracy counts frauds flagged plus clean transactions left alone, over every prediction, so the pile of correctly ignored transactions swamps the cases that matter. Recall, TP/(TP+FN), is the share of real frauds flagged. It is what the investigators are complaining about. Precision, TP/(TP+FP), is the share of flags that turned out to be real fraud. F1 is the harmonic mean of the two, and stays low unless both are decent. A fresh sample of the same traffic carries the same base rate, so retraining changes nothing. Raising the threshold moves the wrong way: fewer flags means higher precision and lower recall. ROC AUC aggregates across every threshold, so it reports nothing about the one running in production. Its false-positive rate, FP/(FP+TN), divides by the enormous negative class. Where true negatives dominate, read the area under the precision-recall curve instead.
Q. A fraud classifier scores 99.2% accuracy on a set where 0.8% of transactions are fraudulent, and the investigators say nothing reaches them. What is the number hiding, and what goes on the report?
A. Answering “not fraud” every time also scores 99.2%, so the headline says only that the model matched the base rate. Report recall (real frauds caught) and precision (flags that were real), with the F1 score as the balance figure. Print the class balance beside any accuracy figure that survives.
Why? Sort the predictions into four groups: frauds caught, frauds missed, clean transactions wrongly flagged, clean transactions correctly ignored. Accuracy adds the first and the last, then divides by all four. When that last group is nearly all the data, it drowns out everything else. Recall counts the misses: lost money, a drained account. Precision counts the false alarms: investigator hours, and a frozen card belonging to somebody buying groceries. Moving the threshold trades one against the other. Weighing a missed fraud against a wrongly blocked payment is a business decision, not a modelling one; choosing a metric from the cost of being wrong works that through. A retrained model can lift the F1 score and the cost per inference together, which is why model metrics and business metrics belong on the same page.