A fast pass over every metric family in the track, sorted by what the component being scored actually emits: a label, a ranked list, a number, or free text. Picking an Evaluation Metric From the Cost of Being Wrong walks the full decision process behind this table; this is the fast-lookup version.
Label metrics at a glance
| Metric | Formula | Gate on it when |
|---|---|---|
| Accuracy | (TP + TN) / all | Classes are balanced and both errors cost about the same |
| Precision | TP / (TP + FP) | A false alarm is the expensive error; counts predictions |
| Recall (sensitivity, TPR) | TP / (TP + FN) | A miss is the expensive error; counts reality |
| Specificity (TNR) | TN / (TN + FP) | You need the share of the genuinely fine left alone; recall’s mirror, rarely the headline |
| F1 | 2 × (P × R) / (P + R) | Positive class is rare AND both errors cost about the same |
| AUC-ROC | area under TPR vs FPR, swept across every threshold | Comparing candidate models, not tuning one operating point |
Ranked-list metrics at a glance
| Metric | What it scores | Read it when |
|---|---|---|
| Recall@k | Whether the right passage arrived in the top k at all | First: a passage that never arrived was never available downstream |
| Precision@k | How much of the top k was actually relevant | Dilution: too much noise crowding the good result |
| MRR | Where the first relevant result landed | One right answer matters more than the whole ranking |
| NDCG@k | Graded relevance, position-weighted | Relevance is not binary and order matters throughout the list |
Regression metrics at a glance
| Metric | Formula | Reach for it when |
|---|---|---|
| MAE | mean of |predicted − actual| | Every error should count in proportion to its size |
| RMSE | root of the mean squared error | One large miss should hurt more than many small ones |
| MAPE | mean absolute percentage error | Comparing across scales, values stay clear of zero |
| R² | share of variance explained | Judging a regression against a naive “always predict the mean” baseline |
Text and GenAI metrics at a glance
| Metric | Scores | Needs a reference |
|---|---|---|
| BLEU | Precision-oriented n-gram overlap | Reference translations |
| ROUGE | Recall-oriented n-gram / LCS overlap | Reference summaries |
| BERTScore | Embedding similarity, tolerant of paraphrase | Any reference text |
| Perplexity | Fluency; how well the model predicts the text, lower is better | None |
| Faithfulness / groundedness | Claims supported by the retrieved context | None (needs the context, not a reference) |
| Answer relevance | Whether the response addresses the question | None |
| Context relevance | Quality of the retrieved chunks themselves | None |
Decision rules
- Name the output shape first: a label, a ranked list, a number, or free text. Each shape has its own metric family, and the families do not substitute for each other.
- For a label, the denominator rule decides the metric: whichever error costs more picks which count you gate on. False alarms cost more → precision. Misses cost more → recall.
- F1 needs both conditions at once: the positive class must be rare enough to rule accuracy out, and the two errors must cost about the same. Imbalance alone does not select F1; it only eliminates accuracy.
- Two situations that read almost identically point at opposite metrics: “missing a case costs more” means recall; “a false alarm costs more” means precision. The wording rarely differs; the cost does.
- AUC-ROC answers “is this model better than that one”, not “is this threshold right”; use it to compare candidates, not to set an operating point. Under heavy imbalance, sweep the precision-recall curve instead: the false-positive rate barely moves when negatives outnumber positives a hundred to one.
- For a retriever, read Recall@k first. A passage that never made the top k was never available to the generator, whatever the ordering metrics say afterwards.
- For a number, ask whether one severe miss should dominate the score. RMSE if yes, MAE if no, and report both: a wide gap between them means a few severe misses hiding under a healthy average. Reserve MAPE for cross-scale comparisons where values stay well clear of zero.
- For free text, ask first whether a human-written reference exists. No reference rules out BLEU, ROUGE, and BERTScore, leaving perplexity (fluency) and faithfulness (grounding) as the metrics that survive contact with live traffic.
- BLEU is translation, ROUGE is summarisation. Both are overlap metrics and both miss a fluent answer that invents facts; only faithfulness catches that, because it comes from a judged rubric read against the source rather than from counting words.
- Report precision and recall alongside F1, never F1 alone. F1 hides which half is failing.
- Report every classification metric with the threshold it was measured at. A number without its threshold cannot be reproduced next week.
Traps
- Reaching for accuracy on a rare positive class. A component that says “no” to everything scores in the high nineties and catches nothing.
- Treating imbalance alone as the signal for F1. Imbalance rules accuracy out; the cost of the two errors still decides between precision, recall, and F1.
- Assuming the wording of a scenario tells you the metric. Mirrored situations are worded almost identically and answer oppositely; only the cost of the error decides.
- Reading precision@k or NDCG before checking Recall@k. Ordering metrics cannot fix a passage that never made the top k; Recall@k caps everything else.
- Applying BLEU or ROUGE to production traffic with no reference to compare against. Overlap metrics need a human-written answer; faithfulness and perplexity are the ones that do not.
- Trusting ROUGE or BLEU to catch a fabricated fact. A fluent, well-formed invention overlaps the reference beautifully and scores well on both; only faithfulness reads whether the claim is actually supported.
- Reporting an F1 score with no precision or recall beside it, which hides exactly which error is happening.
- Comparing models on accuracy when the real question is “which one ranks positives above negatives better”, which is what AUC-ROC is for.
- Using MAPE near zero-valued targets. A trivial absolute error turns into an enormous percentage and the metric stops meaning anything.
Say it in one line
- Output shape first: label, ranked list, number, or free text picks the metric family before anything else does.
- Precision counts predictions, recall counts reality; whichever error costs more picks the denominator.
- F1 needs imbalance to rule out accuracy AND roughly equal error costs; imbalance alone is not enough to select it.
- AUC-ROC compares candidate models across every threshold; it does not judge one operating point.
- Recall@k leads for a retriever, because a passage that never arrived caps everything ranking metrics can do afterwards.
- RMSE when a single big miss should dominate, MAE when every error should count equally, MAPE only away from zero.
- BLEU is translation, ROUGE is summarisation, BERTScore tolerates paraphrase, and perplexity is the one of the four that needs no reference at all.
- Faithfulness reads whether a claim is supported by the source; it is the metric that catches a fluent invention overlap metrics reward.
- Always report precision and recall beside F1, and always report the threshold a classification metric was measured at.