Picking an Evaluation Metric From the Cost of Being Wrong

The situation

The support assistant has been live for seven months. A message arrives and an abuse filter decides whether it reaches the queue at all. Past that gate, a retriever pulls the relevant policy passages, a model drafts a reply, and an extractor lifts the refund amount and the order date out of the message body. Anything asking for money back is scored by a second classifier, which decides whether a human reviews the refund before it gets paid. When a ticket escalates, the model writes a handover note for the person picking it up.

Traffic runs at about 50,000 messages a week. Roughly 1.2% are abusive enough that the filter should stop them. About 8,000 of those messages are refund requests, and roughly 0.8% of those are fraudulent.

The weekly quality report has one line in it: 97% accuracy. It has sat between 96% and 98% every week since launch, including the week finance noticed that refund losses had roughly tripled.

What actually matters

A number scores one component, and this system has five that fail in different ways and cost different amounts when they do. Averaging them produces a figure that cannot move enough to be a warning. The abuse filter and the refund classifier both say “no” to almost everything, both are right almost every time by sheer base rate, and both dominate a blended figure through volume alone. So the first move is to say which component is being scored and what it actually emits. A label, a ranked list of passages, a number, and a paragraph of free text are four different measurement problems with four different families of answer.

For anything emitting a label, everything derives from four cells: the ones it flagged and should have, the ones it flagged and should not have, the ones it let through and should not have, and the ones it correctly left alone. Two questions settle which cell you care about. What is the positive class, meaning the thing being detected? And which mistake costs more, a false alarm or a miss? For the abuse filter, a false alarm silences a paying customer who wrote an angry but legitimate message, and they do not get a second chance to reach support; a miss puts one nasty message in a queue an agent is reading anyway. For the refund classifier, a miss pays out money that never comes back; a false alarm costs four minutes of a reviewer’s time. Same shape, opposite answers. The wording of the two situations mirrors almost exactly, so the wording is not the tell. The cost is.

Class balance decides whether “how often was it right” carries any information at all. At a 1.2% positive rate, a component that answers “no” to every message scores 98.8% and catches nothing. That is roughly the score the weekly report has been showing, and it is why the report survived a tripling of refund losses without a wobble. Once the rare event is the thing you are trying to find, the overall hit rate is a comfort number.

For free text there is no confusion matrix, so two other properties do the filtering. Is there a human-written reference to compare against, and is the failure you fear about wording or about meaning? An invented refund figure in a handover note reads perfectly and overlaps the reference beautifully, which means no amount of word-matching will see it. Whether the claims in a generated answer are supported by the passages it was given is a separate measurement from whether it resembles a good answer.

What we’ll filter on

1. Output shape. Does the component emit a label, a ranked list, a number, or free text?
2. Which error costs more. False alarm or miss, and roughly by what ratio in money or time.
3. Class balance. How rare is the positive class in the data being scored?
4. Reference availability. Is there a human-written answer to compare against, or nothing but the output itself?
5. Threshold or curve. Are we tuning one operating point, or comparing candidates across all of them?

The landscape

Accuracy. Correct predictions over all predictions: (TP + TN) / (TP + TN + FP + FN). Honest when the classes are near balanced and both errors cost about the same. Useless the moment the positive class is rare, for the reason above.

Precision. TP / (TP + FP). Of everything the component flagged, the share that was actually positive. The denominator is what the model claimed. It moves when false alarms move, so it is the number to gate on when a false alarm is the expensive error. A one-line anchor that survives exam pressure: precision counts predictions.

Recall, also called sensitivity or the true positive rate. TP / (TP + FN). Of everything that was actually positive, the share the component caught. The denominator is reality, not the model’s opinion of it. Gate on this when a miss is the expensive error. Anchor: recall counts reality.

Specificity, the true negative rate. TN / (TN + FP). Recall for the negative class: of everything that was genuinely fine, how much was left alone. Rarely the headline number, and worth recognising as recall’s mirror rather than as a separate idea.

F1. The harmonic mean of precision and recall, 2 × (P × R) / (P + R). Reach for it when both errors cost roughly the same and the data is imbalanced enough to rule accuracy out. The harmonic mean punishes lopsidedness: 0.99 precision with 0.10 recall scores 0.18, where an arithmetic mean would flatter it at 0.55.

AUC-ROC. The area under the true-positive-rate against false-positive-rate curve, swept across every threshold. It measures how well the component ranks positives above negatives, independent of where the threshold happens to sit, so it answers “is this model better than that one” rather than “is this setting right”. 1.0 separates the classes perfectly, 0.5 is a coin toss. Under heavy imbalance the precision-recall curve is the more informative sweep, because the false-positive rate barely moves when negatives outnumber positives a hundred to one.

Ranked-list metrics. Recall@k, precision@k, MRR, and NDCG@k score a retriever against a labelled relevance set rather than a single label. Recall@k is usually the first one to read, since a passage that never entered the context window was never available to the generator. Scoring retrieval and generation separately is its own decision and covered on its own terms.

MAE. Mean absolute error, the average of |predicted - actual|, in the same units as the thing being predicted. Every error counts in proportion to its size, so a handful of large misses barely shift it.

RMSE. Root mean squared error, also in the target’s units, but squaring before averaging makes one large miss worth many small ones. Choose it over MAE when a single big error genuinely hurts more than a scattering of small ones. The gap between the two is itself a signal: MAE of a few dollars alongside an RMSE of tens of dollars means a small number of severe misses hiding under a healthy average.

MAPE. Mean absolute percentage error, scale-independent, which makes errors comparable across data sets with different magnitudes. It falls apart as actual values approach zero, where a trivial absolute error becomes an enormous percentage.

R². The share of variance in the target the model explains. 1.0 is perfect, 0 is no better than always predicting the mean, and negative is worse than that.

BLEU. Precision-oriented n-gram overlap against one or more reference translations. The name carries its use case: bilingual evaluation understudy, built for machine translation, where the acceptable outputs are tightly constrained.

ROUGE. Recall-oriented overlap against reference summaries, in n-gram and longest-common-subsequence flavours (ROUGE-N, ROUGE-L). Built for summarisation, where the question is how much of the reference’s content survived.

BERTScore. Similarity computed over contextual token embeddings rather than exact word matches, so a good paraphrase scores well where BLEU and ROUGE would punish it. Still reference-based, but comparing meaning rather than wording.

Perplexity. The exponentiated average negative log-likelihood of a text sample under the model. Lower is better. It is the odd one out in this group because no reference output is involved at all: it measures how well a language model predicts text, not whether an answer is right.

Judged dimensions for generated answers. Faithfulness, or groundedness, asks whether the claims in an answer are supported by the retrieved context. Answer relevance asks whether the response addresses the question. Context relevance scores the retrieved chunks themselves, so it grades the retriever rather than the generator. These come from a rubric applied by a judge or a managed evaluation, not from counting words, which is exactly why they catch the failure that overlap cannot: a fluent answer that contradicts its own sources scores well on ROUGE and badly on faithfulness.

Evaluation

Side by side

Two columns do most of the work. “Needs a reference” splits the text metrics into the ones you can only run against human-written answers and the two you can run on production traffic, which is why faithfulness and perplexity are the ones that survive contact with live output. “Survives imbalance” retires accuracy from every rare-event component in this system, which is all of them.

Reading the metric off the output shape

The mirrored pairs

The dangerous situations are the ones whose wording is nearly identical and whose answers are opposite. Working each one through the two questions takes about ten seconds and gets it right; recognising a familiar sentence shape gets it wrong about half the time.

The first two rows are the pair that catches people. Both are asymmetric-cost situations, both are about filtering unwanted things, and they point at opposite metrics because the expensive error is on opposite sides. The last two definitional rows are worth reading aloud until the denominators stick: “of the ones we flagged” is precision, “of the ones that existed” is recall.

The solution

Retire the single accuracy line and give each component the metric that matches its costly error.

The abuse filter reports precision, gated, with recall alongside. Blocking a real customer is the expensive failure, so precision is the number with a threshold on it and the one that pages someone when it drops. Recall goes in the report next to it, because a precision target is trivially satisfiable by flagging nothing, and a pair of numbers makes that visible. The classifier’s score threshold is the knob that trades one against the other, so it gets set from the cost ratio rather than left at 0.5 because that is the default.

The refund classifier reports recall, gated by review capacity. A miss pays out cash; a false alarm buys four minutes of review. So recall is the target and precision becomes a budget constraint. Choose the recall you need, read the precision that falls out at that threshold, then multiply the flagged count by the cost of a review and check it against what the queue can absorb. This is the component that has been failing silently, and it fails in a way accuracy structurally cannot show.

The retriever reports recall@k first. Whether the passage that answers the question arrived at all caps everything downstream, so it leads, with precision@k and MRR next to it for dilution and ordering.

The extractor reports exact match on the date and both MAE and RMSE on the amount. Dates are either right or wrong, so a match rate is the whole story. Amounts are numbers where one wildly wrong figure matters more than many small roundings, so RMSE is the gate and MAE sits beside it. The gap between them is the early warning that a few extreme misses are hiding under a healthy average. MAPE is the wrong shape here, because refund amounts run down to a few dollars and the percentage error explodes at the bottom of the range.

The handover note reports faithfulness and a judged rubric, with ROUGE as a tripwire only. There is no reference note for live traffic, so overlap metrics have nothing to compare against outside a golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions.. Faithfulness against the retrieved policy passages and the ticket body catches the invented refund figure, which is the failure that actually costs something. ROUGE against the golden set still has a job as a regression tripwire when a prompt changes, and a judge scoring a rubric carries the dimensions overlap cannot reach.

On the managed side, a Bedrock evaluation job computes the reference-based text metrics for you by task type, and its automatic evaluation scores accuracy, robustness, and toxicity. Human evaluation with a workteam you bring covers the subjective dimensions, and RAG evaluation for Knowledge Bases covers the retrieval and faithfulness side. The classifier metrics in this system are not part of that: they come from your own labelled set and a few lines of arithmetic over the four cells, which is where a properly built labelled set pays for itself. The bias and safety scores in fmeval are a separate measurement with separate probes.

Two gotchas worth carrying. F1 hides which half is bad, so publish precision and recall beside it rather than on their own tab. And every classification number in this list moves when the threshold moves, so a metric reported without the threshold it was measured at is not reproducible next week.

Worked example

One week of production traffic, labelled by hand for the audit.

The abuse filter

50,000 messages · 600 genuinely abusive (1.2%)

                    Actually abusive    Actually fine
  Flagged                    480              900
  Let through                120           48,500

  accuracy    = (480 + 48,500) / 50,000   = 0.980
  precision   = 480 / (480 + 900)         = 0.348
  recall      = 480 / (480 + 120)         = 0.800
  F1          = 2(0.348)(0.800)/1.148     = 0.485

98% accuracy, and 900 paying customers had a legitimate message silenced this week. Precision is 0.348, meaning roughly two in three blocks are wrong, and the customers on the wrong end of them do not get a second attempt to reach support. Raising the threshold until precision reaches 0.75 drops recall to about 0.55, which puts around 270 abusive messages into a queue an agent is reading anyway. That trade is worth taking here, and it is invisible in the accuracy figure, which moves from 0.980 to 0.992.

The refund classifier

8,000 refund requests · 64 fraudulent (0.8%)

                    Actually fraud    Actually legitimate
  Flagged                     26                    190
  Paid out                    38                  7,746

  accuracy    = (26 + 7,746) / 8,000      = 0.972
  precision   = 26 / (26 + 190)           = 0.120
  recall      = 26 / (26 + 38)            = 0.406

The same 97% that has headlined the weekly report all year. Recall is 0.406, so 38 fraudulent refunds were paid, at an average of AUD$140, which is AUD$5,320 gone in a week. Reviews cost about AUD$3 of agent time each, so the current 216 flags cost AUD$648.

Drop the threshold until recall reaches 0.80. Now 51 of the 64 are caught and 13 are paid, so losses fall to AUD$1,820. Precision degrades to roughly 0.07 at that setting, which means about 729 flags a week and AUD$2,187 of review time. Spending an extra AUD$1,539 on review to stop AUD$3,500 of loss is the right way round, and the arithmetic is the argument. Accuracy falls from 0.972 to 0.914 when this change is made, which is exactly why it was the wrong number to report.

The handover note

Golden set of 200 escalations, two candidate prompts

  ROUGE-L         prompt A 0.44    prompt B 0.41
  BERTScore       prompt A 0.891   prompt B 0.887
  Faithfulness    prompt A 0.94    prompt B 0.71

On overlap alone, prompt A wins narrowly and prompt B looks like a close second. Faithfulness says prompt B invents unsupported content in nearly three notes in ten, mostly refund amounts and dates it inferred rather than read. Two of those notes went out to agents who acted on the figure. No amount of word overlap with a reference could have surfaced that, because the invented amounts are the same shape as real ones and sit in otherwise well-formed sentences. The invented detailHallucinationAn LLM stating something false with the same confidence it states something true. is a content failure, and only a metric that reads the source can see it.

What’s worth remembering

1. Name the positive class and price the two errors before naming a metric. Mirrored situations are worded almost identically and answer oppositely, so the wording is never the tell.
2. Precision counts predictions and recall counts reality. Whichever error costs more picks the denominator you gate on: false alarms mean precision, misses mean recall.
3. A rare positive class rules accuracy out entirely, and F1 is the fallback when both errors cost about the same. AUC-ROC compares candidate models across every threshold rather than judging one operating point.
4. RMSE when a single large miss hurts more than many small ones, MAE when it does not, MAPE only when the values stay clear of zero. Reporting MAE and RMSE together exposes outliers hiding under a healthy average.
5. BLEU is translation, ROUGE is summarisation, BERTScore is meaning rather than wording, and perplexity is the one that needs no reference at all.
6. Word overlap with a reference cannot see whether a claim is supported. That is faithfulness, judged against the passages the answer was built from, and it is the metric for a fluent answer that contradicts its own sources.