A fast pass over every metric family in the track, sorted by what the component being scored actually emits: a label, a ranked list, a number, free text, or an agent’s trajectory. Picking an Evaluation Metric From the Cost of Being Wrong walks the full decision process behind this table; this is the fast-lookup version.
Label metrics at a glance
| Metric | Formula | Gate on it when |
|---|---|---|
| Accuracy | (TP + TN) / all | Classes are balanced and both errors cost about the same |
| Precision | TP / (TP + FP) | A false alarm is the expensive error; counts predictions |
| Recall (sensitivity, TPR) | TP / (TP + FN) | A miss is the expensive error; counts reality |
| Specificity (TNR) | TN / (TN + FP) | You need the share of the genuinely fine left alone; recall’s mirror, rarely the headline |
| F1 | 2 × (P × R) / (P + R) | Positive class is rare AND both errors cost about the same |
| AUC-ROC | area under TPR vs FPR, swept across every threshold | Comparing candidate models, not tuning one operating point |
Ranked-list metrics at a glance
| Metric | What it scores | Read it when |
|---|---|---|
| Recall@k | Whether the right passage arrived in the top k at all | First: a passage that never arrived was never available downstream |
| Precision@k | How much of the top k was actually relevant | Dilution: too much noise crowding the good result |
| MRR | Where the first relevant result landed | One right answer matters more than the whole ranking |
| NDCG@k | Graded relevance, position-weighted | Relevance is not binary and order matters throughout the list |
Regression metrics at a glance
| Metric | Formula | Reach for it when |
|---|---|---|
| MAE | mean of |predicted − actual| | Every error should count in proportion to its size |
| RMSE | root of the mean squared error | One large miss should hurt more than many small ones |
| MAPE | mean absolute percentage error | Comparing across scales, values stay clear of zero |
| R² | share of variance explained | Judging a regression against a naive “always predict the mean” baseline |
Text and GenAI metrics at a glance
| Metric | Scores | Needs a reference |
|---|---|---|
| BLEU | Precision-oriented n-gram overlap | Reference translations |
| ROUGE | Recall-oriented n-gram / LCS overlap | Reference summaries |
| BERTScore | Embedding similarity, tolerant of paraphrase | Any reference text |
| Perplexity | Fluency; how well the model predicts the text, lower is better | None |
| Faithfulness / groundedness | Claims supported by the retrieved context | None (needs the context, not a reference) |
| Answer relevance | Whether the response addresses the question | None |
| Context relevance | Quality of the retrieved chunks themselves | None |
| Response consistency | Whether repeated runs of the same prompt agree with each other | None (needs repeated runs, not a reference) |
The four output dimensions this track keeps returning to sit on top of that table. Relevance is answer relevance. Factual accuracy is faithfulness, or a judged accuracy score read against the source. Consistency is response consistency across repeated runs of the same prompt. Fluency is perplexity. Naming the dimension a stakeholder cares about and then reading across to the metric is quicker than working backwards from a list of metric names.
What Amazon Bedrock calls them
Bedrock evaluations ship their own built-in metric names, and those strings are what a job selects. Custom metrics cover whatever the built-ins miss on judge and RAG jobs.
| Job type | Built-in metrics |
|---|---|
| Programmatic model evaluation | Accuracy, Robustness, Toxicity. Accuracy means a different calculation per task type: real-world knowledge score for general text generation, BERTScore for summarisation, F1 for question answering, and classification accuracy for text classification |
| Model evaluation with a judge model | Correctness, Completeness, Faithfulness, Helpfulness, Logical coherence, Relevance, Following instructions, Professional style and tone, Harmfulness, Stereotyping, Refusal |
| RAG evaluation, retrieve only | Context relevance, Context coverage. Only these two, and Context coverage needs a ground truth in the prompt dataset |
| RAG evaluation, retrieve and generate | Correctness, Completeness, Helpfulness, Logical coherence, Faithfulness, Citation precision, Citation coverage, Harmfulness, Stereotyping, Refusal |
Agent metrics at a glance
An agent emits a trajectory rather than an answer, so the metrics that score it read the run, not the reply.
| Metric | Scores | Read it when |
|---|---|---|
| Task completion rate | Share of N runs that reached the task’s defined end state | First, before anything else in this table |
| Tool selection accuracy | Whether the tools chosen were the ones the task needed | The wrong tool got called, or the right ones in the wrong order |
| Tool-argument validity | Whether arguments were well-formed and correct against the fixture | A right tool ran with a mangled date, identifier, or enumerated value |
| Steps to completion | Turns and tool calls used against a step budget | The answer is right by a long or wandering route |
| Recovery rate | Share of runs containing a failed tool call that completed the task anyway | A tool returned an error and the run carried on regardless |
| Cost and latency per completed task | Tokens, dollars, and wall-clock per finished task | Comparing agent configurations, or watching spend move |
Answer correctness scores the reply alone. An agent that reaches the right number through a tool that also moved money still scores well on it.
Amazon Bedrock AgentCore Evaluations scores these from the spans an agent exports to CloudWatch, and its built-in evaluators sit at three levels. Session level holds Builtin.GoalSuccessRate, the task completion score. Trace level holds correctness, faithfulness, helpfulness, response relevance, coherence, conciseness, instruction following and the safety set. Tool level holds Builtin.ToolSelectionAccuracy and a tool parameter accuracy evaluator, the two route metrics above.
Supply an expected tool sequence as ground truth and three session-level evaluators check the route without an LLM judge: Builtin.TrajectoryExactOrderMatch (same tools, same order, no extras), Builtin.TrajectoryInOrderMatch (in order, extras allowed between), and Builtin.TrajectoryAnyOrderMatch (all present, any order).
Decision rules
- Name the output shape first: a label, a ranked list, a number, free text, or a trajectory. Each shape has its own metric family, and the families do not substitute for each other.
- For a label, the denominator rule decides the metric: whichever error costs more picks which count you gate on. False alarms cost more → precision. Misses cost more → recall.
- F1 needs both conditions at once: the positive class must be rare enough to rule accuracy out, and the two errors must cost about the same. Imbalance alone does not select F1; it only eliminates accuracy.
- Two situations that read almost identically point at opposite metrics: “missing a case costs more” means recall; “a false alarm costs more” means precision. The wording rarely differs; the cost does.
- AUC-ROC answers “is this model better than that one”, not “is this threshold right”; use it to compare candidates, not to set an operating point. Under heavy imbalance, sweep the precision-recall curve instead: the false-positive rate barely moves when negatives outnumber positives a hundred to one.
- For a retriever, read Recall@k first. A passage that never made the top k was never available to the generator, whatever the ordering metrics say afterwards.
- For a number, ask whether one severe miss should dominate the score. RMSE if yes, MAE if no, and report both: a wide gap between them means a few severe misses hiding under a healthy average. Reserve MAPE for cross-scale comparisons where values stay well clear of zero.
- For free text, ask first whether a human-written reference exists. No reference rules out BLEU, ROUGE, and BERTScore, leaving perplexity (fluency) and faithfulness (grounding) as the two that work on live traffic.
- BLEU is translation, ROUGE is summarisation. Both count word overlap, so a fluent answer full of unsupported claims scores well on either. Only faithfulness catches it, because it comes from a judged rubric read against the source rather than from counting words.
- For an agent, read task completion rate before any route metric. Tool selection accuracy of 100% across the four steps of a run that then stalled is a score computed over a run that produced nothing, and it reads as reassurance.
- Report precision and recall alongside F1, never F1 alone. F1 hides which half is failing.
- Report every classification metric with the threshold it was measured at. A number without its threshold cannot be reproduced next week.
Traps
- Reaching for accuracy on a rare positive class. A classifier that predicts the negative class every time scores 99% on a dataset that is 1% positive and catches nothing.
- Treating imbalance alone as the signal for F1. Imbalance rules accuracy out; the cost of the two errors still decides between precision, recall, and F1.
- Assuming the wording of a scenario tells you the metric. Mirrored situations are worded almost identically and answer oppositely; only the cost of the error decides.
- Reading precision@k or NDCG before checking Recall@k. Ordering metrics cannot fix a passage that never made the top k; Recall@k caps everything else.
- Applying BLEU or ROUGE to production traffic with no reference to compare against. Overlap metrics need a human-written answer; faithfulness and perplexity are the ones that do not.
- Trusting ROUGE or BLEU to catch a fabricated fact. A fluent, well-formed fabrication overlaps the reference heavily and scores well on both; only faithfulness reads whether the claim is supported.
- Reporting an F1 score with no precision or recall beside it, which hides exactly which error is happening.
- Comparing models on accuracy when the real question is “which one ranks positives above negatives better”, which is what AUC-ROC is for.
- Scoring an agent with a prompt-in, answer-out evaluation job. Those jobs score the final response only, so a wrong tool, a mangled argument, an unretried error, and a clean run all come back with the same number. Trajectory scoring needs a trace-based evaluator.
- Using MAPE near zero-valued targets. A trivial absolute error turns into an enormous percentage and the metric stops meaning anything.
Say it in one line
- Output shape first: label, ranked list, number, free text, or trajectory picks the metric family before anything else does.
- Precision counts predictions, recall counts reality; whichever error costs more picks the denominator.
- F1 needs imbalance to rule out accuracy AND roughly equal error costs; imbalance alone is not enough to select it.
- AUC-ROC compares candidate models across every threshold; it does not judge one operating point.
- Recall@k leads for a retriever, because a passage that never arrived caps everything ranking metrics can do afterwards.
- RMSE when a single big miss should dominate, MAE when every error should count equally, MAPE only away from zero.
- BLEU is translation, ROUGE is summarisation, BERTScore tolerates paraphrase, and perplexity is the one of the four that needs no reference at all.
- Faithfulness reads whether a claim is supported by the source, which is how it catches the fluent fabrication that overlap metrics score well.
- Always report precision and recall beside F1, and always report the threshold a classification metric was measured at.