Exam-style
An operations team wants a foundation model to summarise incident reports. Two models are on the shortlist, and an engineer has hand-written a reference summary for 500 past incidents. The team wants one comparable number per model, computed automatically, so they can drop the weaker candidate without a week of reading output. Which metric fits?
Reveal the answer
B. Recall-Oriented Understudy for Gisting Evaluation (ROUGE), because it measures how much of the reference summary turns up in the generated one
Reference summaries in hand and a comparable number wanted quickly point to a reference-based automatic metric: for summarisation, Recall-Oriented Understudy for Gisting Evaluation (ROUGE). It counts n-gram overlap with the reference, focused on recall: shared single words (ROUGE-1), shared adjacent pairs (ROUGE-2), or the longest common subsequence, where word order counts and adjacency does not (ROUGE-L). Covering the ground the reference covered is what a summary is judged on. Bilingual Evaluation Understudy (BLEU) also scores text against a reference, so the setup looks identical. It counts the same overlaps in the other direction, focused on precision, and penalises output much shorter than the reference. That fits machine translation. A good incident summary says less than the report, in its own words, which BLEU scores down. Accuracy is a classification metric, comparing a predicted label with a true one, and two good summaries of one outage may share barely a phrase. Root mean squared error is a regression metric, measuring the distance between a predicted number and an observed one; there is nothing to subtract when both sides are sentences. LLM-as-a-judge has a second model score each summary against a rubric, a reference being optional. It needs an inference call per summary and can inherit the judge model’s bias. None of the five scores a correct paraphrase well, which is why BERTScore, which compares embeddings rather than words, is usually run beside ROUGE.
Q. Two models summarising incident reports, 500 hand-written reference summaries, one automatic number per model wanted. Which metric?
A. Recall-Oriented Understudy for Gisting Evaluation (ROUGE), the summarisation metric, which measures how much of the reference summary appears in the generated one.
Why? Match the metric to the task. ROUGE is recall-focused: a high score means the generated summary covered what the reference covered, and covering the ground is what a summary is for. Bilingual Evaluation Understudy (BLEU) counts the same overlaps focused on precision, and penalises short output. That fits machine translation. It scores a brief, rephrased summary down. Accuracy needs a single right answer to match against, so it has no meaning over free text, and root mean squared error needs two numbers to subtract, so it has none either. LLM-as-a-judge scores against a rubric, with a reference optional, and can inherit the judge model’s bias. It measures something other than the automatic reference-based number this team asked for. Nothing on the shortlist scores a correct paraphrase well, which is why BERTScore, comparing embeddings rather than words, is usually run beside ROUGE.