Exam-style
An operations team wants a foundation model to summarise incident reports. Two candidate models are on the shortlist, and an engineer has already written a reference summary by hand for 500 past incidents. The team wants one comparable number per model, computed automatically, so they can drop the weaker candidate before anyone spends a week reading output. Which metric fits?
Reveal the answer
B. Recall-Oriented Understudy for Gisting Evaluation (ROUGE), because it measures how much of the reference summary turns up in the generated one
Reference summaries in hand and a comparable number wanted quickly points at a reference-based automatic metric, and for summarisation that is Recall-Oriented Understudy for Gisting Evaluation (ROUGE). It counts how much of the reference turns up in the generated text: shared single words, shared adjacent pairs, or the longest run of words in the same order in both. Being recall-oriented, it asks whether the model covered the ground the reference covered. That is what a summary is judged on. Bilingual Evaluation Understudy (BLEU) tempts because it also scores generated text against a reference, so the setup looks identical. It measures the other direction: of the words and phrases the model produced, what share appear in the reference. It also penalises output much shorter than the reference. That suits machine translation, where the band of correct output is narrow and inventing wording is usually an error. A good incident summary deliberately says less than the report in its own words, and BLEU treats that as error. Accuracy is a classification metric. It needs one right answer to match against. Two summaries of the same outage can both be good while sharing barely a phrase, so the share of exact matches over 500 free-text summaries lands near zero and tells the team nothing. Root mean squared error is a regression metric. It averages the distance between a predicted number and an observed one, so it needs two numeric values per record and has nothing to subtract when both sides are sentences. LLM-as-a-judge is a real evaluation approach: a second model scores each summary against a written rubric, and no reference is needed. It costs a model call per summary, it inherits the judge model’s bias, and it answers a different ask than the reference-based automatic number this team wants. None of the five credits a correct paraphrase, which is why BERTScore is usually run beside ROUGE rather than instead of it: it embeds both texts and matches each token to its nearest counterpart by meaning, so a summary saying the database ran out of connections scores well against a reference saying the connection pool was exhausted, where ROUGE sees almost no overlap. It costs more than counting overlaps.
Q. Two models summarising incident reports, 500 hand-written reference summaries, one automatic number per model wanted. Which metric?
A. Recall-Oriented Understudy for Gisting Evaluation (ROUGE), the summarisation metric, which measures how much of the reference summary appears in the generated one.
Why? Match the metric to the task. ROUGE is recall-oriented: it asks whether the model covered what the reference covered, and covering the ground is what a summary is for. Bilingual Evaluation Understudy (BLEU) measures precision of overlap and penalises short output. That suits machine translation. It punishes a summary for being brief and rephrased. Accuracy needs a single right answer to match against, so it has no meaning over free text, and root mean squared error needs two numbers to subtract, so it has none either. LLM-as-a-judge scores against a rubric with no reference needed, and carries the judge model’s bias. It is a different measurement from the automatic reference-based one this team asked for. Nothing on the shortlist rewards a correct paraphrase, which is why BERTScore, comparing embeddings rather than words, earns its place as a second metric run beside ROUGE.