The situation
A media company runs a news and features site. Every article published gets a three-sentence summary generated automatically and shown under the headline, on the section pages, and in the morning email. About 900 articles a day go through it, so roughly 27,000 summaries a month, and the feature has been on the same foundation model in Amazon Bedrock since it launched fourteen months ago.
A cheaper model has appeared in the catalogue. On the same volume it would cut the monthly inference bill by a bit under half, and a saving at that volume is worth chasing. The platform team has asked to switch. The editorial team says the summaries are fine as they are and would rather not find out the hard way, and the head of product asks the reasonable question: would quality actually drop?
Nobody can answer. The only evidence anyone has is that no one has complained, plus a screenshot of six summaries from the cheaper model that somebody generated by hand and thought looked good. There is no evaluation set, no score, and no agreed definition of what a good summary is.
What actually matters
Three questions are getting run together under the word “quality”, and they need different measurements. The first is whether the output is close to a reference answer: given a summary an editor wrote for the same article, how much of that wording and meaning does the model reproduce? The second is whether a person would call the output good, which covers tone, readability, and whether every claim in the summary is actually in the article. The third is whether it is good enough for the job, which is a threshold somebody has to set for this feature rather than a number any metric produces.
The first question can be answered in seconds and the second cannot. Automatic metrics compare generated text with a reference and produce a number in seconds for fractions of a cent. You can run them over hundreds of articles, and re-run them every time the prompt changes. What they measure is narrow. They compare the words, or in one case the embedded meaning of the words, and report nothing about whether the summary is useful, fair, or safe to put under a headline. A person reading the summary sees all of that, and reviewer time runs in days rather than minutes.
Comparability shapes how this gets set up. A score is only meaningful next to another score from the same evaluation set, the same prompt, and the same scoring method. Change the articles between runs and the numbers stop being comparable, which is how a team ends up arguing over a difference that came from the set rather than the model. So the evaluation set gets fixed and held out before any of this starts: a sample of real articles, with reference summaries written by editors, put in one place and not edited afterwards.
A published benchmark score tells you less than it looks. Benchmark datasets are standardised, public collections of tasks and answers that let anyone compare models on common ground, and they are genuinely useful for a first shortlist. They are not your content. A model that tops a summarisation leaderboard has been measured on somebody else’s documents, in somebody else’s register, against somebody else’s idea of a good summary. It says nothing about eight hundred words of council-meeting reporting turned into three sentences for this masthead.
What we’ll filter on
- Reference answers: does the method need a human-written answer for every item, and who writes them?
- Whose content: is the model scored on this site’s articles, or on somebody else’s?
- Cost: what does one run cost, and what does refreshing it after every prompt change cost?
- Turnaround: minutes, hours, or days from starting the job to having a number?
- Coverage of judgement: does it catch tone, helpfulness, and invented detail, or only wording?
- Repeatability: run it again next month on the same inputs and do you get the same number?
The landscape
Four ways to evaluate FM performance are in play here, and one AWS service packages three of them.
Automatic metrics against a reference
These compare generated text with a reference answer and return a score, conventionally on a nought-to-one scale. Three of them come up constantly, and each was built for a different task.
Recall-Oriented Understudy for Gisting Evaluation (ROUGE) measures how much of the reference turns up in the generated text. ROUGE-1 counts single-word overlap and ROUGE-2 counts overlapping adjacent pairs (n-grams). ROUGE-L looks for the longest common subsequence, meaning the longest run of words appearing in the same order in both texts without having to be adjacent. Being recall-oriented, it measures whether the model covered what the reference covered, which is what you want to know about a summary. ROUGE is the summarisation metric. On AWS it is computed by SageMaker AI’s Foundation Model Evaluations (FMEval), which scores summarisation accuracy with ROUGE-N, METEOR and BERTScore. Amazon Bedrock’s own built-in metrics do not include it.
Bilingual Evaluation Understudy (BLEU) measures precision of n-gram overlap: of the n-grams the model produced, what share appear in the reference, with a penalty applied when the output is much shorter than the reference. It was built for machine translation, where there is a fairly narrow band of correct output and producing words that are not in the reference is usually a mistake. BLEU is the translation metric. Neither Bedrock nor FMEval computes it for you, so scoring with BLEU means running your own harness.
BERTScore compares embedded meaning instead of exact words. It embeds both texts with a pre-trained BERT model, then matches words in the generated text against words in the reference by cosine similarity of those embeddings. Take a summary saying “the council rejected the proposal” against a reference saying “the plan was turned down by councillors”. It scores badly on ROUGE and well on BERTScore, which compares meaning rather than wording. It costs more than counting n-grams, because something has to compute the embeddings, and it still measures nothing about whether the summary is fair or the detail appears in the article. It is the accuracy metric Amazon Bedrock computes for the text summarisation task type.
Human-in-the-loop evaluation
People read the output and rate it. A human-in-the-loop evaluation defines the rating up front, usually a short rubric with a handful of dimensions, then puts real outputs in front of a review workforce and collects scores and comments. For this feature the reviewers would be editors, and the dimensions would be something like: is every fact in the summary in the article, does it lead with the right thing, and does it read like the masthead.
This catches everything the automatic metrics miss, and it is the only method that can settle a disagreement about tone. It takes reviewer time, it runs in days rather than minutes, and two reviewers will not agree perfectly, so it goes over a sample rather than the whole set and is repeated rarely.
Benchmark datasets
Public, standardised sets of inputs with agreed answers, published so that models can be compared on identical work. They are free to consult and they cover tasks nobody at the media company would have thought to test. That makes them the sensible way to cut a catalogue down to three candidates, in the same way that modality and context length narrow a shortlist before anything is measured.
Their limit is the one already named: a benchmark score is a measurement of somebody else’s documents. Published scores also drift out of date, and a model whose training data included a public benchmark scores higher on it than its real ability warrants. Use benchmark datasets to decide who gets tested, never to decide who wins.
LLM-as-a-judge
A second model reads the output and scores it against a written rubric, producing a number and usually a sentence of reasoning. LLM-as-a-judge sits between the automatic metrics and the humans: it needs no reference answer, the rubric can weigh things like helpfulness and faithfulness, and a run costs cents and finishes in minutes rather than days.
What comes with it is the judge model’s own scoring behaviour. Judge scores shift with the length of the answer, with the order two candidates appear in, and with which model is doing the judging. None of that stops it being useful. It means the rubric has to be written carefully, the judge validated against human scores at least once, and the same judge model used for every run you intend to compare. Building the rubric and the scoring harness is a job in itself once this goes beyond a one-off comparison.
Amazon Bedrock Model Evaluation
The managed version of most of the above, under Evaluations in the Bedrock console. It runs three kinds of model evaluation job. An automatic job takes a JSONL prompt dataset in S3, up to 1,000 prompts. Each line carries a prompt, plus a referenceResponse holding the ground truth, which the accuracy and robustness metrics both require. The optional key is category, and it groups the reported scores. It scores one model on the built-in metrics for the task type you pick: accuracy, robustness and toxicity. For text summarisation the accuracy metric is BERTScore. A judge job has a second model score the generator’s responses against built-in metrics such as correctness, faithfulness and helpfulness, or against a metric you write, and returns an explanation with each score. A human job routes the responses to a work team you create and manage. The rating methods are thumbs up/down, a five-point Likert scale in individual and comparison forms, choice buttons and ordinal ranking. Only the human job takes two models at once; automatic and judge jobs score a single model, so comparing candidates means one job each against the same dataset.
The service handles the plumbing: dataset handling, running the inference, collecting the scores, and storing a durable record of what was measured and when. The algorithmic scores carry no charge beyond the inference, and human tasks are USD$0.21 each on top of it. What it will not tell you is which metric belongs to which task, or where the threshold sits.
Evaluation
Side by side
| Approach | Needs a reference | Scores your content | Cost | Turnaround | Catches tone and invented detail | Repeatable |
|---|---|---|---|---|---|---|
| ROUGE | ✓ | ✓ | Very low | Minutes | ✗ | ✓ |
| BLEU | ✓ | ✓ | Very low | Minutes | ✗ | ✓ |
| BERTScore | ✓ | ✓ | Low | Minutes | ✗ | ✓ |
| Benchmark datasets | Supplied | ✗ | None | Already published | ✗ | ✓ |
| LLM-as-a-judge | ✗ | ✓ | Medium | Minutes to hours | ✓ | Mostly |
| Human-in-the-loop evaluation | ✗ | ✓ | High | Days | ✓ | ✗ |
Reading the table
The reference column and the tone column pull against each other. Everything cheap and repeatable needs an editor to have written an answer first, and sees only how close the model got to that answer. Everything that can tell you a summary is smug, misleading, or subtly wrong needs either a person or a model standing in for one, and gives up some repeatability to get there.
Nothing in the table is a winner on its own. The cheap, repeatable methods compare several models over three hundred articles in an afternoon. The expensive ones check that the number you just optimised means what you think it means. Run the first over everything, the second over the survivors.
The solution
Fix the evaluation set first. Take 300 published articles, sampled across the sections in the proportion they actually publish, and have editors write the reference summary for each one in the house format. Write each one out as a JSONL line, article text as prompt and editor’s summary as referenceResponse, put the file in S3, and leave it alone. That set, and only that set, is what every number from here on is measured against.
Then run one automatic Bedrock evaluation job per candidate, three in all, text summarisation task type, accuracy metric, same dataset each time. That returns a BERTScore per model in an afternoon for the cost of the inference, and it is enough to drop any candidate that is clearly behind. Take the top two into a human evaluation job with a work team of editors, a hundred articles each, rated on faithfulness, on whether the summary leads with the right thing, and on readability. Compare the two sets of scores against the incumbent’s numbers rather than against an abstract bar, and write down the threshold you used so the next comparison can use the same one.
Four things go wrong reliably here. A high accuracy score can sit on top of a bad summary, because matching the reference’s vocabulary is not the same as being accurate: a summary that copies the article’s opening paragraph will score respectably and be useless, and one that adds a plausible detail the article does not contain loses very little. Scoring a summariser with BLEU is the classic wrong metric, since a good summary says less than the source in different words, and BLEU’s precision measure treats that as error. Changing the evaluation set between runs breaks comparability, so a set that gets topped up with fresh articles each quarter needs its old numbers re-run, not carried forward. And a judge model has to be pinned along with everything else, because scores from two different judges, or the same judge with a reworded rubric, are two different measurements.
Once the switch is made, the evaluation set has a second life. Re-running the automatic job whenever the prompt changes or the provider ships a new model version catches regressions nobody reported, and the stored results are the record you show when someone asks how the model was chosen. Whether the cheaper model was worth switching to in business terms is a separate measurement against a baseline, and it runs on a clock of weeks rather than minutes.
Worked example
The three candidates are the incumbent, a cheaper model from the same provider, and a small fast model that would cut the bill by four fifths.
The three automatic jobs over the same 300 articles return BERTScores of 0.88 for the incumbent, 0.87 for the cheaper model, and 0.79 for the small one. The small model is out: reading twenty of its summaries confirms what the score suggested, which is that it drops the second half of longer articles. The remaining gap of 0.01 is not something anyone should decide on. It is inside the range you would get by swapping which 300 articles were sampled, and it says nothing about which summaries an editor would run.
The human evaluation job on the surviving two, a hundred articles each, is where the answer comes from. Readability comes out slightly ahead for the cheaper model, which writes shorter sentences. Faithfulness does not: in four of the hundred, the cheaper model’s summary attributes a quote to the wrong person or states a number the article does not contain, against one for the incumbent. On a news site, four in a hundred is a correction a day.
That settles it without anyone appealing to a benchmark score. The cheaper model goes into features and reviews, where the summaries are descriptive and the failure mode is a dull sentence. News stays on the incumbent until a rewritten prompt brings faithfulness back to the incumbent’s level on the same fixed set. The saving is smaller than the platform team wanted, and it is defensible.
What’s worth remembering
- ROUGE measures overlap with a reference, including the longest common subsequence, and is the summarisation metric; BLEU measures n-gram precision and is the machine-translation metric; BERTScore compares embeddings, so a correct paraphrase scores well even when it shares no words with the reference.
- Automatic metrics need a human-written reference, run in minutes for almost nothing, and measure wording rather than usefulness, so a high score and a bad summary can sit together.
- Human-in-the-loop evaluation is the only method that reliably catches tone, helpfulness, and invented detail, which is why it runs on a sample of the survivors rather than the whole shortlist.
- Benchmark datasets compare models on standardised public tasks and are for building a shortlist; a published score measures somebody else’s documents, never yours.
- LLM-as-a-judge scores against a written rubric at a fraction of a human’s cost and needs no reference answer, and it inherits the judge model’s bias, so pin one judge and one rubric across every run you compare.
- Amazon Bedrock model evaluation runs automatic scoring, judge-model scoring, and human review by a work team you manage; only the human job compares two models at once, and every job runs against a prompt dataset that has to stay fixed for the numbers to mean anything.