Judging Whether a Foundation Model Is Good Enough

The situation

A media company runs a news and features site. Every article published gets a three-sentence summary generated automatically and shown under the headline, on the section pages, and in the morning email. About 900 articles a day go through it, so roughly 27,000 summaries a month, and the feature has been on the same foundation model in Amazon Bedrock since it launched fourteen months ago.

A cheaper model has appeared in the catalogue. On the same volume it would cut the monthly inference bill by a bit under half, and a saving at that volume is worth chasing. The platform team wants to switch. The editorial team says the summaries are fine as they are and would rather not find out the hard way, and the head of product asks the reasonable question: would quality actually drop?

Nobody can answer. The only evidence anyone has is that no one has complained, plus a screenshot of six summaries from the cheaper model that somebody generated by hand and thought looked good. There is no evaluation set, no score, and no agreed definition of what a good summary is.

What actually matters

Three questions are getting run together under the word “quality”, and they need different measurements. The first is whether the output is close to a reference answer: given a summary an editor wrote for the same article, how much of that wording and meaning does the model reproduce? The second is whether a person would call the output good, which covers tone, readability, and whether every claim in the summary is actually in the article. The third is whether it is good enough for the job, which is a threshold somebody has to set for this feature rather than a number the measurement hands you.

The first question has cheap answers and the second does not. Automatic metrics compare generated text with a reference and produce a number in seconds for fractions of a cent. You can run them over hundreds of articles, and re-run them every time the prompt changes. They pay for that with a narrow view. They see the words, or in one case the meaning of the words, and nothing about whether the summary is useful, fair, or safe to put under a headline. A human reading the summary sees all of that and costs money and days every time you ask.

Comparability is the constraint that decides how this gets set up. A score is only meaningful next to another score from the same evaluation set, the same prompt, and the same scoring method. Change the articles between runs and the numbers stop being comparable, which is how a team ends up arguing over a difference that came from the set rather than the model. So the evaluation set gets fixed and held out before any of this starts: a sample of real articles, with reference summaries written by editors, put in one place and not edited afterwards.

The last thing to be clear about is what a published benchmark score tells you, which is less than it looks. Benchmark datasets are standardised, public collections of tasks and answers that let anyone compare models on common ground, and they are genuinely useful for a first shortlist. They are not your content. A model that tops a summarisation leaderboard has been measured on somebody else’s documents, in somebody else’s register, against somebody else’s idea of a good summary. It says nothing about eight hundred words of council-meeting reporting turned into three sentences for this masthead.

What we’ll filter on

1. Reference answers: does the method need a human-written answer for every item, and who writes them?
2. Whose content: is the model scored on this site’s articles, or on somebody else’s?
3. Cost: what does one run cost, and what does refreshing it after every prompt change cost?
4. Turnaround: minutes, hours, or days from starting the job to having a number?
5. Coverage of judgement: does it catch tone, helpfulness, and invented detail, or only wording?
6. Repeatability: run it again next month on the same inputs and do you get the same number?

The landscape

There are four ways to evaluate FM performance in play here, plus one AWS service that packages three of them.

Automatic metrics against a reference

These compare generated text with a reference answer and return a number between 0 and 1. Three of them come up constantly, and each was built for a different task.

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) measures how much of the reference turns up in the generated text. ROUGE-1 counts single-word overlap and ROUGE-2 counts overlapping adjacent pairs (n-grams). ROUGE-L looks for the longest common subsequence, meaning the longest run of words appearing in the same order in both texts without having to be adjacent. Being recall-oriented, it asks whether the model covered what the reference covered, which is the question you ask of a summary. ROUGE is the summarisation metric.

Bilingual Evaluation Understudy (BLEU) measures precision of n-gram overlap: of the n-grams the model produced, what share appear in the reference, with a penalty applied when the output is much shorter than the reference. It was built for machine translation, where there is a fairly narrow band of correct output and producing words that are not in the reference is usually a mistake. BLEU is the translation metric.

BERTScore compares meaning instead of characters. It embeds both texts, then matches each token in the generated text to its nearest token in the reference by similarity of those embeddings. Take a summary saying “the council rejected the proposal” against a reference saying “the plan was turned down by councillors”. It scores badly on ROUGE and well on BERTScore, because BERTScore credits a correct paraphrase that shares almost no words. It costs more than counting n-grams, because something has to compute the embeddings, and it is still blind to whether the summary is fair or the detail is invented.

Human-in-the-loop evaluation

People read the output and rate it. A human-in-the-loop evaluation defines the rating up front, usually a short rubric with a handful of dimensions, then puts real outputs in front of a review workforce and collects scores and comments. For this feature the reviewers would be editors, and the dimensions would be something like: is every fact in the summary in the article, does it lead with the right thing, and does it read like the masthead.

This catches everything the automatic metrics miss, and it is the only method that can settle a disagreement about tone. It costs the reviewers’ time, it takes days rather than minutes, and two reviewers will not agree perfectly, so it is run on a sample rather than the whole set and repeated rarely.

Benchmark datasets

Public, standardised sets of inputs with agreed answers, published so that models can be compared on identical work. They cost nothing to consult and they cover tasks nobody at the media company would have thought to test. That makes them the sensible way to cut a catalogue down to three candidates, in the same way that modality and context length narrow a shortlist before anything is measured.

Their limit is the one already named: a benchmark score is a measurement of somebody else’s documents. Published scores also drift out of date, and a model that has seen a public benchmark during training will do better on it than it deserves to. Use benchmark datasets to decide who gets tested, never to decide who wins.

LLM-as-a-judge

A second model reads the output and scores it against a written rubric, producing a number and usually a sentence of reasoning. LLM-as-a-judge sits between the automatic metrics and the humans: it needs no reference answer, it can be told to weigh helpfulness and faithfulness the way a person would, and it costs cents and minutes rather than days.

What it brings with it is the judge model’s own behaviour. Judges tend to reward long answers, to prefer output from models like themselves, and to be sensitive to the order two candidates are shown in. None of that stops it being useful. It means the rubric has to be written carefully, the judge validated against human scores at least once, and the same judge model used for every run you intend to compare. Building the rubric and the scoring harness is a job in itself once this goes beyond a one-off comparison.

Amazon Bedrock Model Evaluation

The managed version of most of the above, inside Bedrock. Amazon Bedrock Model Evaluation runs three kinds of job. An automatic job takes a prompt dataset in S3 and scores a model’s output with built-in metrics or with a judge model from the catalogue, and hands back results and a report in S3. A human evaluation job routes the same outputs to reviewers, either a work team you bring yourself or an AWS-managed team, using rating types and instructions you define. Either kind can also be run as a comparison of two models on the same prompts. The output then lands side by side, rather than as two scores from two runs you have to trust were set up identically.

What it buys is the plumbing: dataset handling, running the inference, collecting the scores, and storing a durable record of what was measured and when. What it does not buy is judgement about which metric belongs to which task, or where the threshold sits.

Evaluation

Side by side

Approach	Needs a reference	Scores your content	Cost	Turnaround	Catches tone and invented detail	Repeatable
ROUGE	✓	✓	Very low	Minutes	✗	✓
BLEU	✓	✓	Very low	Minutes	✗	✓
BERTScore	✓	✓	Low	Minutes	✗	✓
Benchmark datasets	Supplied	✗	None	Already published	✗	✓
LLM-as-a-judge	✗	✓	Medium	Minutes to hours	✓	Mostly
Human-in-the-loop evaluation	✗	✓	High	Days	✓	✗

Reading the table

The reference column and the tone column pull against each other, and that is the shape of the whole decision. Everything cheap and repeatable needs an editor to have written an answer first, and sees only how close the model got to that answer. Everything that can tell you a summary is smug, misleading, or subtly wrong needs either a person or a model standing in for one, and gives up some repeatability to get there.

Nothing in the table is a winner on its own. The cheap, repeatable methods compare six models over three hundred articles without spending a fortnight. The expensive ones check that the number you just optimised means what you think it means. Run the first over everything, the second over the survivors.

The solution

Fix the evaluation set first. Take 300 published articles, sampled across the sections in the proportion they actually publish, and have editors write the reference summary for each one in the house format. Put the article text and the reference summary in a prompt dataset in S3 and leave it alone. That set, and only that set, is what every number from here on is measured against.

Then run an automatic Amazon Bedrock Model Evaluation job over the three candidate models, scoring with ROUGE against those references. That gives a comparable number per model in an afternoon for a few dollars of inference, and it is enough to drop any candidate that is clearly behind. Take the top two into a human evaluation job with a work team of editors, a hundred articles each, rated on faithfulness, on whether the summary leads with the right thing, and on readability. Compare the two sets of scores, decide against the incumbent’s numbers rather than against an abstract bar, and write down the threshold you used so the next comparison can use the same one.

Four things go wrong reliably here. A high ROUGE score sits happily on top of a bad summary, because reproducing the reference’s vocabulary is not the same as being accurate: a summary that copies the article’s opening paragraph will score respectably and be useless, and a summary that invents a plausible detail loses very little on ROUGE at all. Reaching for BLEU on a summarisation task is the classic wrong metric, since a good summary deliberately says less than the source in different words and BLEU’s precision measure treats that as error. Changing the evaluation set between runs breaks comparability, so a set that gets topped up with fresh articles each quarter needs its old numbers re-run, not carried forward. And a judge model has to be pinned along with everything else, because scores from two different judges, or the same judge with a reworded rubric, are two different measurements.

Once the switch is made, the evaluation set has a second life. Re-running the automatic job whenever the prompt changes or the provider updates the model catches quiet regressions, and the stored results are the record you show when someone asks how the model was chosen. Whether the cheaper model was worth switching to in business terms is a separate measurement against a baseline, and it runs on a clock of weeks rather than minutes.

Worked example

The three candidates are the incumbent, a cheaper model from the same provider, and a small fast model that would cut the bill by four fifths.

The automatic job over 300 articles returns ROUGE-L of 0.41 for the incumbent, 0.39 for the cheaper model, and 0.28 for the small one. The small model is out: reading twenty of its summaries confirms what the score suggested, which is that it drops the second half of longer articles. The remaining gap of 0.02 is not something anyone should decide on. It is inside the range you would get by swapping which 300 articles were sampled, and it says nothing about which summaries an editor would run.

The human evaluation job on the surviving two, a hundred articles each, is where the answer comes from. Readability comes out slightly ahead for the cheaper model, which writes shorter sentences. Faithfulness does not: in four of the hundred, the cheaper model attributes a quote to the wrong person or states a number the article does not contain, against one for the incumbent. On a news site, four in a hundred is a correction a day.

That settles it without anyone appealing to a benchmark score. The cheaper model goes into features and reviews, where the summaries are descriptive and the failure mode is a dull sentence. News stays on the incumbent until a rewritten prompt brings faithfulness back to the incumbent’s level on the same fixed set. The saving is smaller than the platform team wanted, and it is defensible.

What’s worth remembering

1. ROUGE measures overlap with a reference, including the longest common subsequence, and is the summarisation metric; BLEU measures n-gram precision and is the machine-translation metric; BERTScore compares embeddings, so it credits a correct paraphrase that shares no words with the reference.
2. Automatic metrics need a human-written reference, run in minutes for almost nothing, and see wording rather than usefulness, so a high score and a bad summary can sit together.
3. Human-in-the-loop evaluation is the only method that reliably catches tone, helpfulness, and invented detail, which is why it runs on a sample of the survivors rather than the whole shortlist.
4. Benchmark datasets compare models on standardised public tasks and are for building a shortlist; a published score measures somebody else’s documents, never yours.
5. LLM-as-a-judge scores against a written rubric at a fraction of a human’s cost and needs no reference answer, and it inherits the judge model’s bias, so pin one judge and one rubric across every run you compare.
6. Amazon Bedrock Model Evaluation runs automatic scoring, human review with a workforce you bring, and side-by-side comparison of two models, all against a prompt dataset that has to stay fixed for the numbers to mean anything.