Evaluating LLM Output With Bedrock Eval Jobs

The situation

The ticket-summarisation service has been running on Claude Sonnet 4.5 for six months. Its daily output, a two-sentence summary appended to each resolved ticket, feeds the customer-success team’s retrospective dashboard and a weekly executive email. Quality has been subjectively good; nobody’s complained loudly.

Claude Haiku 4.5 is available through Bedrock at a lower published per-token price, and the product manager asks the question product managers ask: can we switch? Engineering’s answer needs three things. Does the cheaper model produce summaries of equal quality on real tickets? Where does it regress, if anywhere? And if it’s close enough on average but worse on specific categories, can we know which?

The team has 2,000 historical tickets with ground-truth summaries written by the customer-success team (who summarise tickets by hand during quarterly reviews). The tickets cover billing, technical, account, and feature-request categories in roughly equal proportions. The summaries average 40 words and follow a loose style guide: lead with the customer’s problem, state the resolution, note anything unresolved.

What actually matters

Evaluating a language modelLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. output is genuinely harder than evaluating a classifier. A ticket summary isn’t pass-or-fail; it’s on a spectrum of better and worse, and “better” has several dimensions, accuracy (does it say true things about the ticket?), completeness (does it miss key facts?), faithfulness (does it add details the ticket doesn’t contain?), style (does it match the style guide?), length. A single metric is almost certainly wrong; a slate of metrics is almost certainly needed.

The first decision is what to measure. Reference-based metrics (BLEU, ROUGE, BERTScore) compare the model’s output to a human-written reference and produce a number. Reference-free metrics (perplexityPerplexityA measure of how well a language model predicts a sample of text – lower is better., hallucinationHallucinationAn LLM stating something false with the same confidence it states something true. scores, toxicity filters) judge the output alone. Task-specific metrics, exact-match on classification, JSON-schema validity on structured output, apply where they apply. And LLM-as-judge: a second language model scores the first model’s output against a rubric.

The second is who does the scoring. Automated metrics are fast and comparable across runs, but they capture a narrow slice of quality. Humans are slow and not perfectly consistent with each other, but they capture everything else. A mixed strategy, automated at volume, human on a sample, is the realistic shape.

The third is the dataset. Does it cover the full input distribution? Are the categories balanced or weighted by production traffic? Are there known edge cases (long tickets, multilingual, ambiguous resolutions) well represented? A 2,000-example evalBenchmarkA standardised test set used to score and compare models. set that’s 90% billing and 10% everything else will miss regressions in “everything else.” The set also has to fit the job: a custom prompt dataset holds up to 1,000 prompts, so 2,000 tickets is two jobs per model however we slice it.

The fourth is what question we’re answering. “Is model B as good as model A overall?” is a different question from “Is model B better on billing tickets?”, and both are different from “Where specifically does model B regress?” The first needs an aggregate number; the second needs per-category breakdowns; the third needs per-example inspection.

The fifth is cost and time. Running 2,000 examples through two models and scoring them programmatically is a few hours, and Bedrock charges nothing for the algorithmic scores on top of the inference. Human review adds USD$0.21 per completed task, which is small next to the reviewer hours it consumes. Time is the constraint that bites, not the invoice.

Cost belongs in the scoring too, not only in the budget for running the job. A model with a lower per-token price is only cheaper in production if it answers the same questions in the same number of tokens, so cost-performance analysis measures tokens consumed per accepted answer rather than tokens per call: a summary that needs a retry, a longer prompt, or a human correction one time in twenty can cost more than the model it replaced. Latency-to-quality ratios do the same work on the time axis, weighing what the extra seconds return, because half a point of completeness for two more seconds is fine on a nightly batch and unacceptable on a live agent-assist panel. And every score stands in for a business outcome, here summaries the customer-success team trusts enough to paste into the executive email without rewriting them. That outcome is what the thresholds get set against; a dimension that moves without moving anything downstream is one we’re over-weighting.

One more, easy to skip past: what we’ll do with the answer. An eval that shows “Model B is 2% worse on average” only matters if the team has a rule for what to do about that. Without a rule, the number is theatre.

What we’ll filter on

1. Scale, how many prompts fit in one job, and how many jobs does covering the set take?
2. Fidelity, how closely does the score match what humans would actually say about quality?
3. Breakdown capability, can we see per-category, per-length, per-edge-case performance?
4. Reproducibility, does the same run produce the same number, or is noise swamping signal?
5. Cost and latency, what does running the evaluation cost, and how long does it take?

The landscape

Bedrock evaluations, programmatic metrics. Amazon Bedrock evaluations live in the console under Inference and assessment, then Evaluations. A programmatic job takes a task type, a model, and a dataset. For text summarisation it computes three metrics: accuracy as BERTScore against the reference summary, a toxicity score, and robustness as BERTScore and deltaBERTScore under perturbed inputs. There is no BLEU, no ROUGE and no exact-match option on that task type. A programmatic job evaluates one model, so comparing two candidates means running the same dataset twice. Fast, reproducible, and the algorithmic scores carry no charge beyond the inference.

Bedrock evaluations, judge model. The same job framework, created as Automatic: Model as a judge. A generator model answers the prompts and an evaluator model scores each response, with a written explanation per score. Built-in metrics include Correctness, Completeness, Faithfulness, Helpfulness, Logical coherence, Relevance, Following instructions and Professional style and tone. Custom metrics take our own instructions and rating scale, up to ten in a job. The evaluator has to come from the supported list, which in the Claude family reaches Sonnet 4.6, Opus 4.8 and Haiku 4.5: Sonnet 5 can generate the responses but is not selectable as a judge. Judge bias is the standing caveat, since an evaluator scores responses written in its own idiom higher.

Bedrock evaluations, human review. Created as Human: Bring your own work team. A human job takes up to two inference sources, so it is the one shape that puts two models in front of the same reviewer on the same prompt. The work team is a private workforce managed through Amazon Cognito and Amazon SageMaker Ground Truth, capped at 50 workers, with up to 50 email addresses entered at a time. Metrics are ours, each with a rating method: thumbs up/down, individual Likert scale, comparison Likert scale, choice buttons, ordinal ranking. Charged at USD$0.21 per completed task. Slow, highest-fidelity, and the output bucket needs a CORS configuration the other two job types do without.

Custom evaluation pipeline. A Python script, some datasets in S3, a Lambda or batch job running each example, storing outputs in DynamoDB, computing metrics with Hugging Face evaluate or custom scorers. Maximum flexibility; maximum code; lacks the managed workflow features (versioning, reports, audit trail) that Bedrock evaluations provide.

Playground prompting. Type a prompt into the console playground and read what comes back. No reference, no score, nothing written down afterwards. Useful for prompt-engineering exploration, where an evaluation job is what answers “can we switch.”

Evaluation

Side by side

No single row handles the question alone. Programmatic metrics cover scale; a judge model covers fidelity at scale with caveats; human review covers fidelity on a sample. The real answer is all three in a stack.

The layered evaluation, illustrated

The solution

Programmatic metrics, both models. A dataset holds 1,000 prompts and a programmatic job scores one model, so the 2,000 tickets become four jobs: two halves, two models. Each line of the .jsonl carries prompt (the ticket text), referenceResponse (the human summary) and category, and that third key is what produces per-category scores in the report. Task type is text summarisation, which scores accuracy as BERTScore against the reference. If Haiku lands within a point or two of Sonnet, we’re in the noise-floor zone and the answer needs more data. If it drops hard, there’s a real gap and we can stop here.

A judge job, both models, per-dimension metrics. Four more jobs, created as Automatic: Model as a judge with Claude Sonnet 4.6 (anthropic.claude-sonnet-4-6) as the evaluator. Three of the four dimensions are built-ins:

• Correctness (Builtin.Correctness): does the summary state true things about the ticket?
• Completeness (Builtin.Completeness): does it capture the customer problem, the resolution, the unresolved items?
• Faithfulness (Builtin.Faithfulness): does it introduce anything not in the ticket?

Supplying referenceResponse changes how the first two are scored, because Bedrock passes the ground truth into the evaluator prompt for Correctness and Completeness. Style-guide adherence has no built-in that fits, so it goes in as a custom metric with our own instructions (lead with the problem, state the resolution, note anything unresolved) and a numerical rating scale of one to five, each point defined. Leave the output schema enabled, or the results come back as explanations with no scores to aggregate.

The job writes a score and a one-sentence explanation per example, rolled up by category in the report. Now the answer has shape: “Haiku scores 0.2 lower on Completeness for billing tickets, everything else within 0.1.”

Human review on a stratified 200-example sample. The judge is useful and has known biases, so calibrate it. A human job takes two inference sources, which puts both models’ summaries in front of the same reviewer on the same ticket, the comparison a judge job cannot make. Use the individual Likert scale rating method so the numbers line up with the judge’s one-to-five, and stratify the sample at 50 per category with oversampling of edge cases: very long tickets, multilingual, ambiguous resolutions. Compute the correlation between judge scores and human scores per dimension. If the correlation is strong (r > 0.7 per dimension), trust the judge’s aggregate. If it’s weak on Completeness, tighten the metric instructions or lean harder on human scores. Set the CORS configuration on the output bucket first; human jobs need it and the other two don’t.

The decision. Combine the three: programmatic aggregates for a first-pass sanity check; judge aggregates per category for the main signal; human review on the stratified sample to calibrate the judge and to inspect outliers (examples where Haiku and Sonnet diverge most). The decision rule should be set before the numbers come in: “switch if Haiku is within 0.3 on aggregate and within 0.5 on every category, in the judge’s scoring, validated by human review on the stratified sample.” With the rule pre-committed, the answer follows from the data instead of the other way around.

Where the numbers land. Each job writes its per-example scores as JSON to the Amazon S3 output location configured when the job was created, alongside the aggregate report the console renders. That file is what the next candidate model gets compared against, what an auditor reads when somebody asks how the switch decision was reached a year later, and what a stakeholder dashboard reads when the product manager wants the per-category numbers without booking time with engineering. A result that exists only as a console screenshot does none of that.

Worked example

2,000 tickets, Claude Haiku 4.5 against Claude Sonnet 4.5, all eight jobs complete.

Programmatic metrics (text summarisation, aggregate):
  BERTScore, accuracy:     Sonnet 0.892   Haiku 0.884   diff -0.008
  deltaBERTScore, robustness: Sonnet 0.021   Haiku 0.034

Judge model (aggregate, mean of four metrics):
  Overall:   Sonnet 4.21    Haiku 4.04    diff -0.17

Judge model (per category, Overall mean):
  Billing:          Sonnet 4.35   Haiku 4.18   diff -0.17
  Technical:        Sonnet 4.30   Haiku 4.16   diff -0.14
  Account:          Sonnet 4.05   Haiku 3.97   diff -0.08
  Feature request:  Sonnet 4.14   Haiku 3.85   diff -0.29  ← watch this

Human review (200 examples, stratified, individual Likert):
  Correlation with judge, Correctness:   r = 0.78
  Correlation with judge, Completeness:  r = 0.71
  Correlation with judge, Faithfulness:  r = 0.83
  Correlation with judge, Style:         r = 0.62  ← lower

Feature-request category, human scoring:
  Sonnet 4.08   Haiku 3.78   diff -0.30
  (human confirms the judge's feature-request regression)

Decision rule was “aggregate within 0.3 and every category within 0.5”: aggregate diff is -0.17 (within 0.3), every category diff is within 0.5, and the human calibration supports the judge’s finding on feature requests. Technically passes. But the team’s informal rule turned out to be “don’t regress on feature requests, they drive growth.” Feature-request category is down 0.30 in both judge and human scoring. The switch doesn’t happen; Haiku is shelved for summarisation.

What the evaluation also produced: a clear answer to “why not?” that the product manager can act on. Not “the new model isn’t as good”, which is an unhelpful answer, but “the new model is equivalent except for feature-request summaries, where it loses a specific kind of completeness.” That’s actionable: maybe a prompt tweak specific to feature requests would close the gap; maybe routing the other three categories to Haiku and feature requests to Sonnet lowers the bill without the regression.

What’s worth remembering

1. No single metric answers a real quality question. A slate, aggregate, per-dimension, per-category, is the floor.
2. Three layers, each doing what it’s good at. Programmatic metrics for scale, a judge model for rubric-scored breadth, human review for calibration and edge cases.
3. Bedrock evaluations run all three inside limits worth knowing: 1,000 prompts per dataset, one model in a programmatic or judge job, two in a human job.
4. Decision rules before numbers. Commit to the threshold, “within 0.3 on aggregate and 0.5 on every category”, before running the eval. Otherwise the threshold drifts to fit whichever model we wanted to pick.
5. The judge has biases and has to come from the supported evaluator list. Correlation between judge scores and human scores on a stratified sample tells you which dimensions to trust.

The model doesn’t switch. The team has metrics, a pipeline, a decision rule, and a calibrated judge, and next time a candidate model shows up, the same jobs run and the answer arrives in the time it takes to schedule them rather than the time it takes to argue.