Exam-style
A support team generates roughly 40,000 answers a week and wants each one scored for helpfulness and tone. Human reviewers can read a few hundred a week at most. What approach scales, and stays trustworthy?
Reveal the answer
A. Score every output with a judge model against a written rubric, calibrated against a few hundred human-scored examples
A judge model scoring against an explicit rubric is the only option here that reaches 40,000 items a week. The few hundred human-scored examples are the reference baseline, showing how closely the judge tracks human ratings. Judge models score their own output higher than human raters do, and score weak output more generously, so scoring each answer with the model that wrote it runs that bias at its worst. Amazon Bedrock separates the two roles, a generator model and an evaluator model, and its built-in judge metrics include Helpfulness and Professional style and tone. A managed evaluation job takes at most 1,000 prompts, which makes it the calibration run rather than the weekly scoring pass. ROUGE and METEOR measure word overlap against a reference, Comprehend sentiment returns positive, negative, neutral or mixed, and three reviewers an item multiplies work already out of reach.
Q. You need to score thousands of outputs on quality without a human reading each. Approach and caveat?
A. LLM-as-a-judge: a second model scores outputs against a rubric, at scale. Amazon Bedrock separates the two roles, a generator model and an evaluator model. The caveat is that a judge scores its own output higher than human raters do, so calibrate against a small human-scored sample.
Why? The judge gives you scale. The human sample shows how closely its scores track human ratings.