Exam-style
A support team generates roughly 40,000 answers a week and wants each one scored for helpfulness and tone. Human reviewers can read a few hundred a week at most. What approach scales, and stays trustworthy?
Reveal the answer
A. Score every output with a strong judge model against a written rubric, calibrated against a few hundred human-scored examples
A judge model applying an explicit rubric is what makes scoring 40,000 items a week affordable, and the human sample is what makes the scores mean anything, since calibration has to be checked per metric rather than once overall. Judges lean toward output that reads like their own, which is why having the generating model grade itself maximises that bias instead of controlling it. BLEU and ROUGE measure surface overlap with a reference rather than helpfulness, sentiment is a different property from either metric, and full human review is the constraint being worked around.
Q. You need to score thousands of outputs on quality without a human reading each. Approach and caveat?
A. LLM-as-a-judge: a strong model scores outputs against a rubric, at scale. The caveat is that judges lean toward outputs that look like themselves, so calibrate against a small human-scored sample.
Why? Scale comes from the judge; trust comes from calibrating it to humans, per metric.