LLM-as-a-Judge: Designing a Rubric You Can Trust

The situation

A team has a support-summarisation feature on Amazon Bedrock: it turns a long ticket thread into a three-sentence summary for the next agent to read. They already run Bedrock evaluation jobs with programmatic metrics, and on the text-summarisation task type that means BERTScore against one reference summary, with no ROUGE, BLEU or exact-match option. The number tells them little. A summary that picks different but equally correct facts scores low, and one that tracks the reference wording while missing the resolution scores high. They want a quality signal that tracks what an agent would actually say about the summary, and they want it on thousands of examples, not a handful.

So they reach for a second model as the judge. Point one Bedrock model at the summaries and ask it to score them. The first cut is a prompt that says “rate this summary from 1 to 10”. It runs, it produces numbers, and the numbers are useless. Nearly everything lands between 7 and 9. Longer summaries score higher whether or not they are better. And when the judge and the candidate come from the same model family, the candidate looks suspiciously strong.

The judge is producing a score. The open question is whether the score means anything, and what it would take to trust it before wiring it into a release gate.

What actually matters

An LLM judge is a measuring instrument, and an uncalibrated instrument returns numbers with no known relationship to the thing being measured. A judge does not measure quality; it measures agreement with whatever standard the rubric encodes. Where the rubric is vague, the scores fall back on the model’s own priors, and those priors are the biases you are trying to avoid.

The most consequential design choice is what the judge is asked to produce. Pointwise scoring rates one answer against a rubric on its own: accuracy 4 of 5, completeness 3 of 5, and so on. It is easy to aggregate, it gives per-dimension signal, and it maps cleanly onto a threshold you can gate on. Its weakness is that “4 out of 5” has no fixed anchor across examples; what comes out as a 4 drifts from one to the next, so absolute scores are noisier than they look. Pairwise comparison asks a narrower question: given answer A and answer B, which is better? Models are markedly more reliable at ranking two things than at pinning an absolute number on one, because the comparison puts a concrete reference in the prompt rather than leaving the scale implicit. The limit is that pairwise gives you an ordering, not a level, and comparing every pair is quadratic, so at scale you compare against a fixed baseline rather than all-against-all.

The biases are specific and documented. Position bias: in a pairwise prompt the judge scores whichever answer is presented first (or sometimes last) higher, regardless of content. Verbosity bias: judges score longer, more elaborate answers higher even when the extra length adds nothing, which is the failure the team is seeing. Self-preference (or self-enhancement) bias: a judge scores outputs from its own model family higher than a neutral grader would. Each of these has a matching control. Randomise the order of A and B across the run, and ideally score both orders and average, so position cancels out. Control for length, either by holding the two candidates to similar lengths or by explicitly instructing the judge to ignore length and reward concision. Use a judge from a different model family than the one under test, which takes self-preference out of the setup. And ask for the reasoning before the score, not after, because text generated after a number tends to justify that number, while a score generated after the reasoning follows from the criteria already stated.

The last thing that matters, and the one most often skipped, is calibration. Before you trust a judge on the full set, you check it against a few hundred examples that humans have already labelled. If the judge’s scores correlate strongly with the human scores, you have grounds to run it at scale. If they do not, the judge is measuring something other than what you care about, and running it on more data just produces more wrong numbers faster. Calibration is what turns “the model said 8” into “the model said 8 and we know that means what we think it means”.

What we’ll filter on

1. What you are scoring: one answer against a standard (pointwise), or two answers against each other (pairwise)?
2. Rubric concreteness: explicit named criteria on a fixed, anchored scale, or a bare “is this good”?
3. Bias exposure: which of position, verbosity, and self-preference does this setup invite, and is each one controlled?
4. Rationale ordering: does the judge reason first and score second, or emit a bare number?
5. Judge independence: is the judge a different model family from the candidate under test?
6. Calibration: has the judge been checked against a human-labelled set before it grades at scale?

The landscape

Pointwise LLM scoring against a rubric. The judge sees one answer and the rubric, and returns a score per criterion plus a rationale. Strong for per-dimension diagnostics (“completeness is fine, faithfulnessFaithfulnessWhether every claim in an answer is actually supported by the source it was given, regardless of whether it happens to be true. is the problem”) and for gating on an absolute threshold. Weak on cross-example consistency, because nothing anchors the scale from one example to the next. Best when you need to know why an answer is weak, and when a rough absolute level is good enough.

Pairwise LLM comparison. The judge sees two answers to the same input and picks the better one, or declares a tie. More reliable than pointwise on the core “which is better” question because ranking is easier than absolute scoring, which makes it the better instrument for comparing two models or two prompt versions. It carries position bias hard, so order randomisation is mandatory, and it gives an ordering rather than a level, so you cannot read an absolute quality bar off it directly. At scale you compare each candidate against a fixed reference answer rather than all pairs.

Reference-based programmatic metrics. BLEU, ROUGE, BERTScore, exact-match: deterministic, reproducible, low cost, and they all need a gold reference. They measure similarity to that reference, not quality, so a good answer that is worded or focused differently scores badly. Bedrock’s own summarisation task type computes BERTScore and deltaBERTScore plus a toxicity score, and offers none of the others. Fine as a regression tripwire, poor as the primary signal for open-ended generation. This is the tool the team already found wanting.

Human review. The highest-fidelity signal and the standard everything else is calibrated against. Expensive, slow, and not perfectly self-consistent (inter-rater disagreement is real), so it runs on a representative sample, not the full set. Its job in a mature setup is to calibrate the judge and to adjudicate the outliers, not to grade everything.

Bedrock evaluations, model as a judge. Amazon Bedrock evaluations offer three methods for evaluating a model: programmatic metrics, human workers, and a judge model. The judge option pairs an evaluator model with a generator model. It scores against built-in metrics: correctness, completeness, faithfulness, helpfulness, logical coherence, relevance, following instructions, professional style and tone, and the responsible-AI checks harmfulness, stereotyping and refusal. You can also define up to ten custom metrics of your own, each with its own prompt and rating scale, though any one dataset is scored against three metrics at a time. Every score comes with a written explanation, aggregated in the console and written to S3. Three documented limits shape how you use it. An automated job evaluates one model. A judge job takes one response per prompt, so pairwise is not a mode the managed job offers. And a prompt dataset holds 1,000 prompts per job. It runs the pointwise pattern at volume without a harness of your own, and leaves you the rubric design, the bias controls, and the calibration against human labels.

Evaluation

Side by side

Approach	Answers	Needs a reference	Bias exposure	Scales	Trust lever
Pointwise LLM judge	How good, per criterion	✗	Verbosity, self-preference	✓	Concrete anchored rubric
Pairwise LLM judge	Which of two is better	✗	Position (high), verbosity	✓ (vs baseline)	Order randomisation
Programmatic metrics	Similarity to a gold answer	✓	None (deterministic)	✓	Reference quality
Human review	Ground-truth judgement	✗	Inter-rater spread	✗	Representative sample
Bedrock model as a judge	Pointwise quality, managed	✗	Same as pointwise	✓ (1,000 prompts/job)	Rubric plus calibration

No row is trustworthy on its own from a standing start. The pattern that works is pointwise or pairwise LLM judging for breadth, with the rubric and bias controls designed deliberately, calibrated against a human-labelled sample before it gates anything.

The solution

Write a concrete rubric. “Rate this summary from 1 to 10” gives the judge nothing to anchor on, so the scores cluster in the 7 to 9 band and lean on the model’s own priors. Replace the single vague scale with named criteria and a fixed, described scale for each. For the summariser: faithfulness (does every claim in the summary appear in the ticket? 1 means invents facts, 5 means fully grounded), completeness (does it capture the resolution and the open action? 1 means misses both, 5 means both present), and concision (is it three sentences of signal? 1 means padded, 5 means tight). Describe what each level looks like, so a 3 means the same thing on Tuesday as it did on Monday. Bedrock splits that description across two places: the metric prompt carries the scoring guidelines in full, up to 5,000 characters, while the rating scale attached to the metric holds one short definition per level, capped at five words and 100 characters. Keep the two consistent, and put the detail in the prompt. Without a rating scale the results come back as explanations with no scores and no charts.

Prefer pairwise when you need to know which of two is better. Comparing the current prompt against a candidate prompt, or model A against model B, is a ranking problem, and judges rank far more reliably than they score. Show the judge both answers to the same input and ask which better satisfies the rubric, allowing an explicit tie. The catch is position bias, strong enough to flip verdicts. Randomise which answer is A on every example. On the ones that matter, run both orders and keep the result only when they agree. A flip on the swap means the score is tracking position rather than quality. Bedrock’s judge jobs take one response per prompt, so this is a harness you build on the Bedrock runtime API rather than a mode you select. Pointwise still helps when you need a per-criterion diagnostic or an absolute gate (“ship nothing below faithfulness 4”), and the two compose: pointwise for the level, pairwise for the head-to-head.

Control the biases you cannot design out. Verbosity is the one biting this team, so instruct the judge to reward concision and penalise padding, and where you can, hold the compared answers to similar lengths so length is not a free variable. Self-preference is handled by picking a judge from a different family than the candidate under test. And always have the judge state its reasoning against each criterion before it gives the number. Text generated after a number tends to justify that number. Reasoning first makes the number follow from stated evidence, and leaves an auditable trail when a score looks wrong. Bedrock puts a structural constraint on that ordering: in a custom metric prompt the input variables must come last, after the role, the task and the rubric.

Calibrate before you scale, every time. Take one to a few hundred examples. Have humans score them on the same rubric and the same scale, run the judge on that set, and measure how well the two agree, per criterion. Strong agreement supports wiring the judge into a release gate; weak agreement on a criterion means the rubric is underspecified there, so you tighten the wording and re-check rather than shipping the judge as-is. Bedrock splits this cleanly: a judge job for breadth, and a human-worker evaluation job on a stratified sample for the calibration set. The human job takes up to two models and one custom prompt dataset, so it suits a side-by-side on a small set. Recalibrate when the model, the prompt, or the data distribution changes, because a judge calibrated against last quarter’s traffic is only assumed-good against this quarter’s.

Worked example

The first judge prompt was “rate this summary 1 to 10”, judged by the same model family that wrote the summaries. Scores clustered at 8, longer summaries won, and the candidate model looked great against itself. Three biases stacked: no anchor, verbosity, and self-preference.

The rebuild changed four things at once. The rubric became three named criteria (faithfulness, completeness, concision), each 1 to 5, with the level descriptions written out in the metric prompt and short labels on the rating scale. The judge model moved to a different family than the candidates, which removed self-preference. The prompt now asks for a sentence of reasoning per criterion before the scores, and explicitly says to reward concision and ignore length as a virtue. And for the model-versus-model question, the setup switched to pairwise: show both summaries of the same ticket, randomised order, ask which better meets the rubric, run both orders on the tie-break set and discard disagreements.

Before trusting any of it, they pulled 150 tickets, had two support leads score the summaries on the same rubric, and ran the judge on the same 150. Faithfulness and completeness correlated well with the humans; concision correlated weakly, because the rubric’s level descriptions were mushy about what “padded” meant. They rewrote those level descriptions with concrete cues (a summary that restates the ticket verbatim is a 2; three sentences of new synthesis is a 5), re-ran, and the correlation came up. Only then did the judge move to the full set, 1,000 prompts a job. The pairwise comparison stayed in their own harness for the model-swap decision. The human sample became the recurring calibration check. The number the release gate now reads is one they have a reason to believe.

What’s worth remembering

1. An LLM judge measures agreement with the rubric, not quality; a vague rubric returns the model’s own priors as a number.
2. Pointwise scoring gives per-criterion diagnostics and an absolute gate but drifts across examples; pairwise comparison is more reliable because ranking two answers is easier than scoring one.
3. Write named criteria on a fixed scale with each level described, so a 3 means the same thing every time; replace “rate this 1 to 10” with anchored dimensions.
4. Self-preference bias means a model scores its own family higher, so pick a judge from a different family than the candidate under test.
5. Calibrate the judge against a human-labelled sample and check per-criterion correlation before you trust it at scale; weak correlation means fix the rubric, not run more data.
6. A Bedrock judge job scores one model, one response per prompt, and 1,000 prompts per dataset, so comparisons and larger sets mean several jobs.