Lab 09 — Evaluate the pipeline

Scaffold: 1/5. You stop building features and start measuring one. The assistant and the harness are here; you design the judge.

The scenario

Every earlier lab left you with a working feature and a nagging question: is it any good, and would a change make it better or worse? You cannot answer that by eyeballing one reply. You need a golden set, an automatic judge, and a score, so a change becomes a number you can compare. This lab builds that loop over a tiny grounded assistant.

The requirement

Run the eval and get back a score and a per-question verdict: the four answerable questions should pass, and the out-of-scope one should pass only if the assistant declined to answer it.

What’s provided

• template.yaml — a Lambda allowed to call Bedrock (for the assistant and the judge).
• golden.py — the corpus, and a golden set of five questions with reference answers, one of them deliberately out of scope.
• src/handler.py — the assistant under test (answer_system) and the scoring loop that aggregates a score. The gap is judge().
• solution/handler.py — the reference answer.
• scripts/ — deploy, test, teardown.

Your task

Implement judge(question, reference, answer), returning {"pass": bool, "reason": str}:

1. Prompt a model to compare the assistant’s answer to the reference and decide pass or fail (for the out-of-scope item, pass only if the assistant declined).
2. Ask for JSON only, at temperature 0.
3. Parse it; fail closed with a reason when it will not parse, and when it parses into something that is not an object.

This is LLM-as-a-judge. The rubric you write in that prompt is the whole game.

Run it

# Prerequisite: Model access enabled for your model, in your region. The default
# model id, amazon.nova-lite-v1:0, is on-demand in us-east-1; elsewhere use the
# inference profile: MODEL_ID=us.amazon.nova-lite-v1:0 ./scripts/deploy.sh
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

What success looks like

test.sh prints something like "score": 1.0, "passed": 5, "total": 5 with a reason for each verdict. Break the assistant (lower its grounding, or point it at a wrong model) and the score drops. The number moved, so you can tell.

If it fails

• No verdict came back — no record_verdict call was in the response. Name the tool in toolChoice so the call is required on the models that support the named form, and fail the item closed anyway. Check the shape too: a bare true is valid JSON and will break the scoring loop.
• ValidationException about on-demand throughput — the region has no on-demand capacity for that base model id, so it needs an inference profile. Redeploy with MODEL_ID=us.amazon.nova-lite-v1:0 (or the eu. profile in Europe), which routes across the region group.
• The out-of-scope item passes when it should not — your rubric must tell the judge that declining is the correct behaviour there; otherwise it grades a confident wrong answer as fine.
• Scores wobble between runs — judge at temperature 0, keep the rubric tight, and remember a judge is itself a model, so validate it against a few hand-labelled cases.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• Evaluation turns a change into a number. Without a golden set and a score, “better” is a hunch, and you cannot safely swap a model or edit a prompt.
• LLM-as-a-judge scales grading past what humans can read, but the judge is a model with its own rubric and biases; pin it (temperature 0), keep the rubric explicit, and spot-check it against human labels.
• The golden set must include the hard and the out-of-scope cases, so you measure appropriate refusal, not just happy-path answers. Amazon Bedrock evaluation jobs offer this as a managed capability, with automatic or human scoring.
• A score you trust is what makes staged rollout and rollback possible: it is the gate a change has to clear.

Next

Lab 10 — The capstone. No gap to fill and no scaffolding: just data, a requirement, and the pieces you have built across the track. You assemble a grounded, guarded, evaluated assistant from scratch.