Lab 09 — Evaluate the pipeline

Scaffold: 1/5. You stop building features and start measuring one. The assistant and the harness are here; you design the judge.

The scenario

Every earlier lab left you with a working feature and a nagging question: is it any good, and would a change make it better or worse? You cannot answer that by eyeballing one reply. You need a golden set, an automatic judge, and a score, so a change becomes a number you can compare. This lab builds that loop over a tiny grounded assistant.

The requirement

Run the eval and get back a score and a per-question verdict: the four answerable questions should pass, and the out-of-scope one should pass only if the assistant declined to answer it.

What’s provided

template.yaml — a Lambda allowed to call Bedrock (for the assistant and the judge).
golden.py — the corpus, and a golden set of five questions with reference answers, one of them deliberately out of scope.
src/handler.py — the assistant under test (answer_system) and the scoring loop that aggregates a score. The gap is judge().
solution/handler.py — the reference answer.
scripts/ — deploy, test, teardown.

Your task

Implement judge(question, reference, answer), returning {"pass": bool, "reason": str}:

Prompt a model to compare the assistant’s answer to the reference and decide pass or fail (for the out-of-scope item, pass only if the assistant declined).
Ask for JSON only, at temperature 0.
Parse it; fail closed with a reason when it will not parse, and when it parses into something that is not an object.

This is LLM-as-a-judge. The rubric you write in that prompt is the whole game.

Run it

# Prerequisite: Model access enabled for your model, in your region. The default
# model id, amazon.nova-lite-v1:0, is on-demand in us-east-1; elsewhere use the
# inference profile: MODEL_ID=us.amazon.nova-lite-v1:0 ./scripts/deploy.sh
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

What success looks like

test.sh prints something like "score": 1.0, "passed": 5, "total": 5 with a reason for each verdict. Break the assistant (lower its grounding, or point it at a wrong model) and the score drops, which is the entire point: the number moved, so you can tell.

If it fails

Judge output will not parse — models like to wrap JSON in prose or a fence. Insist on JSON only, strip a fence, and fail closed when parsing fails. Check the shape too: a bare true is valid JSON and will break the scoring loop.
ValidationException about on-demand throughput — the region has no on-demand capacity for that base model id, so it needs an inference profile. Redeploy with MODEL_ID=us.amazon.nova-lite-v1:0 (or the eu. profile in Europe), which routes across the region group.
The out-of-scope item passes when it should not — your rubric must tell the judge that declining is the correct behaviour there; otherwise it grades a confident wrong answer as fine.
Scores wobble between runs — judge at temperature 0, keep the rubric tight, and remember a judge is itself a model, so validate it against a few hand-labelled cases.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

Evaluation turns a change into a number. Without a golden set and a score, “better” is a hunch, and you cannot safely swap a model or edit a prompt.
LLM-as-a-judge scales grading past what humans can read, but the judge is a model with its own rubric and biases; pin it (temperature 0), keep the rubric explicit, and spot-check it against human labels.
The golden set must include the hard and the out-of-scope cases, so you measure appropriate refusal, not just happy-path answers. Amazon Bedrock evaluation jobs offer this as a managed capability, with automatic or human scoring.
A score you trust is what makes staged rollout and rollback possible: it is the gate a change has to clear.

Lab 10 — The capstone. No gap to fill and no scaffolding: just data, a requirement, and the pieces you have built across the track. You assemble a grounded, guarded, evaluated assistant from scratch.