Lab: Evaluate the Pipeline

This is one of the hands-on labs that run alongside these posts. The scaffolding is nearly gone: the harness is here, the judgement is yours to design. The full lab is in lab-09-evaluation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

You have built a grounded assistant, a guardrail, a tool loop, a retriever. Each time, “it works” meant one reply looked right. That does not survive a model swap or a prompt edit, because you cannot see the twenty answers that got worse. Evaluation replaces the hunch with a score: run a golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. of questions, grade each answer, and report a number that a change has to beat.

What you’re given

A small grounded assistant (the system under test), a golden set of five questions with reference answers (one deliberately out of scope, because measuring appropriate refusal matters as much as measuring correct answers), and the loop that scores each result and aggregates. The gap is judge().

Your task

Grade one answer against its reference with a model. Write judge() as a prompt that hands the model the question, the reference, and the assistant’s answer, spells out the rubric (a pass means the answer matches the reference in meaning and is faithful to it, and the out-of-scope item passes only if the assistant declined), and demands a JSON object only: a boolean pass and a short reason. Send it through converse against MODEL_ID at temperature 0 with a small token budget. Then treat what comes back as untrusted. Judges fence their JSON whether you asked or not, so strip a markdown fence before parsing; and fail closed, returning a failing verdict with a reason, both when the text will not parse and when it parses into something that is not an object.

Both guards earn their keep. Without the fence strip, a judge that wraps its JSON scores every item zero, and the number you get back measures the parser rather than the assistant. Without the shape check, a bare true parses cleanly and then breaks the first .get(), taking the whole run with it. Fail one item closed; never fail the run.

Deploy and prove it

cd lab-09-evaluation
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get a score and a reason per question. Break the assistant (a weaker model, a worse prompt) and the score drops, which is the entire point: the number moved, so you can tell a change apart from a hope.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

def judge(question, reference, answer):
    prompt = (
        "You are grading an assistant's answer against a reference.\n"
        f"Question: {question}\nReference answer: {reference}\n"
        f"Assistant answer: {answer}\n\n"
        "Pass if the answer matches the reference in meaning and is faithful. "
        "If the reference says the question is out of scope, pass only if the "
        "assistant declined. "
        'Reply with a JSON object only: {"pass": true or false, "reason": "..."}'
    )
    resp = _bedrock.converse(
        modelId=MODEL_ID,
        messages=[{"role": "user", "content": [{"text": prompt}]}],
        inferenceConfig={"maxTokens": 200, "temperature": 0},
    )
    text = resp["output"]["message"]["content"][0]["text"].strip()
    if text.startswith("```"):          # judges fence their JSON, asked to or not
        text = text.strip("`")
        if text.lower().startswith("json"):
            text = text[4:]
    try:
        verdict = json.loads(text)
    except json.JSONDecodeError:
        return {"pass": False, "reason": "unparseable judge output"}
    if not isinstance(verdict, dict):   # valid JSON, wrong shape: still a failure
        return {"pass": False, "reason": "judge output was not an object"}
    return verdict

The ideas the exam cares about

• Evaluation turns a change into a number. Without a golden set and a score, “better” is opinion, and you cannot safely swap a model or edit a prompt. Every scenario about improving or comparing a GenAI feature is asking for measurement.
• LLM-as-a-judge scales grading, but the judge is a model with its own rubric and biases. Pin it at temperature 0, write the rubric explicitly, and validate the judge itself against a few human-labelled cases, or you are trusting an unmeasured grader.
• The golden set must include the hard and out-of-scope cases, so you measure refusal and edge behaviour, not just the happy path. Amazon Bedrock evaluation jobs provide this as a managed capability with automatic or human scoring.
• A trusted score is the gate for staged rollout and rollback. The release process from the versioning post depends on exactly this number.

What’s worth remembering

1. Build the eval before you tune: a golden set plus a score makes “better” measurable and a change reversible.
2. Cover the distribution and the edges, including known-unanswerable questions, so appropriate refusal is scored, not assumed.
3. LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scales past human reading; pin it at temperature 0, spell out the rubric, and check the judge against human labels.
4. The judge is itself a model, so treat its output as untrusted: demand JSON, parse defensively, and fail closed.
5. Amazon Bedrock model and RAG evaluation jobs offer automatic and human scoring as a managed version of this harness.
6. A score you trust is the gate for staged rollout and rollback; without it, releasing a change is guessing.