Lab: Evaluate the Pipeline

This is one of the hands-on labs that run alongside these posts. The scaffolding is nearly gone: the harness is here, the judgement is yours to design. The full lab is in lab-09-evaluation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

You have built a grounded assistant, a guardrail, a tool loop, a retriever. Each time, “it works” meant one reply looked right. That does not survive a model swap or a prompt edit, because you cannot see the twenty answers that got worse. Evaluation replaces the hunch with a score: run a golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. of questions, grade each answer, and report a number that a change has to beat.

What you’re given

A small grounded assistant (the system under test), a golden set of five questions with reference answers (one deliberately out of scope, because measuring appropriate refusal matters as much as measuring correct answers), and the loop that scores each result and aggregates. The gap is judge().

Your task

Grade one answer against its reference with a model. Write judge() around a record_verdict tool whose input schema is a boolean pass and a short reason, hand it to converse against MODEL_ID in toolConfig at temperature 0 with a small token budget, and spell out the rubric in the message: a pass means the answer matches the reference in meaning and is faithful to it, and the out-of-scope item passes only if the assistant declined. The verdict arrives as parsed arguments in a toolUse block, so there is no JSON to find inside a reply and no fence to strip off it.

One guard still matters. A schema fixes the shape of a tool call; it cannot make the model place one, and a judge that answers in prose leaves you with no toolUse block at all. Return a failing verdict with a reason when that happens, so a mute judge costs one item rather than taking the run with it. Fail one item closed; never fail the run.

Deploy and prove it

cd lab-09-evaluation
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get a score and a reason per question. Break the assistant (a weaker model, a worse prompt) and the score drops: the number moved, so you can tell a change apart from a hope.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

VERDICT_TOOL = {
    "toolSpec": {
        "name": "record_verdict",
        "description": "Record the grading verdict for one answer.",
        "inputSchema": {
            "json": {
                "type": "object",
                "properties": {
                    "pass": {"type": "boolean"},
                    "reason": {"type": "string",
                               "description": "One short sentence explaining the verdict."},
                },
                "required": ["pass", "reason"],
            }
        },
    }
}


def judge(question, reference, answer):
    resp = _bedrock.converse(
        modelId=MODEL_ID,
        system=[{"text": "You are grading an assistant's answer against a "
                         "reference. Record your verdict by calling the "
                         "record_verdict tool. Do not reply in prose."}],
        messages=[{"role": "user", "content": [{"text": (
            f"Question: {question}\nReference answer: {reference}\n"
            f"Assistant answer: {answer}\n\n"
            "Pass if the answer matches the reference in meaning and is faithful. "
            "If the reference says the question is out of scope, pass only if the "
            "assistant declined."
        )}]}],
        inferenceConfig={"maxTokens": 200, "temperature": 0},
        toolConfig={"tools": [VERDICT_TOOL]},
    )
    for block in resp["output"]["message"]["content"]:
        if "toolUse" in block:
            verdict = block["toolUse"]["input"]
            return {"pass": bool(verdict.get("pass")),
                    "reason": verdict.get("reason", "")}
    # The schema fixes the shape of a tool call, not that one happens.
    return {"pass": False, "reason": "judge did not call record_verdict"}

The ideas the exam cares about

• Evaluation turns a change into a number. Without a golden set and a score, “better” is opinion, and you cannot safely swap a model or edit a prompt. Every scenario about improving or comparing a GenAI feature is asking for measurement.
• LLM-as-a-judge scales grading, but the judge is a model with its own rubric and biases. Pin it at temperature 0, write the rubric explicitly, and validate the judge itself against a few human-labelled cases, or you are trusting an unmeasured grader.
• The golden set must include the hard and out-of-scope cases, so you measure refusal and edge behaviour, not just the happy path. Amazon Bedrock evaluation jobs provide this as a managed capability with automatic or human scoring.
• A trusted score is the gate for staged rollout and rollback. The release process from the versioning post depends on exactly this number.

What’s worth remembering

1. Build the eval before you tune: a golden set plus a score makes “better” measurable and a change reversible.
2. Cover the distribution and the edges, including known-unanswerable questions, so appropriate refusal is scored, not assumed.
3. LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scales past human reading; pin it at temperature 0, spell out the rubric, and check the judge against human labels.
4. The judge is itself a model, so constrain its output with a tool schema rather than a plea for JSON, and still fail closed when it declines to call the tool.
5. Amazon Bedrock model and RAG evaluation jobs offer automatic and human scoring as a managed version of this harness.
6. A score you trust is the gate for staged rollout and rollback; without it, releasing a change is guessing.