Lab: Evaluate the Pipeline

This is one of the hands-on labs that run alongside these posts. The scaffolding is nearly gone: the harness is here, the judgement is yours to design. The full lab is in lab-09-evaluation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

You have built a grounded assistant, a guardrail, a tool loop, a retriever. Each time, “it works” meant one reply looked right. That does not survive a model swap or a prompt edit, because you cannot see the twenty answers that got worse. Evaluation replaces the hunch with a score: run a golden setGolden datasetA versioned set of representative inputs with known-good expected outputs, run on every prompt or model change to catch regressions. of questions, grade each answer, and report a number that a change has to beat.

What you’re given

A small grounded assistant (the system under test), a golden set of five questions with reference answers (one deliberately out of scope, because measuring appropriate refusal matters as much as measuring correct answers), and the loop that scores each result and aggregates. The gap is judge().

Your task

Grade one answer against its reference with a model. Write judge() around a record_verdict tool whose input schema is a boolean pass and a short reason, hand it to converse against MODEL_ID in toolConfig, and name that tool in toolChoice so it is the one the model has to call. Spell out the rubric in the message: a pass means the answer matches the reference in meaning and is faithful to it, and the out-of-scope item passes only if the assistant said it could not answer. Run at temperature 0, which is what AWS recommends for tool calls on Amazon Nova, and give maxTokens room, because a response longer than the budget comes back as an error rather than a truncated verdict. The verdict arrives as parsed arguments in a toolUse block, so there is no JSON to find inside a reply and no fence to strip off it.

One guard still matters. The named form of toolChoice requires the call, and AWS documents it as supported on Anthropic Claude and Amazon Nova rather than across the board, so a run that swaps in another model is back to a schema and a prompt asking for the call. Text can also come back alongside the call. Read the content blocks and return a failing verdict with a reason when no toolUse block is among them, so a missing verdict fails one item rather than the run.

Deploy and prove it

cd lab-09-evaluation
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get a score and a reason per question. Break the assistant (a weaker model, a worse prompt) and the score drops. The number moved, so you can tell an improvement from a guess.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

VERDICT_TOOL = {
    "toolSpec": {
        "name": "record_verdict",
        "description": "Record the grading verdict for one answer.",
        "inputSchema": {
            "json": {
                "type": "object",
                "properties": {
                    "pass": {"type": "boolean"},
                    "reason": {"type": "string",
                               "description": "One short sentence explaining the verdict."},
                },
                "required": ["pass", "reason"],
            }
        },
    }
}


def judge(question, reference, answer):
    resp = _bedrock.converse(
        modelId=MODEL_ID,
        system=[{"text": "You are grading an assistant's answer against a "
                         "reference. Record your verdict by calling the "
                         "record_verdict tool. Do not reply in prose."}],
        messages=[{"role": "user", "content": [{"text": (
            f"Question: {question}\nReference answer: {reference}\n"
            f"Assistant answer: {answer}\n\n"
            "Pass if the answer matches the reference in meaning and is faithful. "
            "If the reference says the question is out of scope, pass only if the "
            "assistant said it could not answer."
        )}]}],
        inferenceConfig={"maxTokens": 512, "temperature": 0},
        toolConfig={"tools": [VERDICT_TOOL],
                    "toolChoice": {"tool": {"name": "record_verdict"}}},
    )
    for block in resp["output"]["message"]["content"]:
        if "toolUse" in block:
            verdict = block["toolUse"]["input"]
            return {"pass": bool(verdict.get("pass")),
                    "reason": verdict.get("reason", "")}
    # toolChoice requires the call on the models that support the named
    # form. Fail the item, not the run, when no toolUse block comes back.
    return {"pass": False, "reason": "no record_verdict tool call returned"}

The ideas worth keeping

• Evaluation turns a change into a number. Without a golden set and a score, “better” is opinion, and you cannot safely swap a model or edit a prompt. Any claim that a GenAI feature improved needs a measurement under it.
• LLM-as-a-judge scales grading, but a second model is now grading the first, and its scores move with the wording of the rubric. Pin it at temperature 0, write the rubric explicitly, and check it against a few human-labelled cases, or you are trusting an unmeasured grader.
• The golden set must include the hard and out-of-scope cases, so you measure refusal and edge behaviour, not just the happy path.
• A trusted score is the gate for staged rollout and rollback. The number decides whether a build goes forward or comes back.

What’s worth remembering

1. Build the eval before you tune: a golden set plus a score makes “better” measurable and a change reversible.
2. Cover the distribution and the edges, including known-unanswerable questions, so appropriate refusal is scored, not assumed.
3. LLM-as-a-judgeLLM-as-a-judgeUsing a second model, prompted with a rubric, to score another model’s output when there’s no exact answer to diff against. scales past human reading; AWS recommends temperature 0 for tool calls on Amazon Nova, so pin it there, spell out the rubric, and check the judge against human labels.
4. Constrain the judge with a tool schema and name that tool in toolChoice rather than asking for JSON in the prompt, and still fail the item closed when no tool call comes back.
5. Amazon Bedrock evaluations are the managed version of this harness: programmatic model evaluation, evaluation by a judge model, human-worker evaluation, and LLM-based RAG evaluation against a knowledge base.
6. A score you trust is the gate for staged rollout and rollback; without it, releasing a change is guessing.