Exam-style
A team added a CodeBuild step that runs their full 400-example golden set through an Amazon Bedrock model evaluation job on every pull request, with a hard fail below a fixed judged score of 0.8 on Bedrock's 0-to-1 scale. Builds now take fifty minutes. The gate has blocked four times in two months, and three of those four passed on a re-run with no code change. What should the team change?
Reveal the answer
E. Run a small smoke set per pull request and the full set nightly, and express the threshold as a delta against the incumbent measured over repeated runs
Judged output is non-deterministic, so a single absolute score on a single run of 400 examples carries run-to-run variation that a threshold comparison cannot separate from a real regression. Three re-runs that passed are that variation, measured. Splitting the workflow fixes both complaints at once: a fast smoke set keeps the pull request loop short, and the full continuous evaluation run moves to a nightly schedule where fifty minutes delays nobody. Scoring the change against the incumbent version over repeated runs, rather than against a fixed number, is what makes the comparison mean something. Lowering the threshold moves the line without narrowing the variation, so it hides real regressions along with the false blocks. Automatic retries formalise re-running until it passes. One judge reduces neither the sampling variance in the model under test nor the judge’s own. A warning removes the gate rather than repairing it.
Q. Every pull request scores 400 golden examples and hard-fails below 0.8 on Bedrock’s 0-to-1 judged scale. Fifty-minute builds, four blocks, three of them green on a re-run. What changes?
A. Split the work: a small smoke set on the pull request, the full golden set on a nightly continuous evaluation run. Then stop gating on an absolute number and gate on the delta against the incumbent, measured over repeated runs, with a band wide enough to sit outside the observed variation.
Why? Regression testing for model outputs is regression testing against noisy measurements, and an automated quality gate that fires on a single sample of a non-deterministic score blocks on variation as readily as on a defect. A gate people re-run until it passes has already stopped being a gate; the pipeline still has a red step in it, and nobody believes the step.