Exam-style
A team added a CodeBuild step that runs their full 400-example golden set through an Amazon Bedrock model evaluation job on every pull request, with a hard fail below a fixed judged score of 4.0. Builds now take fifty minutes. The gate has blocked four times in two months, and three of those four passed on a re-run with no code change. What should the team change?
Reveal the answer
E. Run a small smoke set per pull request and the full set nightly, and express the threshold as a delta against the incumbent measured over repeated runs
Judged output is non-deterministic, so a single absolute score on a single run of 400 examples carries run-to-run variation the build cannot tell apart from a real regression. Three re-runs that passed are that variation, measured. Splitting the workflow fixes both complaints at once: a fast smoke set keeps the pull request loop short, and the full continuous evaluation run moves to a nightly schedule where fifty minutes costs nobody anything. Scoring the change against the incumbent version over repeated runs, rather than against a fixed number, is what makes the comparison mean something. Lowering the threshold widens the band the noise lives in and hides real regressions with it. Automatic retries formalise re-running until it passes. One judge reduces neither the sampling variance in the model under test nor the judge’s own. A warning removes the gate rather than repairing it.
Q. Every pull request scores 400 golden examples and hard-fails below 4.0. Fifty-minute builds, four blocks, three of them green on a re-run. What changes?
A. Split the work: a small smoke set on the pull request, the full golden set on a nightly continuous evaluation run. Then stop gating on an absolute number and gate on the delta against the incumbent, measured over repeated runs, with a band wide enough to sit outside the observed variation.
Why? Regression testing for model outputs is regression testing against noisy measurements, and an automated quality gate that fires on a single sample of a non-deterministic score blocks on variation as readily as on a defect. A gate people re-run until it passes has already stopped being a gate; the pipeline still has a red step in it, and nobody believes the step.