Exam-style
A hiring-summary feature built on Bedrock passed a fairness evaluation at launch nine months ago. Since then the prompt has been edited four times and the model version has been bumped twice, and the fairness position has not been re-measured. Which arrangement keeps it current?
Reveal the answer
E. Run a scheduled evaluation pass over a group-varied probe set and a sampled slice of the invocation logs, publish the fairness metrics to CloudWatch with alarms on movement, and gate the pipeline on every prompt and model-version change
Bias drift is the movement between two measurements, so an arrangement that never takes the second measurement cannot detect it. The scheduled pass does: a Bedrock model-evaluation job or an fmeval run over a probe set that varies the group signal, plus a sampled slice of the invocation logs so the numbers reflect real traffic, scoring prompt stereotyping, per-cohort refusal rate and quality parity. Those land as pre-defined fairness metrics in CloudWatch, where alarms on movement drive automated alerting and remediation workflows, and the same job runs as a gate on every prompt edit and model-version bump so a change cannot ship unmeasured. SageMaker Model Monitor’s bias-drift job watches feature and label distributions on a deployed classic-ML endpoint, and this feature has no such endpoint. A guardrail enforces safety at runtime and never produces a fairness number, whatever the filter strength. Human review samples individual answers, which is useful for catching a bad summary and cannot produce a cohort comparison. Contextual grounding scores veracity against retrieved passages, not fairness across groups.
Q. The launch fairness evaluation is nine months, four prompt edits and two model versions old. What keeps the number current?
A. A scheduled evaluation pass. Re-run the launch measurement (a Bedrock model-evaluation job or fmeval over a probe set that varies the group signal, widened with a sampled slice of the invocation logs) on a schedule and as a pipeline gate on every prompt or model-version change. Publish prompt stereotyping, per-cohort refusal rate and quality parity as pre-defined fairness metrics in CloudWatch, and alarm on movement so bias drift monitoring feeds automated alerting and remediation workflows rather than a quarterly surprise. The launch measurement is the thing being repeated; repeating it is what turns a one-off score into a monitored position.
Why? Measurement is offline and scheduled; enforcement is runtime. Bedrock Guardrails is the tempting answer because it is the control already wired to this feature and it does block things, and blocking harmful output tells you nothing about whether one cohort gets shorter summaries than another. Model Monitor’s bias-drift job is tempting for the opposite reason, since it carries the right words. It measures feature and label distributions on a classic-ML endpoint, and a Bedrock feature has none. Human review reads one answer at a time, so it can flag an unfair summary and never a pattern across a cohort. A single fairness number is a snapshot. Drift only becomes visible once there are two, taken the same way, far enough apart to move. The mechanics of comparing two runs cleanly are the same ones a prompt or model comparison depends on, and the alarms belong beside the rest of the feature’s operational metrics so the same on-call rotation sees them.