Exam-style
A hiring-summary feature built on Bedrock passed a fairness evaluation at launch nine months ago. Since then the prompt has been edited four times and the model version has been bumped twice, and the fairness position has not been re-measured. Which arrangement keeps it current?
Reveal the answer
E. Run a scheduled evaluation pass over a group-varied probe set and a sampled slice of the invocation logs, publish the fairness scores to CloudWatch with alarms on movement, and gate the pipeline on every prompt and model-version change
Bias drift is the movement between two measurements, so an arrangement that never takes the second one cannot detect it. The scheduled pass does. Run a probe set that varies the group signal through the fmeval library, whose prompt-stereotyping dimension scores exactly this, widened with a sampled slice of the Bedrock invocation logs so the numbers reflect real traffic, and add per-cohort non-answer rate and quality parity as custom scores. Amazon Bedrock evaluations writes its results to S3, and its built-in automatic metrics are accuracy, robustness and toxicity, so the fairness scores reach CloudWatch as custom metrics, where alarms on movement drive alerting and remediation. The same job gates every prompt edit and model-version bump. SageMaker Model Monitor’s bias-drift job compares live predictions against a bias baseline across protected attributes on a deployed classic-ML endpoint, which this feature does not have, and Model Monitor is closed to new customers. A guardrail applies content filters at runtime and produces no fairness number, whatever the strength. Human review reads one answer at a time, which catches a bad summary and cannot produce a cohort comparison. Contextual grounding checks score grounding and relevance against a supplied source, not fairness across groups.
Q. The launch fairness evaluation is nine months, four prompt edits and two model versions old. What keeps the number current?
A. A scheduled evaluation pass. Re-run the launch measurement on a schedule and as a gate on every prompt or model-version change: a probe set that varies the group signal, scored by fmeval’s prompt-stereotyping dimension, widened with sampled Bedrock invocation logs. Publish the scores as custom CloudWatch metrics and alarm on movement, so bias drift monitoring feeds alerting and remediation. The launch measurement is what gets repeated.
Why? Measurement is offline and scheduled; enforcement is runtime. Bedrock Guardrails is the reflex answer, and blocking harmful output says nothing about whether one cohort gets shorter summaries. Model Monitor’s bias-drift job carries the right words, and it compares live predictions against a baseline on a classic-ML endpoint that a Bedrock feature does not have; it is also closed to new customers. Human review reads one answer at a time, so it flags an unfair summary, never a cohort pattern. One fairness number is a snapshot: drift shows up only once there are two, taken the same way, far enough apart to move. A prompt or model comparison needs the same clean-comparison mechanics, and the alarms belong beside the rest of the feature’s operational metrics.