Exam-style
A hiring-summary feature built on Bedrock passed a fairness evaluation at launch nine months ago. Since then the prompt has been edited four times and the model version has been bumped twice, and the fairness position has not been re-measured. Which arrangement keeps it current?
Reveal the answer
E. Run a scheduled evaluation pass over a group-varied probe set and a sampled slice of the invocation logs, publish the fairness scores to CloudWatch with alarms on movement, and gate the pipeline on every prompt and model-version change
Bias drift is the movement between two measurements, so an arrangement that never takes the second cannot detect it. The scheduled pass does. Vary the group signal across a probe set, tagging each prompt with its cohort in category so a judge-based Bedrock evaluation scores per category, and widen it with a sampled slice of the invocation logs, supplied as your own inference response data so production answers get scored. Take the built-in Stereotyping metric and add per-cohort non-answer rate and quality parity as custom metrics in one job. The report lands in the job’s S3 bucket, so the fairness scores reach CloudWatch as custom metrics you publish, where alarms on movement drive remediation. The same job gates every prompt edit and model-version bump. SageMaker Model Monitor’s bias-drift job compares live predictions against a bias baseline across protected attributes on a deployed classic-ML endpoint, which this feature does not have, and both it and the Clarify behind it are closed to new customers. A guardrail applies content filters at runtime and produces no fairness number, whatever the strength. Human review reads one answer at a time, never a cohort comparison. Contextual grounding checks score grounding and relevance against a supplied source, not fairness across groups.
Q. The launch fairness evaluation is nine months, four prompt edits and two model versions old. What keeps the number current?
A. A scheduled evaluation pass. Re-run the launch measurement on a schedule and as a gate on every prompt or model-version change: a cohort-tagged probe set that varies the group signal, scored per category by a judge-based Bedrock evaluation, widened with sampled invocation logs. Publish the scores as custom CloudWatch metrics and alarm on movement, so bias drift monitoring feeds alerting and remediation. The launch measurement is what gets repeated.
Why? Measurement is offline and scheduled; enforcement is runtime. Bedrock Guardrails is the reflex answer, and blocking harmful output says nothing about whether one cohort gets shorter summaries. Model Monitor’s bias-drift job carries the right words, and it compares live predictions against a baseline on a classic-ML endpoint that a Bedrock feature does not have; it is also closed to new customers, along with the Clarify that computes its bias metrics. Human review reads one answer at a time, never a cohort pattern. One fairness number is a snapshot: drift needs two, taken the same way, far enough apart to move. A prompt or model comparison needs the same clean-comparison mechanics, and the alarms belong beside the rest of the feature’s operational metrics.