Keeping Watch on Bias After Launch

The situation

A lender scores incoming loan applications with a model that routes each one into three lanes: auto-approve, auto-decline, and hold for a human reviewer. It went live six months ago, after a pre-launch check that compared approval and decline rates across income bands, employment types and regions, found the gaps defensible, and was signed off. That report is a PDF on a shared drive with a date on it.

Nothing has been measured since. The dashboard the team looks at shows one figure, overall agreement with the reviewers’ decisions, and it has sat between 90 and 92 per cent all year. Two things changed underneath it. A broker channel opened in month three and now sends about a third of all applications, in a different intake format. And the decline letters, which used to come from a fixed template, are now drafted by a foundation model on Amazon Bedrock from the model’s reason codes.

The quarterly risk review asks what nobody can answer: how do you know it is still fair? The work is to build a watch that keeps answering it without a person having to remember to ask.

What actually matters

The material lists the features of responsible AI as “bias, fairness, inclusivity, robustness, safety, veracity”, and names the “tools to detect and monitor bias, trustworthiness, and truthfulness” as analyzing label quality, human audits, and subgroup analysis. Fairness asks whether impacts land equitably across the groups a system serves. Veracity asks whether what comes out is true. The fourth term on that list asks whether behaviour holds up once the inputs stop resembling the training data, which is what a new intake channel tests. All three are properties of a system in production rather than properties of a report written before launch, and the pre-launch check measured them once. A single measurement cannot show movement. Bias monitoring compares two measurements taken the same way. That means the cohort definitions, the metric and the slicing all have to be written down and frozen, or the second measurement is not comparable with the first.

Aggregate numbers are where this fails quietly. Overall agreement of 91 per cent is an average over every applicant, and a cohort that is five per cent of volume can go badly wrong while moving that average by half a point. Subgroup analysis is the response: split the results by an attribute and compute the same measure separately for each slice, rather than trusting one figure over everybody. The unit that gets watched has to be the slice. A dashboard that only ever shows the total is capable of staying green through the entire failure.

Three different things can move, and separating them decides what you do next. The incoming data can drift, meaning the applications arriving now look different from the ones the model was trained on. The model’s outcomes per cohort can drift, which is fairness moving. And the ground truth can drift, meaning the labels the team will retrain on are being produced to a different standard than they were. Drift in the incoming data is not the same as drift in the model’s fairness. Data can shift with every cohort’s outcomes holding steady, and fairness can rot with the input distributions looking untouched, because the labels changed meaning. Each of the three needs its own signal.

The last thing to weigh is what the arrangement produces for someone who has to check it. An automated metric catches movement in things somebody already thought to measure, and it catches them at three in the morning. A human audit catches the failure nobody thought to compute, and it costs a person’s afternoon. Both are on the list because they do different jobs. What makes either of them a control rather than a hobby is the paperwork: a named owner, a date, a written finding, and somewhere the finding is kept. A monitoring job with no reader produces graphs and no evidence.

What we’ll filter on

1. Cadence: does this run once, on a schedule, or continuously against live traffic?
2. Model type: does it work on a classic model with feature columns and labels, on a foundation model, or on both?
3. Automated or human: does it compute a number, or does a person read the cases?
4. Slices or totals: does it report per cohort, or does it hand back one aggregate?
5. Evidence: does it leave a dated, owned, written record an auditor can read?
6. Running cost: what does somebody have to do every week to keep it alive?

The landscape

Seven things are on the table, and they cover different parts of the job rather than competing for the same part.

Measuring bias at build time

Amazon SageMaker AI packaged this as Clarify: pre-training bias metrics computed over the raw dataset, and post-training bias metrics computed over a trained model’s predictions. The pre-training measures ask whether the data is skewed before a model exists, which is the ground checking a dataset before you train on it covers. The post-training measures ask whether the model’s outcomes differ across groups: difference in positive proportions in predicted labels, differences in accuracy and error rates by group, and related figures. Clarify moved to maintenance in June 2026 and closed to new customers at the end of July, so a team already running it keeps its reports and a team starting now computes the same figures in its own processing job. The metric definitions are what a practitioner is expected to recognise, and they outlive the service that packaged them. Either way this is a one-off measurement, run against a model version, and it is the thing the six-month-old PDF contains.

Watching a live endpoint

Continuous monitoring of a deployed classic model has four moving parts: the endpoint captures requests and responses to Amazon S3, a baseline describes what the traffic and the outcomes looked like when the model was signed off, a scheduled job compares a window of live traffic against that baseline, and the comparison is published as metrics plus a violations report. SageMaker Model Monitor is the managed version, with four flavours: data quality (are the incoming features distributed as expected), model quality (are the predictions still right, once ground truth arrives), bias drift (have the post-training bias measures moved since the baseline), and feature attribution drift (has the ranking of which features drive the prediction changed). Model Monitor is maintenance-only on the same timetable as Clarify, and existing schedules keep running. The four parts are straightforward to assemble without it, and the monitoring and retraining loop is the same shape whoever runs it. Feature attribution drift is the subtle one worth knowing by name: the model can keep its accuracy while the reason it reaches a decision changes underneath, and that shows up here before it shows up anywhere else.

CloudWatch alarms on the drift metrics

Drift signals become Amazon CloudWatch metrics, and a CloudWatch alarm on a metric is what converts a number into a message that reaches a person. Set a threshold per cohort rather than one for the total, because a threshold on the aggregate cannot fire for the failure this scenario is about.

Evaluating the generative path

The decline letters are foundation-model output, and none of the above applies to them: there are no feature columns, no labels arriving later, and no endpoint of yours to capture. Amazon Bedrock model evaluation is the managed instrument. An automatic evaluation job scores model output over a dataset you supply against built-in metrics. A human-based evaluation job sends the same outputs to a team of reviewers who score them against a rubric you write, which is how anything subjective gets measured. Run either on a schedule and the two runs become a comparison. The AIP-C01 track goes further into probing a generative feature for bias with fmeval at professional depth. For a mixed estate, knowing that the generative path needs a scheduled evaluation job rather than an endpoint monitor is enough.

Subgroup analysis

Slicing, rather than a service. Every one of the measurements above can be computed per cohort or over everybody, and computing it per cohort is the difference between finding this failure and missing it. It costs a group-by. What it needs is the attribute to slice on, kept in the evaluation data and access-controlled, which is why deleting protected attributes to make a model fair defeats the measurement as well as failing at the fairness.

Analyzing label quality on the incoming ground truth

Ground truth here arrives in two forms: the reviewer’s decision on every held application, and, much later, whether the loan performed. Both are labels, and both are produced by people whose standards move. Analyzing label quality means sampling those labels, having a second person adjudicate the same cases independently, and measuring how often the two agree. A falling agreement rate says the labels are drifting, which matters twice over, because those labels are both the yardstick the model is scored against and the training data for the next version. The technique is the same one used to decide whether a reported accuracy figure means anything, applied to labels arriving now instead of labels collected once.

Scheduled human audits

A person reads a stratified sample of real decisions, cohort by cohort, and writes down what they found. It catches what no metric was configured to catch, which in a mixed estate is most of the interesting failures. It is a control when it has a named reviewer, a fixed cadence, a sampling rule, and a written finding that goes somewhere; it is a favour somebody does when it has none of those.

Evaluation

Side by side

Instrument	Continuous	Classic model	Foundation model	Reports per cohort	Catches the unmeasured	Written record
Build-time bias metrics	✗	✓	✗	✓	✗	✓
Endpoint drift monitoring	✓	✓	✗	✓ if sliced	✗	✓
CloudWatch alarms on drift	✓	✓	✓	✓ if per cohort	✗	✗
Bedrock evaluation, automatic	Scheduled	✗	✓	✓ if sliced	✗	✓
Bedrock evaluation, human-based	Scheduled	✗	✓	✓ if sliced	Partly	✓
Label-quality analysis	Scheduled	✓	✓	✓	✗	✓
Scheduled human audit	✗	✓	✓	✓	✓	✓
Aggregate accuracy dashboard	✓	✓	✗	✗	✗	✗

Read the last two columns together. Only one row catches a failure nobody configured a metric for, and it is the row that needs a person. Only one row leaves no record at all, and it is the row this team currently has.

What moved, and what that means

The rows are not interchangeable. Three of the four signals come from something automated, and the fourth arrives only because a person was scheduled to look. Wire the first three and you will catch every failure that resembles a failure you have already imagined.

The solution

Keep the build-time measurement where it is and stop treating it as the answer. Every model version gets its bias metrics computed before it ships, per cohort, using the same cohort definitions as the version before it, and the numbers go into the model card alongside intended use and limitations. That gives every future comparison a baseline with a version number attached, which is what makes two measurements comparable at all. Version the cohort definitions with the model, because a cohort quietly redefined between runs produces a reassuring comparison of two different things.

In production, monitor the endpoint continuously and alarm per cohort. Capture inference traffic to S3, hold the launch traffic as a baseline, run the comparison on a schedule, publish per-cohort figures to CloudWatch, and put alarms on the cohort series rather than the total. Watch the input side and the outcome side separately, since they answer different questions, and treat a data-drift alarm as a prompt to check outcomes rather than as a fairness finding on its own. Model quality lags here for a structural reason worth naming: whether a loan performs is known months later, so the model-quality signal always arrives late. Pair it with something leading, like per-cohort approval and hold rates, which are available the same day.

Run the generative path on its own schedule. A monthly Bedrock evaluation job over a fixed set of reason-code inputs, scored automatically for the measurable properties and human-based for the ones a rubric has to judge, produces a run you can compare with last month’s. Slice its results by the cohort the underlying application belonged to, because a letter-quality average across all applicants hides the same failure the accuracy average does.

Put a human audit on the calendar quarterly, with a named reviewer, a stratified sample across cohorts, and a written finding filed where the risk review can read it. This is what catches the failure with no metric behind it. It is also the cheapest item on the list and the first one that gets skipped, so give it an owner rather than a team.

Then close the loop on the labels. Sample the reviewer decisions monthly, have a second adjudicator work the same cases blind, and track the agreement rate over time as its own metric. Falling agreement means the yardstick is moving, and every accuracy figure computed against those labels is measuring something different from what it measured in January. Fixing it is guidance and relabelling, not retraining.

Three failure modes are worth naming before they happen. An aggregate figure will hold steady through a cohort-sized failure, so any number reported without a slicing is close to uninformative. A drift alarm on the inputs will fire for changes that have no fairness consequence, and if that happens twice in a row without an outcome check, the alarm gets muted and the next one goes unread. And a monitoring job that nobody reads does not function as a control, however green it stays. What an auditor accepts is a named person, a date, and a finding written down. A graph produces none of those.

Worked example

What the aggregate hid

At launch, the model held 22 per cent of applications for review overall, and 31 per cent for self-employed applicants, a gap the pre-launch report examined and accepted. Six months on, the overall hold rate reads 23 per cent, which is why nobody looked. Sliced, self-employed applicants are held 47 per cent of the time and their approval rate has fallen from 54 to 38 per cent. That cohort is seven per cent of volume, so a sixteen-point collapse in its approval rate moves the overall figure by roughly one point, well inside the noise anybody would tolerate on a dashboard.

The cause is upstream of the model. The broker channel that opened in month three submits income evidence in a format the pipeline does not parse, so the verified-income feature arrives null far more often. It arrives null most often for self-employed applicants, whose evidence was already the least standard. The model treats a null as a thin file and routes accordingly. A data-quality monitor on the input would have flagged the null rate in week one. A bias-drift measure sliced by employment type would have flagged the outcome in week two. The team had neither, and the fix is a parser change rather than anything to do with the model.

What the audit found that no metric did

The first quarterly audit read 60 declined applications, 15 from each of four cohorts, along with the letters sent to them. Every metric on the new dashboard was inside its threshold. The reviewer found that letters drafted for self-employed applicants named a vaguer reason than the letters sent to salaried applicants, because the reason codes for a thin-file decline are less specific, and the model filled the gap with generalities. Nothing about that was wrong enough to fail an automatic evaluation, and it produced a letter the applicant could not act on. The finding turned into a specificity check on the next monthly evaluation job, scored per cohort. That is the pattern: an audit finds it once, and a metric watches for it afterwards.

What’s worth remembering

1. A pre-launch fairness check is one measurement, and bias monitoring needs two taken the same way, so freeze and version the cohort definitions and the metrics alongside the model.
2. Subgroup analysis is what makes a cohort-sized failure visible, because an aggregate accuracy or approval figure can sit unchanged while a small cohort’s outcomes collapse.
3. Drift in the incoming data, drift in per-cohort outcomes, and drift in the labels are three separate signals with three separate responses; a data-drift alarm is a prompt to check outcomes, not a fairness finding.
4. Continuous endpoint monitoring covers a classic model, and a scheduled Bedrock model evaluation job, automatic or human-based, covers the generative path, because a foundation model has no feature columns or labels to watch.
5. Analyzing label quality by having a second adjudicator work the same cases catches a moving yardstick, which corrupts both the accuracy figures and the next training set.
6. Human audits catch the failures nobody configured a metric for, and a monitoring arrangement becomes a control only when it has a named owner, a cadence, and a written finding somebody files.