The situation
A same-day grocery delivery service quotes every customer a thirty-minute arrival window. The window comes from a model trained on 2.4 million completed deliveries over eighteen months, using the round, the drop sequence, the distance from the previous stop, the time of day, the day of week, the basket size and a handful of address attributes. The figure the team reports is the share of deliveries that land inside the predicted window, and it has sat at 94 per cent since March. The model retrains monthly. Nobody has had reason to look at that number twice.
Support has been describing something else for two months. Missed windows are not spread evenly across the customer base; the same two clusters keep coming back. Forty outer postcodes, added to the service area five months ago, report vans arriving late often enough that the depot there has stopped reading the window out loud. And customers over seventy report a different failure: the van arrives on time, the handover at the door takes longer than the round allowed for, and every stop after that one slips.
Sliced, the numbers are blunt. The outer postcodes sit at 71 per cent inside the window. Customers over seventy sit at 88 per cent. Both are far from 94, and both are small enough that fixing either would move the headline figure by about a point. What the team has to establish is why each slice is wrong. Three different causes produce this shape, their repairs pull in different directions, and applying the wrong one deepens the failure it was meant to fix.
What actually matters
The reported figure is an average over everybody, and an average is the one summary guaranteed to hide a small group. Six per cent of 2.4 million deliveries is roughly 144,000 missed windows, and where those misses land decides what kind of problem this is. The material files this under responsible AI rather than under model quality. It asks a practitioner to describe the “effects of bias and variance”, giving as examples “effects on demographic groups, inaccuracy, overfitting, underfitting”. Read that list as one sentence. Inaccuracy is the effect the model produces, and the distribution of that inaccuracy across the people using the service is what turns it into a question about fairness. A model wrong at random is a quality defect. A model reliably wrong for one age band is a fairness defect, whatever the aggregate reads.
Two of the three causes come from a pair of terms worth defining from the ground up, because they get used loosely everywhere else. High bias means the model is too simple for the pattern that is actually in the data. It underfits: it never learned the structure it was shown, including structure that exists only inside a small group. The parameters it shares across all customers are set mostly by the majority of them. Its signature is being wrong on the very rows it was trained on. High variance means the opposite failure of fit. The model has enough capacity to follow its training rows closely and has followed them too closely, learning that set’s noise along with its signal. It overfits: nearly right on the rows it saw, considerably worse on rows it did not. Its signature is a wide gap between training error and validation error. “Bias” is doing two jobs in this material. In the bias-and-variance pair it names error from a model that is too simple. In the list of responsible-AI features it names a systematic skew against a group of people. The complaints here are about the second sense, and the first sense is one of the mechanisms that produces it.
The third cause is neither of those. The model can be the right shape, fitted about right, and still be wrong for a group because the training set holds almost no examples of that group to learn from. Under-representation is a property of the data rather than of the model. It is also the one that gets misread, because on a chart it looks like variance: the handful of rows the group does have are fitted closely, and held-out rows from the same group are missed. It poisons its own measurement too. A slice with two hundred held-out rows gives an error figure that moves several points between one random split and the next, so the number that triggered the investigation may itself be noise.
The repairs pull against each other, which is why the diagnosis has to come first. Underfitting wants more capacity or better features. Overfitting wants less capacity, regularisation, earlier stopping, or more data. Under-representation wants rebalancing, reweighting, or a collection effort aimed at the missing group. Add capacity to a model that is already memorising a thin slice and the memorisation gets worse. Regularise a model that underfits a slice and it slides further towards the majority’s shape, which is exactly the group being harmed. There is no repair that is safe to try first. What separates the three is one comparison run twice: training error against validation error, computed overall and then again for every slice, because a model can underfit one group and overfit another in the same training run.
What we’ll filter on
- Slice size: how many training rows does the group contribute, and how many held-out rows is its error computed from?
- Training error on the slice: is the model fitting even the rows of this group that it was shown?
- The gap on the slice: how far above the training error does the validation error sit for this group specifically?
- Direction: are the misses systematic in one direction (always late, always early), or scattered either side?
- Stability: does the slice’s error move by several points when the model is refitted on a different split?
- Missing structure: is there a real driver of delivery time for this group that no feature in the model represents?
The landscape
Three explanations account for a slice that is worse than the average, and three families of repair answer them. They belong on the page together because each repair helps exactly one of the explanations and works against at least one of the others.
The model is too simple: high bias and underfitting
The model’s form cannot express the pattern. A linear model on a relationship that bends, a shallow tree on an interaction that needs depth, or any model asked to predict from features that do not contain the answer. On a whole dataset this shows as a training error that stays stubbornly high no matter how long training runs, with validation error sitting close beside it. Per slice, it shows as a group the model gets wrong on rows it was trained on, and the misses run in one direction, because the shared parameters are pulled towards the majority and the group sits consistently to one side of that fit. A group can be plentiful in the data and still be underfit this way.
The model is fitted too closely: high variance and overfitting
The model has capacity to spare and has spent it on detail that will not repeat. It follows the training rows so closely that it has absorbed their noise, so training error falls towards zero while validation error stops falling and starts rising. High-cardinality features are the usual accelerant: an identifier with thousands of distinct values gives a model somewhere to store a per-row answer, which looks like learning and generalises to nothing. Per slice, this looks like near-perfect training error on a group and a validation error several times higher.
There are too few examples of the group
Not a fitting failure at all. The group is 0.5 per cent of the training rows, so nothing about it can be learned reliably, and whatever the model does there is closer to extrapolation than prediction. Anything genuinely different about the group, a longer average distance between stops or a longer time at the door, is invisible to the loss function, drowned out by the other 99.5 per cent. Two tells separate this from plain overfitting: the row count itself, and the instability of the group’s numbers across splits. Checking a dataset’s balance before anything is trained is the cheaper version of this, covered in deciding whether a dataset is fit to train on.
Adding capacity or features
The repair for underfitting. More depth, more trees, more parameters, a less constrained model family, or, usually better value, a feature that carries the structure the model is missing. A feature beats capacity when you can name the mechanism: if handovers take longer at some addresses, a feature measuring past handover duration teaches the model something no amount of extra depth can invent. This repair is actively harmful applied to a model that is already overfitting.
Regularising, simplifying, and stopping earlier
The repair for overfitting. Regularisation is a penalty that discourages the model from relying too heavily on any one input, which pushes it towards patterns general enough to survive on new rows. Alongside it sit reducing capacity, stopping training at the epoch where validation error bottoms out, and dropping or coarsening the feature the model is memorising. Applied to a slice that is underfit, all of these make that slice worse.
Rebalancing, reweighting, and collecting more
The repair for under-representation. Reweighting tells training to count the group’s rows for more than one each; resampling changes how often they appear; collecting more is the only one of the three that adds information rather than redistributing attention. All three raise the group’s influence on the fit, and all three trade a little accuracy on the majority for a lot on the minority, which is a decision for whoever owns the service rather than for whoever owns the model.
Evaluation
Side by side
| Diagnosis | Training error on the slice is high | Wide train-to-validation gap on the slice | Slice is a thin share of the rows | Overall accuracy still looks fine | More capacity or features helps | Regularising or simplifying helps | More data from the slice helps |
|---|---|---|---|---|---|---|---|
| High bias, underfitting the slice | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ |
| High variance, overfitting the slice | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ |
| Under-representation of the group | Unstable | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
Two columns do the separating. Training error on the slice splits underfitting from the other two on its own: a model that is wrong about rows it was shown is too simple, and nothing about regularisation or resampling addresses that. The thin-share column then splits the remaining pair, which is the split people get wrong, because overfitting and under-representation produce the same wide gap and the same instinct to regularise. Notice also that the two right-hand columns agree for those two rows: more data from the slice helps either way. That is the safe move when the numbers are genuinely ambiguous, and it is also the slowest.
Which diagnosis the numbers point at
The gates run in that order for a reason. Row count comes first because every figure below it is computed from those rows, and a slice too small to measure will send you confidently to the wrong gate. Training error comes next because it is the one signal that cannot be explained away: a model wrong about rows it was trained on is too simple for them, full stop.
The solution
Start by writing down the slices, before touching the model. Delivery round, postcode cluster, age band, tenure, basket size band, time of day. The attributes have to be present in the evaluation data for any of this to be computable, which is the practical argument against deleting a demographic attribute to make a system fair: you lose the measurement rather than the effect. Freeze the definitions and version them with the model, so this month’s per-slice figures can be compared with next month’s.
Then run the one comparison that separates the three causes. Compute training error and validation error for the whole set, and again for every slice, and print the row count beside each. Read the row counts first and set aside any slice whose held-out set is too small to carry a stable number; that slice needs data before it needs a diagnosis. For the rest, read training error before the gap. High training error on a slice means the model never learned it, and the repair is capacity or features. A low training error with a wide gap means the model learned those rows too well, and the next question is how many rows there were. Thin slice: the group is under-represented and the repair is reweighting, resampling or collection. Well-populated slice: it is overfitting, and the repair is regularisation, less capacity, or removing the feature the model is leaning on.
Re-measure per slice after the repair, not overall. This is where teams lose the thread, because every one of these repairs moves the headline number a little and the slice number a lot, in either direction. A reweighted model can improve the outer postcodes by nine points, cost the majority half a point, and report an overall figure that has barely moved; a team watching only the aggregate would conclude nothing happened. Set the acceptance criterion per slice up front so the result is readable when it arrives.
Three ways this goes wrong are worth naming. Regularising to fix a group the model underfits is the most common, because a wide gap on a small slice looks like overfitting and the reflex is to simplify. Simplifying pushes the fit further towards the majority, and the group gets worse. Adding capacity to fix an under-represented group is the mirror image, and it produces a model that memorises the few rows the group has, scores beautifully in training, and behaves unpredictably in production. And chasing a slice whose numbers are noise costs a fortnight and changes nothing, which is why the row count is the first gate rather than an afterthought.
Whatever comes out, the finding goes into the model card. Amazon SageMaker Model Cards exist to hold exactly this: intended use, training data, evaluation results, and known limitations, in one versioned document that travels with the model. Per-slice accuracy alongside the overall figure, the diagnosis for each slice that fell short, the repair applied and what it cost the majority, and any slice still too small to measure. That last entry is the honest one and the one most often left out. Recording it turns an unmeasurable group into a known gap somebody can close, and it feeds the ongoing watch described in keeping watch on bias after launch, which needs a baseline written down before it can detect movement. A group-sized inaccuracy is a fairness finding even while overall accuracy looks healthy. The group most likely to carry one is the group with the fewest rows, and so with the least chance of showing up in the average.
Worked example
The outer postcodes
Forty postcodes, added five months ago, 11,000 of the 2.4 million training rows, about 2,100 in the held-out set. Validation error there is more than triple the average. Training error on the same slice is low, which sends the first reader straight to overfitting and a proposal to regularise. The row count says otherwise. Refitting on three different splits moves the slice’s figure between 68 and 79 per cent, a spread wide enough to make any single reading unreliable. And 0.45 per cent of the rows is not enough to learn a genuinely different pattern from. Stops out there average four minutes of driving apart rather than ninety seconds, and unsealed access roads add time no feature captures. Regularising would have pushed the model further onto the urban pattern and made those postcodes worse. The repair is reweighting the outer rows for the next training run and a targeted collection effort, plus a wider quoted window for those postcodes until the numbers stabilise.
The over-seventies
Nine per cent of deliveries, so 216,000 training rows and no shortage of held-out ones. Training error on the slice is high, and validation error sits right beside it, which rules out both of the other causes immediately: the model is wrong about rows it was trained on. The misses are systematic in one direction, with arrival at the next stop consistently later than predicted. The mechanism is not hard to find once someone looks. Handovers at these addresses take two to three minutes longer, and no feature in the model represents time at the door. This is a group the model underfits because the parameters it shares with everyone else are set by the eight-minute-per-stop majority. Regularising would deepen it and more data would not touch it. The repair is a feature: median handover duration for the address over its last ten deliveries, which lifted the slice to 93 per cent and cost the overall figure nothing.
The one that really was overfitting
The third slice turned up during the same sweep, uninvestigated by anyone. One dense inner postcode of apartment blocks, well represented at four per cent of rows, showed training error near zero and validation error triple it. That is the textbook signature. The cause was a building identifier used as a feature: thousands of distinct values, one for almost every block, which gave the model a place to store an answer per building instead of learning what makes a block slow. Replacing it with three attributes of the building (lift or no lift, number of floors, whether the entry is secured) closed most of the gap. Three slices, three diagnoses, one training run.
What’s worth remembering
- Inaccuracy that clusters on a demographic group is a fairness finding rather than a quality one, which is why the effects of bias and variance sit under responsible AI rather than under model performance.
- High bias means the model is too simple and underfits, showing as high training error on the slice; high variance means it is fitted too closely and overfits, showing as a wide gap between training and validation error on that slice.
- A third cause is neither: too few examples of the group to learn from, which mimics the overfitting signature and is separated from it by the row count and by how much the slice’s figure moves between splits.
- The repairs pull in opposite directions, so applying one before the diagnosis makes the group worse: capacity and features for underfitting, regularisation and simplification for overfitting, reweighting and collection for under-representation.
- Compare training and validation error overall and again per slice, since one model can underfit one group and overfit another in the same run, and read the row count before either figure.
- Record the per-slice results, the diagnosis, the repair and any slice still too small to measure in the model card, because a gap written down is one somebody can close.