Exam-style
A bank's fraud training set holds 200,000 legitimate transactions and 1,100 fraudulent ones. A review of a 600-row sample finds the two annotators who tagged the historical rows disagreed on roughly one label in six, mostly on disputed chargebacks one called fraud and the other a billing error. What should the team do before training?
Reveal the answer
D. Treat them as separate defects: resample or reweight for the imbalance, and adjudicate and relabel for the disagreement
Two defects sit in this dataset and no single action repairs both. The first is class imbalance: 1,100 positives against 200,000 negatives, near 1 in 180, so a model that outputs “not fraud” every time scores over 99 per cent accuracy and catches nothing. That is a tally of rows per class, countable before training. The repairs are resampling, reweighting so each fraud row counts for more in the loss, or collecting more positives. The second defect is label quality, which none of those touches. One label in six disputed puts roughly 180 of the 1,100 fraud rows in doubt. Resampling carries every wrong label through, and oversampling multiplies it. The repair is adjudication by a third reviewer, an annotation guideline covering disputed chargebacks, and a relabelling pass over the affected history. Analysing label quality sits with human audits and subgroup analysis as a bias-detection tool, and it alone corrects the ground truth. The fifth option merges the two. The first two change how the model fits what it is shown, adding no fraud example and correcting no label. The third is the trap: deleting those attributes is fairness through unawareness. Spend pattern, device and merchant mix act as proxies, so the skew survives, and the deleted column is the one needed to measure harm. Subgroup analysis surfaces both: read fraud recall per cohort rather than one aggregate accuracy figure.
Q. 1,100 fraud rows against 200,000 legitimate ones, and two annotators who disagreed on one label in six. What gets fixed first?
A. Both, separately. Resampling or reweighting addresses the imbalance; only adjudication and a relabelling pass addresses the labels.
Why? Class imbalance is a count of rows per class, visible from a tally before any training job runs, and repaired by resampling, reweighting or collecting more positives. Balanced data sits alongside inclusivity, diversity and curated sources as a mark of a dataset fit to train on. Amazon SageMaker Clarify reported pre-training bias metrics for checks of this kind. Its Class Imbalance metric compares the sizes of two facet values rather than the two label classes, and AWS has closed Clarify to new customers. Label disagreement is a defect in the ground truth, and resampling carries every wrong label through with it. Deleting the protected attributes is fairness through unawareness: proxies keep the skew, and the deletion removes the ability to measure it. Keep those columns and restrict who reads them. Slice the results by cohort and both defects show up before training does.