Exam-style
A demand model that forecasts how many boxes each depot will need was deployed fourteen months ago. It was accurate at launch and has degraded steadily ever since, and is now missing badly enough that depots are over-ordering. The input data has the same columns it always had, and the distributions of those columns look the same as they did at launch. The training pipeline, the feature code and the deployed artefact have not changed. What has changed is the business: it opened in two new cities and added a weekday delivery option that roughly a third of subscribers took up. What is happening, and what fixes it?
Reveal the answer
C. Concept drift: the relationship between the inputs and the target has changed, so the fix is model re-training on recent data, with SageMaker Model Monitor watching for the next occurrence
The stem rules out three of these with facts rather than argument. Data drift means the inputs themselves change shape: a new age profile, a different mix of box sizes, a column that starts arriving mostly empty. It is the drift SageMaker Model Monitor detects most readily, because it can compare live traffic against a baseline taken from the training data. Here the distributions were checked and they match, so the inputs are not the thing that moved. Training-serving skew is a pipeline bug, where the features a model is scored on are computed differently from the features it learned on. It is wrong from the first request, not fourteen months in, and this pipeline has not been touched. Overfitting is a property of the fit between a model and its training data, and it is visible at training time as a gap between training and validation performance. It does not sit dormant for a year and then emerge. The same goes for badly chosen hyperparameters: a model tuned poorly would have launched poorly, and this one launched accurate. What is left is concept drift, where the inputs look the same but the correct answer for them has changed. Two new cities and a weekday delivery option do exactly that. The same depot, the same day of week, the same subscriber count now imply a different box count, because the meaning of a Wednesday changed when a third of subscribers could suddenly order for one. No amount of retuning helps, because the model is faithfully reproducing a relationship that used to hold. Model re-training on recent data, once the new cities and the weekday option have enough history to learn from, gives the model the current relationship. Model Monitor is the second half of the fix: it will not spot this one on distributions alone, so the job to schedule is the one that joins yesterday’s forecasts to what the depots actually used and alarms on accuracy through a floor.
Q. A demand model has degraded steadily over fourteen months. Same columns, same input distributions, untouched pipeline. Meanwhile the business opened two new cities and added weekday delivery. What is happening, and what fixes it?
A. Concept drift: the relationship between the inputs and the target has changed. The fix is model re-training on recent data, with SageMaker Model Monitor watching for the next one.
Why? The stem hands you the discriminator. Data drift is the inputs changing shape, and these inputs were checked and have not. Training-serving skew is a pipeline bug and bites from the first request, not a year in. Overfitting and untuned hyperparameters both show up at training time, in the gap between training and validation scores, and this model launched accurate. That leaves the case where the inputs look the same and the right answer for them has moved: a Wednesday means something different once a third of subscribers can order for one, and two new cities have demand patterns the model never saw. Concept drift is the half of model monitoring that distribution checks miss, so the practical MLOps answer is a scheduled job that joins forecasts to what the depots actually used and alarms when accuracy drops through an agreed floor.