Exam-style
A model that forecasts how many boxes each depot will need was deployed fourteen months ago, accurate at launch and degrading steadily since; the depots are now over-ordering. Its inputs are the columns they always were (depot, weekday, week of year, subscriber count), with distributions matching the training data. Pipeline, feature code and deployed artefact are unchanged. What changed is the business: the box price went up twice, and a pause-any-week option went live that a third of subscribers now use. What is happening, and what fixes it?
Reveal the answer
C. Concept drift: the relationship between the inputs and the target has changed, so re-train on recent data and monitor each forecast against what the depot used
Three of these are ruled out by the stem. Data drift is the inputs changing shape, and these distributions match the training baseline. Training-serving skew is a pipeline bug, wrong from the first request rather than fourteen months in. Overfitting and untuned hyperparameters show at training time, as a gap between training and validation scores, and this model launched accurate. That leaves concept drift: the same depot, weekday and subscriber count now imply fewer boxes, because a third of subscribers can pause a week at short notice and two price rises made them likelier to. Neither is in the feature set. Re-training on recent data fits the current relationship. The monitoring half is a model quality job: merge what each depot used with the captured forecasts, and alarm through Amazon CloudWatch when accuracy falls below a floor. SageMaker Model Monitor was the managed form of that, and is closed to new customers.
Q. A demand model has degraded steadily over fourteen months. Same columns, same input distributions, untouched pipeline. Meanwhile the box price rose twice and a third of subscribers took up a pause-any-week option. What is happening, and what fixes it?
A. Concept drift: the relationship between the inputs and the target has changed. The fix is re-training on recent data, with a model quality job watching for the next one.
Why? The discriminator is in the stem. Data drift is the inputs changing shape, and these inputs were checked and have not. Training-serving skew is a pipeline bug, wrong from the first request rather than a year in. Overfitting and untuned hyperparameters show at training time, in the gap between training and validation scores, and this model launched accurate. That leaves the case where the inputs look the same and the number of boxes they imply has moved. Neither the pause option nor the price is in the feature set. Distribution checks miss that, so detecting it takes a scheduled job that merges each forecast with what the depot used and alarms when accuracy drops through an agreed floor. SageMaker Model Monitor did that as a managed service, and is closed to new customers.