Exam-style
A merchant's weekly demand forecast, fitted on four years of branch sales, scored 91% against actual sales in March and reads 68% by November. It slid a little at a time rather than stepping down in any one week. Nothing has been deployed, no feed run missed, no incident raised. Over the same months the merchant took on 14 acquired branches in batches, and new trade terms moved contractor buying from Friday to Monday as accounts came up for renewal. What should the sponsor conclude?
Reveal the answer
C. Nothing has broken. The market the model was fitted on has moved, so fund a standing line: weekly accuracy against actual sales, a retraining trigger with an agreed floor, and a named owner
Breakage steps accuracy down in the week it lands and holds it there. A zeroed field would also show in that input’s distribution. This slid week over week from March. The 91% came from production forecasts joined back to actual sales, not training data, and four years of history covers four Novembers. Retraining does restore the number, and then the same slide starts again from the new figure. The scenario names both causes itself. The 14 acquired branches are not in the four years the model was fitted on, and a block of contractor demand now falls on a different weekday. That is model drift: a standing condition with an owner and a budget line, not an incident with a fix.
Q. A forecast that launched at 91% against actual sales reads 68% eight months on. It slid a little at a time, with nothing deployed and no feed missed. What has happened?
A. Nothing has broken. The model still describes the market it was fitted on, and that market has moved. Fund a standing line rather than a repair: weekly accuracy against actual sales, a retraining trigger with an agreed floor, and one named owner.
Why? Breakage steps the number down in one week and shows in an input’s distribution. A slide over months, with every input arriving as expected, is the market moving away from what the model learned. That is drift. It lands in the operating budget rather than the project one, which is why the run cost decides these cases more often than the build cost.