The situation
A produce-box delivery business runs a churn model in production. Every night it scores active subscribers and writes the two hundred most likely to cancel into a table, and every morning the retention team works down that list with a discount offer. It has done that for eight months.
Three things are now true about it. The retention team says the list has gone stale: fewer of the people on it cancel, and more of the people who cancel were never on it. That is a complaint rather than a number, because nothing measures the model once it has answered. The data scientist who built it left in March. And the notebook that produced it reads from a folder in Amazon S3 called churn-training-final-v2. That folder sits beside churn-training-final and churn-training-v3, with no record anywhere of which one the live model was fitted on.
So the model cannot be improved, because it cannot be rebuilt. Retraining on today’s data produces a different model, and nobody can say whether the difference came from the new data or from a folder chosen by guesswork. The engineering manager wants to know how much machinery to put around it. The options in the room run from a documented runbook and a calendar reminder, through a managed pipeline with a versioned model catalogue and a monitor on the endpoint, to a build pipeline that retrains and redeploys on its own when a drift alarm fires.
What actually matters
MLOps is the operating practice around a machine learning model. It covers everything after somebody proves an idea works and before anybody trusts the result on a Tuesday morning. It exists because two activities that look like the same activity are not. Experimentation is meant to be fast and disposable. Somebody tries eleven feature sets and nine algorithms, keeps notes in cell comments, and throws almost all of it away. Slowing that down with review gates and packaging standards makes the team worse at the part where the value is found. The production path is the opposite. It needs repeatable processes, meaning the same steps run the same way each time rather than in whatever order a person remembers them. It also needs to be reproducible: the same inputs produce the same model artefact next March as they do today. This team lost that second property, which is why nobody can touch the model now.
Managing technical debt in machine learning starts when the experiment code becomes the production code without anybody deciding that it has. Three forms of it are visible here. Undeclared data dependencies: the notebook reads a folder, and nothing anywhere records which folder, which snapshot of it, or which upstream job filled it. The single-laptop problem: the code runs against one machine’s installed library versions, and when that laptop leaves the building the model becomes unbuildable. And skew between training and serving, where a feature is computed one way in the training notebook and another way in the nightly scoring job. Boxes per month over the trailing quarter might be an average over ninety days in one place and over three calendar months in the other. The model then scores everybody slightly wrong, and no error appears in any log.
Production readiness is a short checklist, and this model fails most of it. A versioned artefact, so the thing running has a number and can be named. Recorded lineage from data to model, so you can point at the exact dataset, code and settings behind that number. An approval gate, so a model becomes live because somebody approved it rather than because it was the most recent one anybody trained. A named owner. And a rollback, so last week’s model can be back in service in minutes. Scalable systems is the sibling requirement, and scale here has three directions: more data, more models, and more people. One model looked after by one person needs almost none of this. Six models looked after by four people needs all of it, because the thing that stops scaling first is a human memory of how each one was built.
The last two pieces close the loop. Model monitoring watches the live endpoint rather than the training report. Two things go wrong out there. Data drift is the input changing. The subscriber base skews younger, a new city launches, the average box size moves, and the model is being asked about people unlike the ones it learned from. Concept drift is the relationship changing while the inputs look normal, which is what a competitor’s price cut does to churn. Amazon SageMaker Model Monitor compares live traffic against a baseline captured from the training data and reports when the distributions separate; Amazon CloudWatch carries the alarm. Model re-training is what the alarm is for, and it runs on one of three triggers: a schedule, a drift threshold, or an error budget where the model is left alone until measured accuracy falls under an agreed floor. Monitoring without a retraining path produces a dashboard nobody acts on. Retraining without monitoring produces churn in the deployment history and no evidence that anything improved.
What we’ll filter on
- Change rate: how often does this model actually change, in releases per year?
- Cost of staleness: what does a week of a degraded model cost, in money or in risk?
- Reproducibility demand: how many people need to rebuild a given result, and how long after it was first produced?
- Lineage and audit: does anyone outside the team need to see which data produced which decision?
- Drift exposure: is drift expected here, and does it arrive over days or over quarters?
- Engineering time available: what can this team build and then keep running, on top of the work they already have?
The landscape
The three levels below are steps on a ladder rather than rival products. Each one contains the one before it, and the higher rungs cost engineering time every week, not just once.
Manual, with a documented runbook
The model is trained by a person, on purpose, when somebody decides it needs retraining. Three things make this a level rather than an absence of one. The training data is pinned to a specific versioned location in Amazon S3. The training code lives in source control with its dependencies declared. And a written runbook says how to rebuild, how to evaluate, how to deploy and how to roll back. A model card records what the model is for, what it was trained on and what its measured performance was.
Reproducibility is achievable here and often achieved. It is achieved by discipline rather than by machinery, so it survives exactly as long as the person who has the discipline. Monitoring at this level is whatever CloudWatch reports about the endpoint’s latency and errors. Drift is found by somebody noticing that the results feel wrong.
A pipeline, a registry and a monitor
Amazon SageMaker Pipelines turns the runbook into a definition: process data, train, evaluate, register. Running it produces the same steps in the same order every time. Each run records which data and which code produced which artefact, which is lineage without anybody writing it down. The SageMaker Model Registry catalogues the results as versions in a model group, each carrying an approval status. Deployment reads from the registry, so an unapproved model cannot reach production and the previous approved version is sitting right there for a rollback.
SageMaker Model Monitor takes a baseline from the training data, captures live inference traffic, and compares the two on a schedule for data quality and drift. Breaches become CloudWatch metrics, and CloudWatch alarms turn a breach into a message to the model’s owner. The retraining decision stays with a person: the alarm says the world has moved, and someone starts the pipeline.
CI/CD with automated re-training
Everything above, plus automation of the last human step. A commit to the training code runs the pipeline in a test account, evaluates the resulting model against a held-out set, and registers it. A drift alarm can start the same pipeline without a commit. Approval becomes conditional rather than personal: if the new model beats the live one on the agreed metric by an agreed margin, it is approved and deployed automatically, often to a small share of traffic first.
This is the level that scales to many models, and the level with the sharpest edge on it. Automated retraining and automated deployment are separate decisions, and coupling them without a real evaluation gate ships a worse model faster than a human ever could. It also depends on something the churn problem does not have much of: labels that arrive quickly. You find out whether a subscriber churned about a month after predicting it, so an automatic decision made today is scored on ground truth from four weeks ago.
Evaluation
Side by side
| Level | Reproducible rebuild | Lineage an auditor can read | Detects drift | Re-trains without a person | Carries many models | Engineering time |
|---|---|---|---|---|---|---|
| Manual, with a runbook | ✓ (by discipline) | ✗ | ✗ | ✗ | ✗ | Days to set up, hours per release |
| Pipeline, registry and monitor | ✓ | ✓ | ✓ | ✗ | ✓ | Two to four weeks, then low |
| CI/CD with automated re-training | ✓ | ✓ | ✓ | ✓ | ✓ | Two to three months, plus ongoing care |
Read the first column and the last one together. Reproducibility, which is the property this team has actually lost, is available on the cheapest rung. Nothing about a broken model requires automation to fix. The middle rung adds two properties a runbook cannot give. Lineage exists whether or not anybody remembered to write it, and a drift signal arrives before the retention team complains. The top rung removes a person from a loop. That is worth doing when the loop runs often enough to be a burden, and dangerous when the evaluation gate is weaker than the person it replaced.
Which rung this model belongs on
The solution
Take the middle rung. This model changes a few times a year rather than weekly, its labels are a month late, and one week of a degraded list costs a few dozen retained subscribers rather than a regulatory finding. That combination does not repay a build system that retrains on its own. It comfortably repays a pipeline, a registry and a monitor. The failure here was never a shortage of automation. No artefact recorded which data produced which model, and everything else followed from that.
The first three weeks have a shape. Week one is archaeology and pinning. Retrain on each of the three S3 folders and compare the predictions against the stored scores from a night the model was healthy. Whichever matches becomes the recorded training dataset, with S3 versioning switched on so a folder can never change unseen again. The notebook’s forty cells of cleaning move into a script with declared dependencies, in source control, producing the same clean dataset from the same input. Nothing is automated yet; the goal is one command that any of the four engineers can run to rebuild the current model exactly.
Week two is the pipeline and the catalogue. That script becomes a SageMaker Pipelines definition with four steps: process, train, evaluate, register. The evaluate step scores the candidate against a held-out set and refuses to register anything below the floor the live model set. The registry holds versions in one model group, each with its lineage, its metrics and an approval status. The nightly scoring job reads the approved version rather than a file path somebody typed. The rollback stops being a rebuild and becomes an approval change. Write the model card in the same week, while the archaeology is still fresh.
Week three is the watch. Capture live inference data from the scoring job, take a baseline from the pinned training dataset, and schedule a SageMaker Model Monitor job to compare them daily for data quality and drift. Put a CloudWatch alarm on the breach metric, pointed at the named owner. Then add the slower measurement that catches concept drift. A monthly job joins the scores from thirty days ago to who actually cancelled, and reports precision on the top two hundred. That number is the one to set an error budget against, and when it falls through the floor, someone runs the pipeline. This is the same closing loop that a feedback loop from users back into a model builds for generative features, and the same production standard a proof of concept has to reach before it is production.
Two gotchas are worth naming before the build starts. Model Monitor needs data capture switched on and a baseline computed, and neither is retrospective; nothing can be said about last month. And the daily monitoring jobs are real instances on a schedule, so an hourly monitor on a model that is scored once a night is a bill without a benefit. Match the cadence of the watching to the cadence of the deciding.
What’s worth remembering
- MLOps separates experimentation, which should stay fast and disposable, from the production path, which needs repeatable processes and a rebuild that produces the same artefact from the same inputs.
- The technical debt worth managing is undeclared data dependencies, code that runs on one person’s laptop, and features computed differently at training and serving time, none of which shows up as an error.
- Production readiness is a versioned artefact, recorded lineage from data to model, an approval gate, a named owner and a rollback; SageMaker Pipelines produces the lineage and the Model Registry holds the versions and the approval.
- Model monitoring watches the live endpoint for data drift, where the inputs change, and concept drift, where the relationship changes while the inputs look normal; SageMaker Model Monitor compares live traffic to a training baseline and CloudWatch raises the alarm.
- Model re-training runs on a schedule, a drift threshold or an error budget, and automated re-training without an evaluation gate is a way to ship a worse model faster.
- Scalable systems matter as the count of models and of people rises: one model tended by one person sits happily on a documented runbook, and six tended by four does not.