The situation
A produce-box business has 260,000 active subscribers. Every subscriber’s weekly box page shows one suggested add-on item, and since last year a model has picked it. Four suggestions a month per subscriber, so roughly 1.04 million suggestions shown; behind them the model scores every subscriber against the whole catalogue overnight, about 120 million scores a month.
In June the team replaced the original model with a larger one. The quarterly review slide reports the result: the F1 score went from 0.74 to 0.81, and accuracy is 96.4%. The room nods. Then the finance lead points at a line in the cloud bill. The old model ran on a CPU inference fleet at about $2,100 a month. The new one runs on GPUs at about $6,300. Building it took two engineers a quarter and cost roughly $180,000, including paying people to label a test set.
Her question is whether to keep paying for it, and nobody in the room can answer. Everything on the slide describes how often the model is right. Nothing on it describes what being right was worth, what the previous approach earned before any model existed, or how much of the bill is a one-off that has already been spent.
What actually matters
There are two families of number here and they answer different questions. Model performance metrics are measured against a held-out set of examples where somebody has written down the right answer, and they tell you whether the model is right. Business metrics are measured against the ledger and the subscriber base, and they tell you whether being right returned more than it cost. Neither family can substitute for the other. A model with a superb F1 score that nobody uses returns nothing, and a cheap model that gets the wrong subscribers the wrong items has a low bill and a negative return.
Because they measure different things, they can move in opposite directions, and that is what happened in June. The larger model is measurably better at the prediction and costs three times as much per inference to run. Whether the improvement covers the extra bill depends on how much money one additional correct suggestion is worth, which is a number the modelling work never touched. A team that reviews only the model metrics will approve every upgrade that raises them.
The composition of the test set decides how much the model metrics are worth saying out loud. Roughly 3.5% of suggestions get added to a box, so the outcome being predicted is rare. A model that predicted “nobody will ever add this” would be right 96.5% of the time. Accuracy at 96.4% is therefore slightly worse than answering nothing at all, which is why the figure is comforting and useless. When the positive class is rare, report precision and recall, and read accuracy as background.
The last thing to sort before choosing metrics is one-off against run rate. Development costs are spent once and do not recur; inference cost recurs every month for as long as the endpoint is up. Comparing this month’s uplift against a number that includes a $180,000 build makes the model look terrible, and comparing the whole year’s uplift against one month of inference makes it look wonderful. Put the one-off and the recurring figure in different columns and the arithmetic stops depending on which month somebody happens to run it.
What we’ll filter on
- What decision does this number inform: keep the model, retrain it, change the threshold, cancel the project?
- Is it a one-off cost or a run rate, and is it being compared against a matching period?
- Is the positive class rare enough that a headline percentage flatters the model?
- Is there a human-labelled ground truth to measure against, or is the number an estimate?
- Who reads it: the person tuning the model, the person paying the bill, or the person deciding whether the project continues?
The landscape
The metrics worth putting on a review page fall into three groups: what the model got right, what the model cost, and what the business got back.
The confusion matrix, and the four numbers built on it
Every prediction the model makes about a suggestion lands in one of four cells. A true positive is a suggestion the model flagged as likely to be added, which the subscriber then added. A false positive is one it flagged that nobody added. A false negative is an item the subscriber would have added, which the model did not flag. A true negative is an item nobody wanted and the model did not flag. Those four counts, laid out as a two-by-two grid, are the confusion matrix, and every metric in this group is arithmetic over them.
Accuracy is the fraction of all predictions that were correct, true positives and true negatives together over everything. People reach for it first, and it misleads first. When 96.5% of cases are negative, the negatives dominate the fraction and swamp any signal about the rare thing you care about.
Precision answers how many of the flags were real: of everything the model marked as likely to be added, what share actually was. Low precision shows up as subscribers being pestered with suggestions they ignore.
Recall answers how many of the real cases were caught: of everything a subscriber would have added, what share the model flagged. Low recall shows up as revenue that was available and never offered.
Precision and recall pull against each other. Flag more items and you catch more real ones and also collect more wrong ones. The F1 score is a single number that balances the two, the harmonic mean of precision and recall, which stays low unless both are decent. It is a reasonable summary for a review page and a poor basis for a decision on its own, because it hides which of the two moved. For the deeper treatment of how to pick between them from the cost of each kind of mistake, there is a whole method for choosing an evaluation metric.
What the model cost
Cost per inference is the running cost of the endpoint divided by the number of predictions it served. At $6,300 a month over 120 million scores that is about $0.0000525 each, against $0.0000175 for the model it replaced. Small numbers over large volumes. Track the unit figure rather than the monthly total, because volume grows with the subscriber base and the total moves for reasons that have nothing to do with the model.
Cost per user takes the same monthly cost and divides it by active subscribers instead: $6,300 across 260,000 subscribers is about $0.024 a subscriber a month. This is the version that compares against revenue per subscriber, so it is the one a commercial reader can use without a calculator.
Both come out of AWS Cost Explorer, which reports spend by service and, once resources carry cost allocation tags, by project. Tagging the endpoint, the training jobs and the storage with the same project tag is what turns “the SageMaker AI line went up” into “this model costs $6,300 a month”. AWS Budgets does the forward-looking half. Set a monthly amount against the same tag and get an alert when actual or forecast spend crosses it. A runaway retraining job or an oversized endpoint then surfaces in days rather than at the next review.
Development costs are the one-off: engineering time, data preparation, paying humans to label a test set, and the experiments that went nowhere. They are real and they belong in the first-year return calculation. They do not belong in a monthly running comparison, and they are already spent, so they should never decide whether to switch a model off today.
What the business got back
Customer feedback is the signal gathered directly from the people on the receiving end: the thumbs-down control next to a suggestion, survey responses, support contacts complaining that the suggestions are irrelevant. It catches things no held-out test set can, because the labelled set records what subscribers did and feedback records what they thought about it. It arrives biased towards the annoyed, so read the trend rather than the level.
Return on investment (ROI) is the number that decides whether the project is renewed: the value the model produced, less everything it cost, as a proportion of what it cost. It requires two things the model metrics never needed. It needs money attached to a correct prediction, which here is the margin on an added item. And it needs a baseline. The value of the model is the difference between what happened with it and what would have happened without it, rather than the whole revenue of every suggestion that got added.
Evaluation
Side by side
| Metric | What it measures | What it misses | Rare-class safe | Who asks for it |
|---|---|---|---|---|
| Accuracy | Fraction of all predictions that were correct | Everything, when one class dominates | ✗ | Nobody, once they know the base rate |
| Precision | Share of flagged cases that were real | The real cases never flagged | ✓ | The team tuning the threshold |
| Recall | Share of real cases that were flagged | The wrong flags raised along the way | ✓ | Whoever bears the cost of a miss |
| F1 score | The balance of precision and recall | Which of the two moved, and in which direction | ✓ | The review page, as a summary |
| Cost per inference | Running cost of one prediction | Whether the prediction was worth making | ✓ | Engineering and platform |
| Cost per user | Monthly running cost per active subscriber | One-off build spend | ✓ | Commercial and finance |
| Development costs | One-off spend to get the model built | Anything about ongoing viability | ✓ | The budget holder, once |
| Customer feedback | What subscribers think of the output | Silent subscribers, and magnitude | ✓ | Product |
| Return on investment (ROI) | Value returned against total cost | Why the model behaves as it does | ✓ | Whoever decides on renewal |
The table splits along one line. Everything above cost per inference is computed from a labelled test set and needs no commercial input at all. Everything below it needs a price, a baseline and a period, and none of those come out of the modelling work. A review that only ever sees the top half will keep approving models that raise the F1 score, because that is the only lever it can see.
The solution
The review should have had a one-page scorecard with three columns: one model metric, one cost metric, one value metric, all covering the same period, with the previous period beside them.
The model column carries precision, recall and the F1 score, measured on a held-out set that nobody trained on, with the class balance printed next to them so the numbers can be read honestly. Accuracy goes on the page only with the base rate beside it. Refresh it whenever a model is retrained or promoted, and keep the test set fixed between refreshes so a change in the number means a change in the model. Amazon SageMaker AI produces these figures as part of a training or evaluation job. The work is storing them somewhere durable, tagged with the model version that produced them, rather than leaving them in a notebook.
The cost column carries cost per inference and cost per user, both pulled from AWS Cost Explorer filtered to the project’s cost allocation tag, and both stated as a monthly run rate. Development costs sit on the page too, clearly labelled as a one-off with the date it was incurred, so nobody accidentally amortises it twice. An AWS Budgets alert on the same tag means the cost column is never a surprise at the review, because the surprise arrived by email in week two.
The value column carries the uplift against a stated baseline and the return on investment computed from it, plus one customer feedback figure such as the share of suggestions given a thumbs-down. Write the baseline on the page. “Against the popularity rule we used before the model” is a baseline. “Against nothing” is not. A return computed against nothing counts every add-on the business would have sold anyway as a win for the model.
Two failure modes are worth designing out. The first is a scorecard where the periods do not line up, with a quarter of value against a month of cost. Fix the period once, at the top of the page, and make every number obey it. The second is a value figure nobody can trace, where a product manager’s estimate of margin per add-on has become an input to a renewal decision without ever being checked with finance. Put the assumptions on the page with their source. The scorecard’s job is to make disagreement land on the assumption rather than on the conclusion.
Worked example
Run the June decision through the scorecard. The baseline is what the business did before any model existed: the box page suggested whichever item was most popular that week, and 2.1% of those suggestions got added. Margin on an added item averages $1.80. Across 1.04 million suggestions a month, that rule produced 21,840 add-ons and $39,312 of margin.
| Popularity rule | Model A (to June) | Model B (from June) | |
|---|---|---|---|
| F1 score | n/a | 0.74 | 0.81 |
| Acceptance rate | 2.1% | 3.4% | 3.5% |
| Add-ons a month | 21,840 | 35,360 | 36,400 |
| Margin a month | $39,312 | $63,648 | $65,520 |
| Uplift over the rule | n/a | $24,336 | $26,208 |
| Inference cost a month | n/a | $2,100 | $6,300 |
| Cost per inference | n/a | $0.0000175 | $0.0000525 |
| Cost per user a month | n/a | $0.008 | $0.024 |
Month by month, Model B returns $1,872 more margin than Model A and costs $4,200 more to run. It is $2,328 a month worse while showing a materially better F1 score, and no amount of staring at 0.81 will reveal that.
The first-year return on investment tells the same story with the build included. Model A: value $292,032 for the year, against $180,000 of development costs and $25,200 of inference, giving a return of about 42%. Model B: value $314,496, against the same $180,000 build and $75,600 of inference, giving about 23%. Both models pay for themselves. The one with the better prediction pays for itself less well.
That is not an argument against the larger model in general, and it is not a claim that a better prediction is worthless. It says that the seven points of F1 bought a tenth of a percentage point of acceptance, and at $4,200 a month that trade is a poor one at this volume and this margin. Halve the inference cost by batching the nightly scoring instead of serving it from a live endpoint, or double the margin per add-on, and the same model becomes the right choice. The decision belongs to the value column, and the modelling work cannot make it. Getting this comparison in front of the people who fund the work is much of what taking a model from proof of concept to production actually involves. It is the same discussion as asking whether the problem needed a model at all, arriving a year later with real numbers attached.
What’s worth remembering
- Model performance metrics say whether the model is right and business metrics say whether being right returned more than it cost, and a model can improve on the first while going backwards on the second.
- Accuracy is the fraction of predictions that were correct, and it flatters a model whenever the outcome being predicted is rare, so report precision and recall and read accuracy against the base rate.
- Precision is how many of the flags were real, recall is how many of the real cases were caught, and the F1 score balances the two without saying which one moved.
- Cost per inference and cost per user come from AWS Cost Explorer once resources carry cost allocation tags, and AWS Budgets turns the monthly figure into an alert before it becomes a surprise.
- Development costs are a one-off that belongs in the first-year return and never in a monthly running comparison, because money already spent cannot decide whether to switch a model off today.
- Return on investment (ROI) needs a price for a correct prediction and a stated baseline, and customer feedback catches what a labelled test set cannot, which is what subscribers thought of the answer rather than what they did.