The situation
A produce-box business packs and despatches around 40,000 boxes a day out of four depots. The operations team has one budget line for the year and four proposals competing for it, each introduced in the same meeting as an AI project.
The first is demand. Somebody has to decide how many boxes tomorrow’s packing line is staffed and stocked for, and today that number comes from a supervisor’s spreadsheet and six years of experience. When it is low, agency staff get called in at short notice; when it is high, produce is packed that nobody ordered.
The second is refunds. A subscriber whose box arrives late, short or damaged is entitled to money back under terms published on the website. A fixed percentage for a late delivery, the item value for a missing item, a full refund inside a defined window. About 600 refund requests a day are worked out by hand by support agents reading those terms off a wiki page.
The third is address quality. Roughly one delivery in every 160 fails because the address is wrong. A new-build with no unit number, a rural property with a gate code in a comment field, a flat block where the driver cannot get past the lobby. Two people spot-check a couple of thousand addresses a day and catch perhaps a fifth of the failures before they happen.
The fourth is a subsidised box scheme funded by a state health department. Eligibility is set out in the funding agreement: a household income below a stated threshold, an address in one of a listed set of postcodes, and a referral signed by a clinician within the past twelve months. An administrator works through about 300 applications a week against a printed checklist.
All four have data behind them. Two of them are not machine-learning problems at all, and one of those two is the most expensive mistake available in this room.
What actually matters
Start with what a trained model actually hands back, because it settles two of the four before anything else gets a vote. A model does not look up an answer. It produces a score across the possible answers and returns the one that scores highest, and that ranking comes from patterns in the examples it was trained on. Trained well, it is right most of the time. It is also wrong on individual cases without any signal attached saying which ones, and an accuracy figure of 99.4% describes a population of past cases rather than promising anything about the next one.
That property disqualifies any task whose requirement is that a named input always produces one fixed answer. Refund entitlement is fixed by published terms; two days late is 25%, and it is 25% for every subscriber, every time, or the business is in breach of what it wrote down. Eligibility for the subsidised scheme is fixed by a funding agreement, and an applicant who meets the three stated criteria is eligible as a matter of fact rather than as a matter of probability. These are the situations where a specific outcome is needed instead of a prediction, and they belong to a rule, a lookup or a calculation no matter how good the model looks in evaluation. The 99.4% classifier gets around four refunds a day wrong, roughly a thousand a year. Each one is either a subscriber underpaid against published terms or the business paying out money it did not owe. Nobody downstream can tell which four they were without redoing the calculation by hand.
The other half of the decision is where AI genuinely adds value, and it comes in three shapes. It can assist human decision making, where the model narrows, ranks or flags and a person makes the call, so its mistakes land in front of somebody able to absorb them. It can deliver solution scalability, where the volume of judgements has outrun the number of people you could plausibly hire. A consistent, mediocre judgement applied to 40,000 cases every night beats an excellent one applied to the 2,000 you have time for. And it can deliver automation, where a repeated judgement is handed over entirely because the error rate is acceptable, measurable and cheap to be wrong about. The demand and address proposals sit in the second and third shapes; the refund and eligibility proposals sit in none of them.
Then the money, which is where proposals usually go quiet. Cost-benefit analyses for a model have four cost lines, not one. There is the development cost. There is the labelling cost, because supervised learning needs somebody to say what the right answer was on thousands of past cases, and on the address proposal that means a person marking up historical deliveries. There is the running cost per prediction, small individually and worth multiplying by the volume anyway. And there is the cost of the review process that catches the model’s mistakes. That line never appears on the slide and is often the biggest of the four, because it is a standing human commitment rather than a one-off build. Against that sits the value of the decisions improved. A model that saves two minutes on a task performed nine times a day saves about 75 hours a year, and no build, labelling exercise and review process is ever recovered out of 75 hours.
The last thing to weigh is who catches a wrong answer. A wrong demand forecast is visible by lunchtime and corrected by calling in agency staff. A wrongly flagged address costs a person thirty seconds of looking at it. A wrong refund is invisible: it is a number on a statement that looks exactly like a right one. Error tolerance is not a property of the model, it is a property of the process the model sits in, and it is decided by whether anybody would notice.
What we’ll filter on
- Fixed outcome or best estimate: does a named input have to produce one defined answer every time, or is a good estimate genuinely acceptable?
- Cost of a wrong answer, and who catches it: is a mistake cheap and visible, or expensive and invisible?
- Payback: does the value of the improved decisions cover the build, the labelling, the running cost and the review process?
- Human in the loop: is there a person between the output and the consequence, and do they have enough context to overrule it?
- Writable as a rule: could somebody write the decision down as criteria and arithmetic that another person could follow?
The landscape
Five ways of getting a repeated decision made sit on the table here, and only three of them involve a model.
A rule, a lookup or a calculation
Ordinary deterministic code. The published refund terms become a table of conditions and percentages, read by a small function; the eligibility criteria become three checks and an answer. The same input gives the same output today, next year and during an audit, and the reasoning can be printed out and handed to the person who wrote the terms. The build is a fortnight of engineering, with no labelling, no training and no retraining. At 600 calls a day on AWS Lambda, with the terms held in Amazon DynamoDB, the running cost rounds to nothing. Testing is unusually easy, because the worked examples in the terms themselves become the test cases. Where the rules are numerous and change often, they belong in a table the policy owner can edit rather than in a code branch nobody outside engineering can read.
A model that a person acts on
The model produces a score, a flag or a ranked list, and a human decides. Error tolerance is high because a person is absorbing the mistakes, and the benefit is that human attention gets spent where it pays instead of spread evenly over everything. This is the assist human decision making shape, and it is the safest place to put a first model, because a bad one shows up as people ignoring the list rather than as money leaving the business.
A model that acts without a person
The output goes straight into a system that does something: reorders stock, routes a van, sends a message. This is automation, and it is justified when the volume makes human review impossible, the wrong answer is cheap and reversible, and the error rate can actually be measured after the fact. It needs monitoring and a sampled review of decisions from the day it launches, because an unwatched model that drifts produces confident nonsense for months.
A managed AI service instead of a model you train
Amazon Comprehend, Amazon Textract, Amazon Rekognition and Amazon Personalize give you somebody else’s trained model behind an API, which removes the development and labelling costs and most of the delay. Amazon SageMaker AI is where you go when the judgement is specific to your own data, as tomorrow’s box demand is. A managed service changes the cost line and the time to launch. It does not change the nature of the output, which is still an estimate, so it does nothing for the two proposals that need a fixed answer.
Leaving it with the people
The spreadsheet, the checklist and the two staff doing spot checks are the baseline every proposal is measured against, and costing that baseline honestly is the step most often skipped. The boring baseline sometimes wins outright, and where it does not, its error rate is the number the model has to beat before anybody claims a benefit.
Evaluation
Side by side
| Project | Estimate is acceptable | Wrong answer cheap and caught | Payback clears | Human in the loop | Not writable as a rule |
|---|---|---|---|---|---|
| Demand forecast | ✓ | ✓ | ✓ | ✓ | ✓ |
| Refund entitlement | ✗ | ✗ | ✗ | ✗ | ✗ |
| Address flagging | ✓ | ✓ | ✓ | ✓ | ✓ |
| Statutory eligibility | ✗ | ✗ | ✗ | ✗ | ✗ |
The first column does nearly all of the filtering, and the two rows that fail it fail everything after it for related reasons. A decision that has to come out one fixed way is already written down as criteria, which is the last column. A mistake in it is a breach rather than an inconvenience, which is the second. And the fixed answer can be computed for a rounding error, so a model has nothing to pay back with, which is the third.
The human-in-the-loop column is worth reading carefully on the eligibility row, because there is an administrator reviewing every application and it still gets a cross. A reviewer who has to check the three criteria in order to know whether the model was right has done the entire job the model was meant to do. A person in the loop absorbs errors only when checking is cheaper than deciding, which is true of a flagged address and false of a rule the reviewer must apply from scratch.
Which way the decision runs
The cost-benefit arithmetic
Address flagging is the proposal with a payback worth writing down. One delivery in 160 fails on the address, so 40,000 a day produces about 250 failures. A failed delivery costs a redelivery run plus a support contact, call it fourteen dollars all in, or around $3,500 a day. The two staff doing spot checks catch about a fifth of them. A model that ranks the day’s deliveries by how likely the address is to fail, handed to the same two people as a worklist of 1,500, plausibly gets them to two thirds. That is roughly $1,600 a day recovered, near enough $400,000 a year. Set against it: a build in the low hundreds of thousands, a labelling exercise over historical failures that a person can do in a fortnight, and a running cost of cents per thousand predictions. No new headcount either, because the review is done by the staff already doing the spot checks. It clears in the first year and keeps clearing.
Compare the ask that was dropped before this shortlist was drawn up. Finance wanted help drafting the weekly supplier note, nine of them a week, two minutes saved on each. That is about 75 hours a year of somebody’s time, which no build recovers. The arithmetic runs the same way every time. Multiply the saving by the frequency before designing anything, because a real saving on a rare task is still a rounding error. That is what cost-benefit analyses are for.
The solution
Refund entitlement goes to deterministic code. The published terms become a versioned table of conditions and amounts, the calculation becomes a small function over it, and the worked examples in the terms become the test suite. Every refund is then reproducible two years later, the support team stops reading a wiki page under time pressure, and a change to the terms is one edit with a date on it. A model still has a job nearby. The free-text complaint arriving with the request has to be sorted, so the system knows whether the subscriber is reporting a late box, a missing item or damaged produce, and Amazon Comprehend does that without a training set. The classification is an estimate and an agent can correct it. The entitlement that follows from it is arithmetic.
Statutory eligibility goes the same way, and harder, because a rejected applicant has a right to be told which criterion they failed and the funder can audit any decision made under the agreement. Three checks against an income threshold, a postcode list and a referral date are a morning’s work to write and a permanent asset. Where the clinician’s referral arrives as a scanned form, Amazon Textract lifts the fields off the page and a person confirms them, and the eligibility decision is then made in code against the confirmed values. Reading the form is a machine-learning problem. Deciding eligibility is not.
Demand forecasting goes to a model. Nobody can know tomorrow’s number, so an estimate is the only thing on offer and the supervisor’s spreadsheet is already one with a worse error rate. Six years of despatch history gives Amazon SageMaker AI plenty to learn from. The decision is made once a day per depot, which is low volume with high value per decision, so the payback comes from accuracy rather than from scale. Keep the supervisor’s override, publish the forecast next to what actually happened, and measure both against the spreadsheet for a season before retiring it.
Address flagging goes to a model with a person in front of every action it prompts. This is the solution scalability case: 40,000 judgements a night is beyond any headcount the business would fund, and the two people already doing the work become the review capacity rather than the bottleneck. The model ranks and never edits an address by itself, so a false flag costs half a minute and a missed flag leaves the business exactly where it is today. Where the flagged address gets corrected automatically, that step becomes automation and needs its own sampled audit, because a confidently rewritten address is a new failure mode rather than the old one fixed. Placing the human deliberately is what keeps the error rate survivable while the model is still new.
Two habits are worth carrying out of this budget round. Write down the baseline’s error rate before building anything, because a model with no baseline to beat can only be evaluated on how impressive it feels. And decide, for every proposal, what the wrong answer costs and who would notice it, because that answer decides whether you are looking at automation, at assistance, or at a rule you should have written instead. The same reasoning applies when the model on offer is a generative one, which is worked through in deciding whether to use generative AI at all and, from the engineering side, in the case against reaching for a language model.
Worked example
The refund model’s error budget
Suppose the refund proposal went ahead anyway and trained beautifully: 99.4% accuracy on a held-out set of past refunds, better than the support team’s own consistency, and a demo that lands well. At 600 refunds a day that model is wrong about four times a day and roughly a thousand times a year. Each of those is either a subscriber given less than the published terms promise, or a payment the business did not owe, and every one of them is a plausible-looking number on a statement.
Now try to catch them. A reviewer would have to read the delivery record, find the relevant clause and work out the entitlement. That is the calculation you declined to write, done by hand, a thousand times, without knowing which cases to look at. Meanwhile the rule version is wrong only when somebody has misread the terms while writing it, which surfaces the first time a test case disagrees and is fixed once for every future case. Same decision, two very different failure shapes.
Where the two shapes meet
The refund path shows both shapes in one flow. A subscriber writes “box turned up Thursday, no eggs again”. Which of those is a lateness claim and which a missing item is a language judgement over messy text. Being wrong is cheap, because an agent reads the result before any money moves, and the answer shape is a label drawn from a fixed set. That is a classification problem and a model does it well. What the subscriber is owed, once those facts are agreed, is a percentage of a box price and the value of a carton of eggs, which is arithmetic and belongs in code. Splitting the flow at the seam gets the speed of the model and the certainty of the rule, and it is the shape most of these decisions want.
What’s worth remembering
- A model returns a best estimate and can be wrong on any single case without warning, so any task where a specific outcome is needed instead of a prediction belongs to a rule, a lookup or a calculation.
- Published terms, statutory criteria and contractual thresholds are already written down as rules, and writing them as code is cheaper, reproducible and auditable in a way no accuracy figure makes a model.
- AI/ML adds value in three recognisable shapes: assist human decision making, solution scalability where the volume outruns any headcount you would fund, and automation of a repeated judgement with a measurable error rate.
- Cost-benefit analyses have to count development, labelling, the running cost per prediction and the standing cost of the review process, against the value of the decisions improved.
- Multiply the saving by the frequency before designing anything, because two minutes saved nine times a day never repays a build.
- Ask who would notice a wrong answer: cheap and visible points at automation, expensive or invisible points at a person deciding with the model assisting, or at no model at all.