Proving a GenAI Feature Paid for Itself

The situation

An online retailer sells clothing and homeware to 180,000 paying members. Members browse and order whenever they like; the membership covers delivery and early access to new ranges. The site takes about 2.1 million sessions a month.

Six weeks ago the team shipped a shopping assistant built on Amazon Bedrock. It sits on product pages and in the basket, and it handles four kinds of question: what a product is like, when it will arrive and how to send it back, what size to order, and what to buy someone as a gift. It has held about 240,000 conversations in the last full month. The Bedrock bill for that month was $9,600, with another $1,900 for the surrounding infrastructure.

A finance review is a fortnight away, and the decision on the table is whether the feature continues. The numbers currently on the slide are an evaluation set of 300 questions scored at 91% accuracy, a ROUGE-L score of 0.42 on the summaries the assistant writes of product reviews, and an average human helpfulness rating of 4.2 out of 5. Every one of those numbers is real, and none of them answers the question being asked.

What actually matters

There are two different claims hiding in the word “works”. The evaluation scores support the first one: given a question, the model produces an answer that is correct and reads well. That claim is settled by measurement against a held-out set of questions and reference answers, which is exactly what an Amazon Bedrock evaluation job produces. The second claim is that the retailer sells more, keeps members longer, or spends less on support because the assistant exists. Nothing in an evaluation job can tell you that, because the evaluation never leaves the test set.

The two families also run on different clocks and go to different readers. A quality score moves the same afternoon somebody changes the prompt or swaps the model, and it is read by the people building the feature. A business value figure moves over weeks as behaviour changes, and it is read by the person deciding whether to renew the budget. Presenting one to the audience expecting the other is how a feature with excellent scores gets switched off.

Attribution is where most of these reviews fall apart. Members who open an assistant conversation about a jacket are already closer to buying that jacket than members who never open one, so comparing those two groups measures intent as much as it measures the assistant. The way out is a baseline captured before launch, or better, a holdout: a slice of members who never see the feature, chosen at random, whose numbers run alongside the exposed group for as long as the comparison is needed. A metric with no baseline measured before launch proves nothing, and it cannot be reconstructed afterwards from a data warehouse.

The last thing to weigh is whether one model covers all four jobs. Cross-domain performance is the term for how well a single model holds up across tasks that are unlike each other, and it decides how many models the retailer has to run. One model that is adequate on product questions, delivery policy, sizing and gifting means one prompt catalogue, one evaluation set and one bill. A model that collapses on one of the four means either accepting bad answers in that lane, routing it to a second model, or turning it off, and each of those has a cost that belongs in the same sum as the token spend.

What we’ll filter on

1. Audience: who reads this number, and what decision does it let them make?
2. Clock speed: does it move in a day, a month, or a year, and is a six-week review long enough to see it?
3. Attribution: can the change be traced to the feature, and is there a baseline or a holdout to compare against?
4. Gameability: could somebody improve this number without the business being any better off?
5. Source: does it come from an evaluation job, from application telemetry, or from the finance and support systems?

The landscape

The measurements available fall into four groups. A review page needs something from more than one of them.

Model-performance metrics

These score the output against a reference. Accuracy here means the share of test questions the assistant answered correctly, judged against answers a human wrote first; the 91% on the slide is that. ROUGE compares generated text against a reference summary by counting overlapping words and phrases, so it suits summarisation and says little about a conversational answer. Faithfulness, sometimes called groundedness, asks whether every claim in the answer is supported by the retrieved product data rather than invented, which is the score that catches hallucination. Human ratings, such as the 4.2 out of 5, catch tone and usefulness that automated scores miss, and cost money every time you refresh them.

All of these share the same properties. They move fast, they are fully attributable to the model because nothing else feeds them, and they are computed from a fixed test set, so they are honest as long as nobody trains on that set. They are also the metrics engineers reach for first, and the ones a finance reviewer has no use for. Choosing which of them to optimise is a separate exercise driven by what a wrong answer costs.

Cross-domain performance

The same evaluation set, sliced by the kind of question rather than reported as one number. Here that breakdown reads: product questions 94%, delivery and returns 96%, size and fit 88%, gift suggestions 71%. The headline 91% is a traffic-weighted average of those four, and it conceals the only interesting figure in the set.

Cross-domain performance is what tells you whether one foundation model is serving several tasks well enough to avoid running several models. Gift suggestions are 12% of conversations and score 71%, so roughly 8,400 conversations a month get an answer the team would not defend. Fixing that with a second, fine-tuned model means a second evaluation set, a second prompt catalogue and a second line on the bill, all for an eighth of the traffic. Routing gifting to a curated list instead, or dropping it, are the other two answers. Reporting one blended accuracy figure hides the choice entirely.

Operational metrics

Cost per interaction is the running cost divided by conversations: $11,500 over 240,000 comes to about 4.8 cents. It is the figure to track rather than the monthly total, because the total moves with traffic for reasons that have nothing to do with the model, and it is the figure that the token bill drives directly. Latency matters because a member abandons a slow answer, so a median and a 95th percentile belong on the page next to it. Task completion rate is the share of conversations that reached the thing the member came for, measured by what happened next: an item added to the basket, a returns label printed, a size chosen.

Task completion sits between the two families. It is computed from application telemetry rather than a test set, it moves in days, and it is the earliest honest signal that quality changes are reaching real behaviour. It is also the easiest of these to game, because the definition of “completed” is written by the same team that reports it.

Business metrics

Conversion rate is the share of sessions that end in an order. Average revenue per user is total revenue over active members for the period, and it captures both people ordering more often and people ordering more each time. Customer lifetime value is the margin a member is expected to produce over the whole time they stay, usually average revenue per user multiplied by gross margin and by expected membership length; it is the metric a retention improvement shows up in, and it moves too slowly to be measured six weeks after launch. Efficiency covers the cost side: support contacts per hundred orders, average handle time on the contacts that still happen, deflection rate for questions the assistant answered that would otherwise have become a contact. ROI, return on investment, is the sum that puts value and cost in one number: value produced, less what it cost, over what it cost.

These are the numbers the finance review is asking for, and they all need three inputs that no evaluation job provides. A price attached to the behaviour, a period, and a baseline. The same exercise for a traditional predictive model, where the quality side is a confusion matrix rather than an evaluation set, is a scorecard with one model metric, one cost metric and one value metric. That one has a baseline waiting for it: a rule was already picking the suggestion before any model existed, so the uplift can be read off against what the rule earned. Here nothing was running before the assistant shipped, and a baseline that does not exist has to be built on purpose.

Evaluation

Side by side

Metric	What it says	Audience	Moves in	Attributable	Hard to game
Accuracy	Share of test questions answered correctly	The team building it	Hours	✓	✗
ROUGE / faithfulness	Output matches a reference, or the retrieved source	The team building it	Hours	✓	✓
Human rating	Whether an answer is actually useful	Product	Days	✓	✓
Cross-domain performance	Whether one model covers every task	Product and platform	Hours	✓	✗
Cost per interaction	Running cost of one conversation	Engineering and finance	Days	✓	✓
Latency	Whether members wait long enough to leave	Engineering	Minutes	✓	✓
Task completion rate	Conversations that reached what the member wanted	Product	Days	✗	✗
Conversion rate	Sessions that ended in an order	Commercial	Weeks	✗	✓
Average revenue per user	Revenue per active member per period	Commercial and finance	Weeks	✗	✓
Efficiency (contacts, handle time)	Support work removed	Operations	Weeks	✗	✗
Customer lifetime value	Margin over a member’s whole membership	Finance and the board	A year	✗	✓
ROI	Value returned against everything it cost	Whoever renews the budget	A quarter	✗	✓

The attributable column is where the table splits. Everything marked attributable comes from a test set or a service metric and belongs to the model alone. Everything marked otherwise needs a holdout or a before-and-after baseline before the number means anything, and that is the whole bottom half of the table, which is also the half the review cares about.

The gameable column sorts on one question: who writes the rule that produces the number. Conversion rate, revenue per member and the token bill are counted by systems the team does not own, and a figure from the ledger is hard to argue with. Every row marked otherwise is one where the team reporting the number also decides what it counts: which questions stay in the evaluation set, which of the four lanes gets quoted as the headline, what a completed conversation looks like. None of those move because somebody was dishonest; they drift, slowly, in the direction the target points. Anything from the second sort that has to carry a target needs its definition written down and dated first, so a rise can be checked against the rule it was measured by.

The solution

Give each job the assistant does one quality metric and one business metric, on the same page, for the same period, against a stated baseline. Agreeing that pairing early is part of getting a feature out of proof of concept, because the baseline has to be captured while the feature is still off.

For product questions and sizing, that pairs faithfulness against conversion rate. For delivery and returns, it pairs accuracy against support contacts per hundred orders. For gifting, it pairs the 71% cross-domain score against the gift-order rate, which is the pair that makes the case for fixing it, routing it elsewhere, or removing it. Two numbers per job, no more, because a page with forty metrics on it gets read as a page with none.

Keep the holdout running. Ten per cent of members, chosen at random, who never see the assistant, is enough at this size and costs the 10% of the uplift you decline to collect. It is the only thing that converts “members who used the assistant converted better” into “the assistant caused members to convert better”, and once it is switched off it cannot be recreated. Where a holdout is impossible, the fallback is a baseline period measured before launch on exactly the metrics you intend to report, written down with its dates.

Report customer lifetime value as an assumption rather than a measurement. Six weeks is not long enough to observe a change in how long members stay, so state the model being used, state the retention figure feeding it, and revisit at twelve months. A lifetime value number produced from six weeks of data is arithmetic on a guess, and finance reviewers can tell.

Then do the ROI sum, once, with the run rate and the build cost both in it, and put the assumptions on the page next to the answer. Where the sum comes out uncomfortable, the levers are the same ones that reduce a Bedrock bill without hurting quality: shorter prompts, capped output lengths, a smaller model on the easy lanes, batching anything nobody is waiting for.

Worked example

Here is the sum the review needs, run on the last full month. The holdout is 18,000 members; 162,000 see the assistant. Gross margin is 40%, and the average order is $52.

	Holdout	Exposed	Difference
Average revenue per user, monthly	$38.10	$38.31	+$0.21
Conversion rate	3.08%	3.19%	+0.11pp
Support contacts per 100 orders	5.2	4.4	-0.8

Revenue first. $0.21 more per member per month, at 40% margin, is $0.084 of margin per member. Across 162,000 exposed members that is $13,608 a month.

Then efficiency. The exposed group placed about 119,000 orders in the month, so 0.8 fewer contacts per hundred orders is roughly 950 contacts that never happened. At a fully loaded $4.60 to handle a contact, that is $4,370 a month.

Total value: $17,978. Running cost: $9,600 of Bedrock tokens plus $1,900 of surrounding infrastructure, so $11,500. The feature covers its running cost with about $6,478 a month left over.

The build cost $140,000 in engineering time, data preparation and paying humans to write the 300 reference answers. First-year ROI is therefore ($215,736 of value less $278,000 of total cost) over $278,000, which is about -22%. On the same run rate the second year, with no build to pay for, returns about +56%, and the build is repaid somewhere around the twenty-second month.

Now the number nobody should have shown. Sessions containing an assistant conversation converted at 5.4% against 3.0% for the rest of the site. Applying that 2.4-point gap to 240,000 conversations gives 5,760 extra orders, $119,808 of margin a month, and a return that looks like ten times the cost. The holdout says the real figure is $13,608, less than an eighth of it. That gap is members who were already going to buy, asking a question on the way. Both numbers came out of the same database on the same afternoon, and only one of them survives being asked how it was worked out.

The honest answer to the finance review is that the assistant covers its running cost today, repays its build in the second year, and has one lane at 71% that needs a decision. That answer holds up under questioning. The 91% on the original slide does not, because nobody in the room was asking about the test set.

What’s worth remembering

1. Model-performance metrics such as accuracy, ROUGE and faithfulness say the model produces good answers; business value metrics say the organisation is better off, and a feature can be excellent on the first while showing nothing on the second.
2. A business metric with no baseline measured before launch proves nothing, and a randomised holdout is the only cheap way to turn a correlation between usage and revenue into an attributable effect.
3. Conversion rate and average revenue per user move in weeks and can be measured at a six-week review; customer lifetime value moves over a year and should be reported as a stated assumption until there is enough history to observe it.
4. Cross-domain performance, meaning the evaluation set sliced by task rather than blended into one figure, decides whether one model covers every job or whether a lane needs a second model, routing away, or switching off.
5. ROI needs a price for the behaviour, a period, and the one-off build cost alongside the monthly run rate, and it should be published with its assumptions so disagreement lands on an input rather than on the conclusion.
6. The measures easiest to improve without helping anybody are the ones whose counting rule the reporting team writes, such as task completion and the sliced accuracy figures, so fix and date those definitions before either becomes a target.