Exam-style
An ecommerce team shipped a product-recommendation assistant on Amazon Bedrock six weeks ago. It holds a ROUGE score of 0.42 on its summarised product blurbs and a human-preference win rate of 68% against the previous rules-based widget. The CFO has asked the team to justify next year's budget for the feature. Which number answers what has been asked?
Reveal the answer
C. Report conversion rate and average revenue per user for sessions that saw the assistant, measured against the baseline captured before launch
Sort the numbers by who is asking. ROUGE, the human-preference win rate and accuracy on a benchmark dataset are model-quality metrics: they say the model does its job well. Token throughput and p99 latency are system metrics: they say the software is healthy and fast. All five describe the assistant. A budget decision asks a different thing, which is what the business got back for what it spent, and that is business value. Conversion rate, the share of sessions ending in an order, and average revenue per user, revenue over active shoppers for the period, both move within weeks and both attach to the feature the money is being spent on. Together they cover shoppers buying more often and shoppers spending more when they do. Put those against the cost of running the assistant and you have ROI, return on investment: value produced, less what it cost, over what it cost. That is the frame a CFO is working in. Customer lifetime value belongs on the same list but answers a slower question, since it depends on how long shoppers stay, and six weeks of history cannot show a change in retention; report it as a stated assumption and revisit it after a year. None of this works without a baseline. Conversion rate of 3.1% after launch proves nothing on its own, because the number needs a before to be measured against, and the before has to have been captured while the old experience was still running. Where no baseline exists, a holdout group that never sees the assistant gives you one from here, at the cost of a few more weeks.
Q. Six weeks of a live recommendation assistant, a ROUGE score of 0.42, a 68% human-preference win rate, and a CFO asking whether to fund it again. Which number answers that?
A. Neither of the ones the team has. Report conversion rate and average revenue per user for sessions that saw the assistant, against the pre-launch baseline.
Why? ROUGE, a human-preference win rate and accuracy on a benchmark dataset say the model is good. Token throughput and p99 latency say the system is healthy. A funding decision asks what the business got back, so the answer has to be business value: conversion rate for how often shoppers buy, average revenue per user for how much they spend, and both set against what the feature costs to run, which is ROI. Customer lifetime value is on the same list and moves too slowly to settle a six-week question; state the assumption behind it and revisit at twelve months. And the comparison is what makes any of it evidence. Choosing the metric is only half the work; without a baseline captured before launch, or a holdout group that never sees the assistant, a conversion figure is a number with nothing to be better than.