Exam-style
An ecommerce team shipped a product-recommendation assistant on Amazon Bedrock six weeks ago. It scores 0.42 on ROUGE for its summarised product blurbs and a 68% human-preference win rate against the previous rules-based widget. The CFO has asked the team to justify next year's budget for it. Which number answers that?
Reveal the answer
C. Report conversion rate and average revenue per user for sessions that saw the assistant, measured against the baseline captured before launch
Sort the numbers by who is asking. ROUGE, the human-preference win rate and accuracy on a benchmark dataset are model-quality metrics: they describe the output. Token throughput and p99 latency are system metrics: they describe the running software. All five describe the assistant. A budget decision asks what the business got back for what it spent. Conversion rate, the share of sessions ending in an order, and average revenue per user, revenue over active shoppers for the period, both move within weeks. Set them against the cost of running the assistant and you have ROI, return on investment: value produced, less what it cost, over what it cost. Customer lifetime value sits on the same list but moves more slowly, since it depends on how long shoppers stay, and six weeks cannot show a change in retention; state it as an assumption and revisit after a year. None of this works without a baseline. A conversion rate of 3.1% after launch proves nothing alone, because the figure needs a before, captured while the old experience was still running. Where none exists, a holdout group that never sees the assistant supplies one from here, after a few more weeks.
Q. Six weeks of a live recommendation assistant, a ROUGE score of 0.42, a 68% human-preference win rate, and a CFO asking whether to fund it again. Which number answers that?
A. Neither of the ones the team has. Report conversion rate and average revenue per user for sessions that saw the assistant, against the pre-launch baseline.
Why? ROUGE, a human-preference win rate and accuracy on a benchmark dataset describe the model’s output. Token throughput and p99 latency describe the running system. A funding decision turns on what the business got back, so the answer has to be business value: conversion rate for how often shoppers buy, average revenue per user for how much they spend, and both set against what the feature costs to run, which is ROI. Customer lifetime value is on the same list and moves too slowly to settle a six-week question; state the assumption behind it and revisit at twelve months. And the comparison is what makes any of it evidence. Choosing the metric is only half the work; without a baseline captured before launch, or a holdout group that never sees the assistant, a conversion figure is a number with nothing to be better than.