Exam-style
An ecommerce team shipped a product-recommendation assistant on Amazon Bedrock six weeks ago. It scores 0.42 on ROUGE for its summarised product blurbs and a 68% human-preference win rate against the previous rules-based widget. The CFO has asked the team to justify next year's budget for it. Which number answers that?
Reveal the answer
C. Report conversion rate and average revenue per user across the whole site since launch, measured against the baseline captured before launch
Sort the numbers by who is asking. ROUGE, the human-preference win rate and accuracy on a benchmark dataset are model-quality metrics: they describe the output. Token throughput and p99 latency are system metrics: they describe the running software. A budget decision asks what the business got back for what it spent. Conversion rate, the share of sessions ending in an order, and average revenue per user, revenue over active shoppers for the period, both move within weeks. Set them against the cost of running the assistant and you have ROI, return on investment: value produced, less what it cost, over what it cost. Customer lifetime value moves more slowly: it depends on how long shoppers stay, and six weeks cannot show a change in retention; state it as an assumption and revisit after a year. A conversion rate of 3.1% after launch proves nothing alone: the figure needs a before, captured while the old experience was still running, and it needs the whole site, since the sessions that opened a conversation were shoppers already closer to buying. Where no baseline exists, a holdout group that never sees the assistant supplies one from here, after a few more weeks.
Q. Six weeks of a live recommendation assistant, a ROUGE score of 0.42, a 68% human-preference win rate, and a CFO asking whether to fund it again. Which number answers that?
A. Neither of the ones the team has. Report conversion rate and average revenue per user across the whole site since launch, against the pre-launch baseline.
Why? ROUGE, a human-preference win rate and accuracy on a benchmark dataset describe the model’s output. Token throughput and p99 latency describe the running system. A funding decision turns on what the business got back, so the answer has to be business value: conversion rate for how often shoppers buy, average revenue per user for how much they spend, and both set against what the feature costs to run, which is ROI. Customer lifetime value moves too slowly to settle a six-week question; state the assumption and revisit at twelve months. The comparison is what makes any of it evidence. Choosing the metric is only half the work; without a baseline captured before launch, or a holdout group that never sees the assistant, a conversion figure is a number with nothing to be better than. Narrow it to the sessions that opened a conversation and the lift is mostly shoppers already buying.