Exam-style
A utility company's self-service assistant is up for its annual renewal. It scores ROUGE-L 0.51 against a set of reference answers written by the support content team, and containment, the share of conversations that never reach a human, has risen 30% since launch. Two other things have also moved: the share of customers who contact support again within 24 hours is up, and the agents taking the handovers report that the customer arrives further from a resolution than before. Which measurement answers whether the assistant is meeting the business objective it was funded for?
Reveal the answer
C. Task completion rate, the share of conversations that ended with the customer's problem resolved, read alongside user satisfaction
The guide gives three business objective alignment metrics for AI applications: task completion rate, user satisfaction and cost per interaction. Each catches something the others miss. Task completion rate is the share of conversations in which the customer’s problem was actually resolved, and it is the one this scenario is missing. Containment counts conversations that ended, not problems that were solved, so a customer who gave up, or who left with a wrong answer and rang back the next morning, is counted as a success by containment and as a failure by task completion rate. That gap is exactly the shape of the evidence here: repeat contacts within 24 hours and handovers arriving further from a resolution are both what an ended-but-unsolved conversation looks like from the other side. User satisfaction, gathered from a rating at the end of the conversation or a short survey afterwards, catches the case where the assistant technically resolved the request and the customer hated the experience of getting there, and it also stands in for user engagement, since people come back to a channel they like. Cost per interaction is the third of the three and belongs on the renewal paper, because a channel that resolves the job for a fraction of an agent contact is worth funding. It answers what each conversation costs, not whether the job got done, so on its own it argues for whichever channel is cheapest to fail in. ROUGE-L is a model-output metric. It says the wording of the assistant’s answers resembles wording the support team approved, which is a useful check on the model and no evidence at all about outcomes; a fluent, well-phrased, wrong answer scores well. Average tokens per response measures the size of the output and belongs on an inference bill, not a renewal case. The habit worth building: name the business objective first, in the language the business used when it funded the work, whether that was agent productivity, user engagement or cost, then pick the metric that moves when that objective is met.
Q. ROUGE-L 0.51, containment up 30%, repeat contacts within a day rising and handovers arriving worse. Which number decides the renewal?
A. Task completion rate, read alongside user satisfaction. Containment counts conversations that ended; task completion rate counts problems that were solved.
Why? Task completion rate, user satisfaction and cost per interaction are the three business objective alignment metrics, and each one catches a different failure. Task completion rate catches the customer who gave up or left with a wrong answer, which containment scores as a win. User satisfaction catches the resolved request that felt awful to get, and it tracks user engagement, because people return to a channel they like. Cost per interaction says what each conversation costs, which the renewal paper needs and which says nothing about whether the job got done. ROUGE grades wording against a reference; a fluent wrong answer scores well. Start from the business objectives as they were written when the work was funded, whether that was agent productivity, engagement or cost, then pick the number that moves when those objectives are met.