Exam-style
A utility company's self-service assistant is up for annual renewal. It scores ROUGE-L 0.51 against reference answers written by the support content team, and containment, the share of conversations that never reach a human, has risen 30% since launch. Two other numbers have moved too: repeat contacts within 24 hours are up, and agents taking the handovers report customers arriving further from a resolution. Which measurement shows whether the assistant meets the business objective it was funded for?
Reveal the answer
C. Task completion rate, the share of conversations that ended with the customer's problem resolved, read alongside user satisfaction
Task completion rate, user satisfaction and cost per interaction are the three business objective alignment metrics for AI applications, and task completion rate is the one missing here. Containment counts conversations that ended, not problems that were resolved. A customer who gave up, or who took a wrong answer and rang back the next morning, counts as a success under containment and a failure under task completion rate. That is what repeat contacts within 24 hours and worse handovers describe. User satisfaction, from an end-of-conversation rating or a short survey, covers the request that was resolved badly, and it tracks user engagement. Cost per interaction belongs on the renewal paper, but it measures what a conversation costs rather than whether the job got done. ROUGE-L scores word overlap against a reference, from 0 to 1, so an answer reusing approved wording scores well whether or not it resolved anything. Average tokens per response measures output length. Name the business objective first, in the words used when it was funded, whether productivity, user engagement or task engineering, then pick the metric that moves when it is met.
Q. ROUGE-L 0.51, containment up 30%, repeat contacts within a day rising and handovers arriving worse. Which number should settle the renewal?
A. Task completion rate, read alongside user satisfaction. Containment counts conversations that ended; task completion rate counts problems that were resolved.
Why? Task completion rate, user satisfaction and cost per interaction are the three business objective alignment metrics, and each one catches a different failure. Task completion rate catches the customer who gave up or took a wrong answer, which containment counts as a success. User satisfaction covers the request that was resolved badly, and it tracks user engagement, because people return to a channel they like. Cost per interaction measures what a conversation costs. The renewal paper needs that number, and it is silent on whether the job got done. ROUGE scores word overlap against a reference, so an answer that reuses the approved phrasing scores well whether or not the customer’s problem was resolved. Start from the business objectives as they were written when the work was funded, then pick the number that moves when those objectives are met.