The situation
Redgum Mobile has about 1.4 million customers and a support centre of two hundred and forty agents. Three months ago it launched an assistant in the customer app. A question goes in, the assistant searches the help centre and the customer’s own account state, writes an answer, and offers a handover to a human if the customer asks for one.
The numbers presented at the quarterly review look good. The team ran a scoring job against a set of two hundred reference answers written by the support content team and the assistant returned a ROUGE-L of 0.44, comfortably past the 0.35 bar set before launch. Deflection, meaning the share of assistant conversations that never reached a human, is 38%. Inference spend is under budget.
The head of support wants it turned off. Her agents say the conversations that do reach them arrive worse than the ones that used to: the customer has already repeated themselves four times, is annoyed, and often has a wrong answer in their head that has to be unpicked before anything else can happen. Average handle time on escalated contacts is up seven minutes. Nobody in the room can say which of these two accounts of the feature is right, because the two sides are holding different measurements and neither has the one that settles it.
What actually matters
There are two layers being confused, and separating them is most of the work. A model metric scores one output against something: a reference answer, a retrieved passage, a rubric. ROUGE, BLEU and BERTScore all live here, and so does a judge model’s rating. These say whether the model produced a good piece of text for a given input. They are cheap, they run in minutes, and they are how you choose a model in the first place. A business metric asks something the model cannot answer on its own: is the organisation better off for running this. It comes from product events, finance and the people doing the work, and it moves in weeks.
A high ROUGE score sitting on top of a failing feature is not a contradiction, because ROUGE is measuring word overlap with a reference answer to a question somebody already decided was the right question. It says nothing about whether the assistant answered what the customer actually asked, whether the customer could act on the answer, or whether the conversation ended anywhere useful. Redgum’s evaluation set is two hundred well-formed questions with tidy reference answers. The app receives half-typed questions about a bill the customer is holding in their other hand. The model is scoring well on a set that does not resemble the traffic.
The second confusion is measuring the model when what shipped is an application. What a customer touches is a retrieval step, a prompt, a model call, a handover rule and an app screen. Applications built on a foundation model come in three shapes, and each one is evaluated differently: RAG, agents and workflows. Each of those has more than one place to fail, and a single quality score averaged over the whole thing tells you a number went down without telling you which piece moved.
Third, a metric that becomes a target starts being optimised, including in ways nobody wanted. Deflection rate is the clearest case here. An assistant that answers badly and makes the handover link hard to find will show a rising deflection rate. Deflection counts conversations that did not reach a human, and it cannot tell a solved problem from an abandoned one. That is exactly the pattern Redgum’s agents are describing, and the metric they are being shown is the one that hides it.
What we’ll filter on
- Layer: does this number grade a single output, or does it say whether the feature is worth running?
- Failure attribution: when it drops, does it tell you which part of the application moved, or only that something did?
- Source: can it be collected from product events, logs and billing that already exist, or does it need a survey, a baseline or a finance exercise?
- Cadence: does it refresh daily, weekly or quarterly, and does that match how often somebody has to make a decision on it?
- Gaming resistance: if this number became a team target, what would improve without the customer being better off?
The landscape
There are two layers to walk through. The first is how you evaluate an application rather than a model. The second is the set of business objective alignment metrics that say whether the application is meeting the business objectives it was funded against.
Measuring a RAG application
A retrieval augmented generation application has two halves, and they fail for different reasons, so measure them separately. Retrieval is the first half: for a question, did the passage that contains the answer come back in the retrieved set? Score that over a labelled question set where somebody has recorded which document holds each answer. The number is a hit rate. Generation is the second half: given the passages that did come back, is every claim in the answer supported by something in the retrieved text? A human sample or a judge model with a written rubric gives you that.
Splitting them turns one useless signal into a diagnosis. A wrong answer whose retrieval hit rate is low is a retrieval problem, and prompt changes will not touch it: the fixes are chunking, the embedding model, how many passages you fetch, and metadata filters. A wrong answer whose retrieval was correct is a generation problem, and the fixes are the prompt, the model, and how much of the retrieved text you pass. Teams that measure one blended quality score spend weeks rewriting prompts against a retrieval fault. Requiring the answer to cite the passage it used, as in a citations-required retrieval build, makes the second half checkable by anyone reading the output rather than only by an evaluation job.
Measuring agents
An agent chooses its own steps, so there is no fixed sequence to compare against. What it can be measured on is whether the task completed: given a request, did the agent reach the state the user wanted, and can that be checked from the outside? Booking moved, ticket raised, refund issued. Alongside that sit the cost measures: how many turns and how many tool calls it took to get there. An agent that completes the same job in nine steps this week and four last week has regressed, even though completion is flat. And side effects, because an agent that writes to real systems can complete the wrong task successfully: measure the rate of actions taken that had to be reversed. An agent is only worth the arrangement when the steps genuinely cannot be written down in advance, and its measurement follows from that: you are scoring outcomes and their cost, not adherence to a script.
Measuring workflows
A workflow is a fixed sequence with a model call somewhere in it, so it can be measured the way any pipeline is measured: throughput, meaning items processed per hour or per day; error rate, meaning runs that failed outright plus runs that produced a wrong result and were caught downstream; and human intervention rate, meaning how often a person had to step in to correct or complete something. That last one is usually the most informative number a workflow produces, because it converts directly into the labour the workflow was supposed to remove.
Business objective alignment metrics
These are the numbers the review actually needs. Task completion rate is the share of interactions that reached what the user came for, which needs a definition somebody writes down: for Redgum, a conversation where the customer’s question was answered and they did not ask the same thing again within twenty-four hours or open a contact about it. User satisfaction is collected in the product rather than inferred, as a thumbs up and down on each answer or a short CSAT prompt at the end of a conversation. Cost per interaction is total spend, meaning inference plus retrieval plus the surrounding infrastructure, divided by conversations, and it is the number that tells you whether a longer prompt has quietly changed the bill.
Around those sit three more. Productivity measures whether the work gets done faster: handle time, contacts resolved per agent per shift, output per person. User engagement measures whether people come back and stay in the feature: repeat usage, how deep a session goes, and the abandonment rate part-way through. Return on investment (ROI) weighs the whole thing against everything it cost, including build, run and the people supporting it. It is the only one of these that needs a pre-launch baseline to mean anything, which is a measurement you have to set up before you launch.
One term ties the two layers together. Task engineering is shaping the job the model is given so it matches the business outcome you are measuring. Redgum asked its model to write a helpful answer to a question. What the business wanted was for a customer to leave able to act. Those are different jobs, and the second one implies things the first does not: say when you do not know, offer the handover early rather than after four attempts, and end with the specific next step. Redefine the task and the metric becomes measurable at the same time, because a task defined as an outcome has an outcome you can count.
Evaluation
Side by side
| Measure | What it answers | Where the number comes from | Available weekly | Safe as a target on its own |
|---|---|---|---|---|
| ROUGE / BERTScore | Does the output match a reference answer | Fixed evaluation set | ✓ | ✗ |
| Retrieval hit rate | Did the right passage come back | Labelled question set, retrieval logs | ✓ | ✗ |
| Faithfulness | Does the answer follow from what came back | Human sample or a judge model | ✓ | ✗ |
| Agent task completion | Did the requested job actually finish | Agent traces, end state per run | ✓ | ✗ |
| Steps and tool calls per task | What one completed job cost in work | Agent traces | ✓ | ✓ |
| Workflow error and intervention rate | How often it failed or needed a person | Pipeline logs, review queue | ✓ | ✓ |
| Task completion rate | Did the user get what they came for | Product events plus a written definition | ✓ | ✗ |
| User satisfaction | Would the user call this good | Thumbs or CSAT collected in the product | ✓ | ✗ |
| Cost per interaction | What one conversation costs to run | Billing divided by conversation count | ✓ | ✗ |
| User engagement | Do people come back and finish | Repeat usage, session depth, abandonment | ✓ | ✗ |
| Productivity | Is the work getting done faster | Handle time, resolutions per agent | ✗ | ✗ |
| Return on investment (ROI) | Did it return more than it cost | Finance, against a pre-launch baseline | ✗ | ✓ |
The last two columns carry the argument. Everything available weekly can be watched on a dashboard and acted on inside a sprint; productivity and ROI move on an operational and a financial clock, which is why they belong in a quarterly review and not on a wall screen. The final column is a warning rather than a ranking: almost nothing in this table survives being made a team target by itself. Task completion improves by loosening the definition of completion. Satisfaction improves by asking only the customers who did not escalate. Cost per interaction improves by cutting retrieved passages until answers get worse. Each one needs the metric it trades against sitting beside it, which is why the working set is a small group rather than a single headline number.
Where a wrong answer came from
The solution
Start by writing down what a completed task is, because every number after this depends on it. At Redgum that becomes: the customer’s question was answered, they did not ask it again within twenty-four hours, and they did not open a support contact on the same topic within seven days. That definition is a product decision, not an engineering one, so the head of support signs it off along with the platform team. Instrument it from events the app already emits plus a join against the contact system.
Add an in-product satisfaction signal next, because it is the cheapest honest number available. A thumbs up and down under every answer, with an optional one-line reason on a thumbs down, gives a rate within a fortnight and a stream of failure descriptions immediately. Sample the thumbs-down conversations weekly and read twenty of them; that habit finds more than any dashboard.
Then compute cost per interaction properly: model inference, retrieval, and the compute around it, divided by conversations for the same period. Redgum’s version came out at 11 cents against a fully loaded 4.80 dollars for an agent-handled contact. That ratio is what makes the feature arguable at all. It also stops being flattering the moment poor answers force two follow-up conversations per resolution.
Split the RAG failures before touching a prompt. Build a labelled set of three hundred real customer questions with the help-centre article that answers each one recorded against it, run retrieval over that set, and get a hit rate. Then take a sample of answers where retrieval was correct and score faithfulness. Only after those two numbers exist does prompt work start, and it starts on whichever half the numbers point at.
Retire deflection as a headline. Keep collecting it, because it is useful next to other things, and stop reporting it alone: an assistant that ends conversations badly raises deflection, and the same conversations show up later as longer, angrier contacts. Report it beside task completion and satisfaction so the trade is visible on one screen.
Set a cadence and match it to how fast each number moves. Weekly: retrieval hit rate, faithfulness sample, task completion, satisfaction, cost per interaction. Monthly: productivity and user engagement, which need enough volume to be readable. Quarterly: return on investment (ROI) against the pre-launch baseline, in the same review that decides whether the budget continues.
Worked example
Redgum runs the labelled set and gets a retrieval hit rate of 61%. Four in ten questions never had the right article in front of the model at all, which no amount of prompt work would have fixed. Reading the misses shows why: the help centre is chunked by article, and the long billing articles are being truncated so the part covering pro-rata charges on a mid-month plan change never makes it into a chunk of its own. Splitting those articles by section takes the hit rate to 84%.
Faithfulness on the correct-retrieval sample comes back at 91%, which is respectable and not where the complaints are coming from. So the ROUGE score was not lying. It was answering a question nobody had asked.
Task completion, once instrumented, lands at 52%, against a deflection rate of 38% that had been read as success. The two together say what the agents had been saying: roughly one conversation in seven ends without a human and without the customer’s problem solved. Satisfaction confirms it, at 63% thumbs up overall but 24% on the billing topics that the retrieval fix has just addressed.
A task engineering pass follows the chunking change. The assistant is now instructed to offer the handover as soon as it cannot support an answer from a retrieved passage. Six weeks later, task completion is 71%, deflection has fallen to 34%, satisfaction is 78%, and handle time on escalated contacts is back to within a minute of its pre-launch level. Deflection went down and the feature got better. The old dashboard could not have shown that.
What’s worth remembering
- Model metrics such as ROUGE and BERTScore grade one output against a reference and are how a model is chosen; business objective alignment metrics say whether the feature is worth running, and a healthy score in the first set is compatible with a failing feature.
- Evaluate an application built on a foundation model by measuring its parts rather than the whole: a RAG application splits into retrieval hit rate and faithfulness, an agent into task completion plus steps, tool calls and reversed side effects, and a workflow into throughput, error rate and human intervention rate.
- Diagnose a wrong RAG answer by asking whether the right passage came back before asking whether the answer followed from it, because the first is a chunking and embedding problem and the second is a prompt and model problem.
- The named business objective alignment metrics are task completion rate, user satisfaction and cost per interaction, sitting alongside productivity, user engagement and return on investment (ROI), and only the last needs a baseline captured before launch.
- Deflection rate counts conversations that avoided a human and cannot tell a solved problem from an abandoned one, so report it beside task completion rate and satisfaction or not at all.
- Task engineering is shaping the job you give the model so it matches the outcome you are measuring, and redefining “write a helpful answer” as “leave the customer able to act” changes both the prompt and the metric at once.