The situation
A Perth company services industrial pumps and compressors on mine sites across the Pilbara and the Goldfields. Eleven service coordinators handle roughly six hundred callouts a week. Each arrives as free text from a site supervisor: what stopped, what it sounded like, what they already tried. A coordinator reads it, picks a job category, sets a priority, and decides which parts go on the truck. Name the wrong parts and a technician travels out, finds the part missing, and the job takes a second trip.
Two weeks ago a developer spent a week on a demo. It sends a callout to a Claude model in ap-southeast-2 through the Converse API and returns the three fields as JSON. She ran thirty callouts through it and twenty-seven came back looking right. The demo went to the executive team on Tuesday and went well. There is now a proposal on the table for AUD$310,000: two engineers for a quarter, plus the integration work into the job-management system.
What nobody can say is what the demo established. Thirty callouts, chosen by the person who wanted it to work, scored by eye, with no record of what a coordinator picked for the same thirty. The wrong-parts rate today is not measured either, so there is no number the model has to beat.
What actually matters
A demo shows that a shape is possible. A go or no-go needs a result that could have come out the other way, which means a threshold written down before the run and a sample the team did not choose. So the design question for the next week is not how to make the output better. It is what would have to be true for this to be worth AUD$310,000, and what result would show it is not. If no outcome would have produced a no, the week produces a slideshow.
There are two ways to be wrong and they cost differently. A false go funds a quarter of engineering and discovers at pilot that accuracy on the awkward half of the traffic never reaches the bar. A false no-go kills a workable idea because the demo ran on a curated sample and a prompt nobody iterated. The first shows up as money and the second never shows up at all, which is why it is more common. A structured week is where a wrong answer does the least damage, and that only holds if the week produces evidence rather than polish.
The measurements also have to survive the trip to production, and most demo measurements do not. Latency taken over a home connection against a default Region says nothing about the Region the job system runs in. A token cost worked out from a guessed average misses the retrieved context, the system prompt, and the fact that tokenisation is model-specific. Throughput inferred from a handful of sequential calls says nothing about Monday morning, because the limit that bites is a per-model, per-Region token quota the demo’s load never approached. Instrument the run with the mechanisms production will use and the numbers carry forward instead of being redone.
Last, somebody has to own the threshold. Whoever can veto the funding agrees it before the run, or it becomes a negotiation afterwards about whether 84% is close enough to a figure nobody wrote down. Tie it to something the business already counts. Second trips, not F1.
What we’ll filter on
- A falsifiable threshold. A number, agreed before the run, that the result is able to fail.
- A sample nobody chose. Drawn at random from real traffic and stratified across the shapes production sees, with the awkward strata present in proportion.
- Measured, not estimated. Token counts, latency and first-token time read from actual responses and metrics rather than derived from an assumed average.
- Attribution per candidate. Each model or prompt under test separable in the logs and on the bill, with one variable moving at a time.
- A line to a business number. The quality measure converts into something the organisation already counts, with the conversion stated rather than implied.
- Carries forward. The dataset, the harness and the metrics survive into the production build.
The landscape
Every instrument here is in Amazon Bedrock or CloudWatch. What separates them is the grain they measure at, and whether the number they produce still means something once the feature is real.
The console playground and a hand-driven notebook
This is what the demo was, and it is the right tool for the first afternoon: it answers whether a prompt shape exists at all. Past that it produces nothing defensible. The inputs are whatever was to hand, the scoring is a person nodding, and the only token accounting is what the caching metrics pop-up shows for the call in front of you. Nothing about the run is reproducible, so nothing about it can be compared against a second candidate a week later.
A scripted harness on the Converse API
Converse and ConverseStream return a usage object on every call carrying inputTokens, outputTokens and totalTokens, plus cacheReadInputTokens and cacheWriteInputTokens when prompt caching is in play, and a metrics.latencyMs alongside it. With caching active, inputTokens counts only the non-cached tokens, so the total input for a request is the sum of all three fields. Getting that wrong understates the input side by exactly the portion that was cached.
Each call also accepts requestMetadata, up to 16 key-value pairs of 256 characters each, which is recorded in the model invocation logs under a top-level requestMetadata field. Tag every call with the run identifier, the candidate under test and the stratum the input came from, and the logs partition by all three without any further plumbing. Two conditions apply: model invocation logging is disabled by default and has to be enabled in the Region where the calls are made, and the metadata reaches the logs only, never Cost Explorer. For the input side of a corpus there is also CountTokens, which incurs no charge and returns the count that would be billed for the same input, so a corpus can be priced before any inference runs.
Amazon Bedrock evaluations
The managed alternative to scoring by eye. Bedrock runs programmatic evaluation jobs, jobs scored by a second model acting as judge, and jobs scored by human workers. The dataset is JSONL in Amazon S3, up to 1,000 prompts per job, each line carrying a prompt and optionally a referenceResponse and a category. Built-in judge metrics include Builtin.Correctness, Builtin.Completeness, Builtin.Faithfulness, Builtin.FollowingInstructions and Builtin.Relevance, and a ground-truth referenceResponse feeds only the first two. The category field is what produces a score per stratum rather than one aggregate, which is how a candidate that averages well but collapses on one input shape gets caught. An inference profile can be named as the model to evaluate, so the same resource that meters a candidate can be the one that scores it. This is the same machinery a production evaluation runs on, which is what makes the dataset worth building properly the first time.
CloudWatch runtime metrics and the quota arithmetic
The AWS/Bedrock namespace publishes Invocations, InvocationLatency, InvocationThrottles, InputTokenCount, OutputTokenCount, CacheReadInputTokenCount and CacheWriteInputTokenCount, dimensioned on ModelId. TimeToFirstToken is published only for ConverseStream and InvokeModelWithResponseStream, so a harness that calls the non-streaming operation produces no first-token figure at all, however much the perceived latency of the real feature depends on it. InvocationLatency covers request to last token, which rises both when the service slows and when answers get longer; output tokens per second separates the two, as OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000 in a metric math expression.
Throughput, though, is arithmetic rather than observation, because a PoC’s load never reaches the ceiling. On-demand inference on the bedrock-runtime endpoint is governed per model, per Region by an “On-demand InvokeModel tokens per minute” quota that counts input and output together, with a daily quota defaulting to that figure times 1,440. Requests-per-minute quotas apply to some models and not others. Output tokens convert through a model-specific burndown rate, so on a 5x model a 240-token answer draws 1,200 tokens from the quota. max_tokens is deducted in full at the start of the request and the unused remainder returned at the end, so a generous ceiling caps concurrency even when the answers are short. Cache reads are not counted. There is an EstimatedTPMQuotaUsage metric, and the documentation is explicit that it should not be the sole basis for capacity planning, since throttling runs on that up-front reservation rather than on the estimate.
Cost attribution: application inference profiles and request metadata
Neither mechanism yields per-request dollars, and which one is reached for decides what the finance conversation can be about. An application inference profile is a per-model resource whose ARN replaces the model id in the call; its cost allocation tags flow to Cost Explorer and the Cost and Usage Report once activated in the billing console, with up to 24 hours before they appear and no retroactive effect on spend already incurred. The finest grain is per usage type per day. Request metadata is per call, reaches the invocation logs alone, and turns into money only by multiplying the logged token counts by a rate card kept in USD$ on the Amazon Bedrock pricing page. One profile per candidate splits the bill; request metadata supplies the per-prompt detail underneath it. Attribution across many teams works the same way, at a larger scale.
Evaluation
Side by side
| Instrument | Falsifiable score | Representative at volume | Per-request token detail | Per-candidate on the bill | Latency and first token | Carries into production |
|---|---|---|---|---|---|---|
| Console playground, hand-driven | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
Scripted Converse harness with requestMetadata |
✗ | ✓ | ✓ | ✗ | ✓ | ✓ |
Bedrock evaluation job, JSONL with category |
✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| CloudWatch runtime metrics | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ |
| Application inference profile per candidate | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ |
No row carries the whole load, which is why a serious proof of concept runs three of them together: the harness generates and meters, the evaluation job scores against a threshold, and the profile splits the bill by candidate. The crosses in the first row are the honest reading of the demo that has already been given to the executive team. The two crosses against the evaluation job are not shortcomings; a scoring job reports quality, and latency and token detail come from the call path beside it.
What each question needs
The solution
Start with the criterion, in writing, before a single call. It has three parts and each has a number: a quality bar with a floor per input category, a cost and latency ceiling, and a business return stated as an arithmetic conversion from the quality bar. Send it to whoever can veto the funding and get a yes on the threshold, not on the idea. A criterion drafted after the results arrive is a negotiation.
Build the sample next, and draw it rather than pick it. Pull a stratified random sample from real callout history, sized to the evaluation job’s 1,000-prompt limit, with strata that reflect the shapes the queue actually contains: well-written supervisor reports, transcribed phone calls, callouts from sites with unusual equipment, and the ones that arrive as four words and a photo reference. Put the stratum in the category field so the job reports a score per stratum. Take the ground truth from what the technician used on the job, not from what the coordinator originally guessed, and hold back a slice that no prompt iteration ever sees. In the same pass, mine the job history for the baseline: the coordinators’ own match rate and the current second-trip rate. Without those two numbers the model’s score has nothing to be measured against.
Then fix everything except the one thing under test. One prompt, one parsing path, one Region, one max_tokens setting; vary the candidate model. Give each candidate its own application inference profile with a tag naming it, and pass requestMetadata on every call carrying the run identifier, the candidate and the stratum. The bill then splits by candidate in Cost Explorer a day later, and the invocation logs split by all three immediately, which is what lets a surprising cost number be traced back to the stratum that produced it.
Measure from the responses rather than from a rate page. Record inputTokens, outputTokens, cacheReadInputTokens and cacheWriteInputTokens per call, remembering that with caching on the first of those excludes the cached portion, so the total input is the sum. The common estimating error is not using an average instead of a distribution; cost is linear in tokens, so the average is fine for the mean bill. The error is that the average is guessed. Somebody counts the words in a sample prompt and forgets the system prompt, the retrieved context, the JSON schema in the instructions, and that tokenisation differs per model. CountTokens removes that guess for the input side at no charge, returning the count that would be billed. The output side has to be generated to be known, which is one of the reasons the run exists.
Where the distribution genuinely matters is latency and throughput. Set the p95, not the mean, against the ceiling. If the production feature streams, call ConverseStream in the harness, because TimeToFirstToken is published for the streaming operations alone, and a non-streaming run leaves that number permanently unmeasured. For throughput, take the measured per-request tokens, apply the model’s burndown rate to the output side, add the max_tokens reservation, multiply by the peak-minute arrival rate, and compare against the “On-demand InvokeModel tokens per minute” quota for that model in that Region. Do this as arithmetic. The run’s own traffic is too small to reveal the ceiling, and EstimatedTPMQuotaUsage is explicitly not a capacity-planning signal.
Two things to decide before the run rather than during it. Pick the Region on production’s constraint, since a data-residency rule that keeps processing in Australia rules out the apac. geographic inference profile, whose whole geography spans several countries, and leaves single-Region on-demand in ap-southeast-2 with whatever quota that carries. And decide what a no looks like, so that writing one is a result rather than a failure. The go or no-go document states the threshold, the measured number, the gap, the conversion to money and, above all, the link in the value chain the run could not test. That last line is what separates evidence from advocacy. What the document does not cover is everything that turns a proven idea into a running feature, which is a separate set of dimensions entirely and a separate quarter of work.
Worked example
The criterion, signed on the Monday: the drafted parts list matches the technician’s final pick on at least 88% of sampled callouts, with no category below 80%; wrong parts causing a return on no more than 4%; p95 end to end under 6 seconds; and second trips avoided over a year worth more than AUD$310,000 on a conversion stated in the document. The baseline from twelve months of job history is a coordinator match rate of 84% and about 22 second trips a week at an average AUD$1,250 each.
The sample is 320 callouts, stratified into five categories, ground truth taken from parts actually consumed. Two candidates run against it, each behind its own tagged application inference profile, each call carrying requestMetadata with the run, the candidate and the category.
The measured token distribution surprises nobody who has done this before and everybody who has not. The mean is 1,850 input tokens and 240 output, against the developer’s original estimate of “about a thousand”; the missing portion was the system prompt, the parts catalogue excerpt and the JSON schema. The p95 is 4,900 input and 610 output. At the candidate’s current per-token rates that comes to AUD$0.041 a callout, roughly AUD$25 a week at six hundred callouts, which is noise against the funding ask and settles the cost question in one line.
Throughput takes the arithmetic. About 14 callouts arrive in the busiest Monday minute. On a model with a 5x output burndown, each request draws 1,850 plus 240 times 5, or 3,050 quota tokens once settled, but the reservation at request start is the input plus max_tokens. Left at 4,096 that reservation is 5,946 a request, so the peak minute reserves roughly 83,000 tokens rather than the 43,000 it ends up consuming. Dropping max_tokens to 800, comfortably above the p95 output of 610, brings the peak reservation to about 37,000 and moves the workload from uncomfortably close to the quota to plainly inside it. That one parameter, set from measured data, is the difference between a throttling problem at launch and no problem at all.
On quality the larger candidate scores 91.4% overall, above the 88% bar, with its weakest category at 86% and wrong parts at 3.1%. The cheaper candidate scores 88.6% overall, which clears the pooled bar, and 71% on transcribed phone calls, which is 9% of traffic and below the 80% floor. A single aggregate would have passed it. The per-category breakdown, which exists only because the category field was populated, rules it out.
The value conversion is written down and its weak link named. A 7.4 point lift in match rate over the coordinators’ 84%, applied to the parts-bearing share of the queue, works out to roughly eight fewer second trips a week, or about AUD$520,000 a year against a AUD$310,000 one-off. The conversion assumes mismatches and second trips track one for one, which the run did not test and could not. So the recommendation is a go, funded in two stages, with the pilot instrumented to measure the actual second-trip rate against the model’s suggestions before the second stage is released. A demo that impressed people on Tuesday has become a decision somebody can defend, and a dataset and harness the production build inherits.
What’s worth remembering
- A proof of concept that no result could have failed has answered nothing; write the threshold, with a floor per input category, and get it signed before the first call.
- Draw the sample at random and stratify it, take ground truth from what actually happened, and measure the human baseline, because the model’s score means nothing without something to beat.
- Read token counts from the
usageobject on every response rather than estimating them, addcacheReadInputTokensandcacheWriteInputTokensto get the true input, and use the freeCountTokenscall to price an input corpus in advance. TimeToFirstTokenis published only for the streaming operations, so a non-streaming harness cannot produce a first-token figure at all; measure it in the Region production will run in.- Throughput is arithmetic, not observation: apply the model’s output burndown rate, add the
max_tokensreservation taken at request start, and compare the peak-minute draw against the per-model, per-Region tokens-per-minute quota. - Give each candidate its own tagged application inference profile for the bill and
requestMetadataon every call for the logs, then state the conversion from the quality number to money and name the link in that chain the run could not test.