The situation
A knowledge-assistant team has shipped a Bedrock-backed feature that answers staff questions over an internal document set. A request retrieves passages from a vector store, stuffs them into a prompt with the conversation history, calls the model, and sometimes makes a tool call to look up a live figure before answering. It works, and users say it feels sluggish. The complaint is vague: sometimes it is fine, sometimes it hangs for what feels like an age before anything appears.
The team has one number to go on, an average end-to-end time of about 4.2 seconds, and they have been arguing about it from taste. One camp wants a bigger vector index with more results per query, sure the retrieval is thin. Another wants to switch to a larger, smarter model, sure the answers are the bottleneck. A third has been told streaming will fix everything and wants to ship that first. Nobody has measured where the 4.2 seconds actually goes, and the average is hiding the thing users are reacting to, which is the occasional request that takes twelve seconds while the rest take two.
The real task is not choosing a lever. It is finding out which stage owns the time, at the tail as well as the middle, and then choosing the lever that stage responds to.
What actually matters
An end-to-end generative-AI request is a pipeline of stages that each add wall-clock time, and they do not add up the way people assume. The stages are retrieval (embedding the query, searching the index, fetching passages), any pre-processing and prompt assembly, the model call itself, any tool calls the model triggers mid-generation, and the network hops between all of them. The model call is not one number either. It splits into time-to-first-token, the wait before the first output token appears, and per-token generation time for everything after, and those two respond to completely different levers.
Time-to-first-token is dominated by how much the model has to read and how contended the endpoint is. A long prompt, a big pile of retrieved context, and a large model all push it up, because the model processes the whole input before it emits anything. Per-token generation time is set mostly by model size and how many tokens you ask it to produce; a verbose answer costs linearly in tokens whether or not the user reads them all. This split is the single most useful thing to hold in your head, because it explains why streaming helps a slow feel without touching total time, and why cutting output length helps total time without touching the first-token wait much.
The tail is what users feel, and an average erases it. If p50 is two seconds and p99 is twelve, the average might read a comfortable-looking four, and the twelve-second requests are the ones generating the complaints and the abandoned sessions. Latency has to be measured per stage as percentiles, at least p50 and p99, so you can see both the typical path and the bad one, and so you can tell whether the tail lives in retrieval, in the model, in a tool call that occasionally times out, or in a cold dependency. Attacking the mean optimises the wrong request.
The trade under all of it is latency against quality. The fastest single change is almost always a smaller model, and a smaller model can answer worse. So the goal is not minimum latency; it is the lowest latency that still clears the quality bar the feature needs, which means every latency win from swapping models has to be checked against an evaluation of the output, not just the stopwatch.
What we’ll filter on
- Which stage owns the time, retrieval, prompt assembly, first-token wait, generation, tool calls, or network?
- p50 versus p99 per stage, is the pain in the typical path or the tail?
- Perceived versus total latency, does the user need the answer faster, or just to see it start sooner?
- Input size versus output size, is the cost in what the model reads or in what it writes?
- Quality headroom, how much answer quality can this feature trade for speed before it fails its job?
- Contention and reuse, is the endpoint queuing under load, and does a shared prompt prefix repeat across calls?
The latency landscape
The levers each target a specific stage, and naming the stage each one touches is most of the skill.
Streaming cuts perceived latency, not total latency. Instead of waiting for the full response, you stream tokens as they generate, so the user sees the first words at time-to-first-token rather than after the whole answer is written. On Bedrock this is the streaming Converse call or the response-stream invoke; the total generation time is unchanged, but a two-second wait that starts producing text at 400 milliseconds feels dramatically faster. This is the highest-leverage change for a chat-shaped feature and does nothing measurable for a batch job nobody watches.
A smaller or distilled model cuts both first-token wait and per-token generation time, because a smaller model reads and writes faster. This is the biggest single lever on raw latency and the one with the sharpest trade, since the smaller model may answer worse. Model families on Bedrock span this range deliberately: a fast, small model for the latency-sensitive path and a larger one where the answer quality justifies the wait. The right version of this lever is often routing, sending easy requests to the small model and only the hard ones to the large one.
Fewer output tokens cuts total time directly, because generation is linear in tokens produced. Tightening the prompt to ask for a shorter answer, capping max tokens, and cutting the model off from restating the question all reduce the part of the request that grows with length. This is free latency when the answer was padded anyway, and it interacts with streaming: a shorter answer finishes streaming sooner.
Prompt caching cuts first-token wait when a large chunk of the prompt is identical across calls. Bedrock prompt caching lets you mark a stable prefix, a long system prompt, a fixed instruction block, a document reused across a session, so the model skips reprocessing it on subsequent calls within the cache lifetime. The saving is on input processing, so it helps most when the shared prefix is large relative to the variable part, and it does nothing for output generation.
Trimming retrieved context and prompt size cuts first-token wait by giving the model less to read. Retrieval that returns twenty passages when three would do inflates the input, and every extra token is time before the first output token and money on the bill. RerankingA second pass that re-scores a wide set of retrieved candidates and keeps only the few most relevant, so the expensive model reads less. to the few passages that actually matter, and cutting conversation history to what the turn needs, shrinks the input the model must process. Smaller prompts are faster prompts, and they often improve answer quality by removing distractors.
Running retrieval and other work in parallel cuts total time when stages are independent. If a request needs a vector search and a separate metadata lookup, and neither depends on the other, running them concurrently makes the pair cost the slower of the two rather than the sum. The same applies to independent tool calls. Anything on the critical path that does not depend on an earlier result is a candidate to move off the serial chain.
Latency-optimised inference, where the model and region offer it, cuts first-token wait and generation time by serving the request on infrastructure tuned for speed. Bedrock exposes this as a latency-optimised setting for supported models, trading a higher price for lower latency on the same model without dropping to a smaller one. It is the lever to reach for when you have hit the quality floor and cannot shrink the model further.
Cross-region inference profiles cut the tail that comes from contention. A cross-region inference profile lets Bedrock route a request across multiple regions, spreading load so a busy region does not queue your call. It does little for a single uncontended request, but it flattens the p99 spikes that come from regional saturation at peak, which is often exactly where the twelve-second tail lives.
Provisioned Throughput cuts the tail that comes from on-demand queuing by reserving dedicated model capacity, so requests are not competing in a shared pool. It is a capacity and cost decision more than a per-request tweak, and it earns its keep for steady, high-volume, latency-sensitive traffic rather than spiky low-volume workloads.
Side by side
| Lever | Stage it targets | Total latency | Perceived latency | Tail (p99) | Quality risk |
|---|---|---|---|---|---|
| Streaming | First-token to display | ✗ | ✓ | ✗ | none |
| Smaller / distilled model | First-token and generation | ✓ | ✓ | ✓ | high |
| Fewer output tokens | Generation | ✓ | ✓ | ✗ | some |
| Prompt caching | First-token (input reuse) | ✓ | ✓ | ✗ | none |
| Trim retrieved context | First-token (input size) | ✓ | ✓ | ✗ | can improve |
| Parallel retrieval / tools | Independent stages | ✓ | ✓ | ✓ | none |
| Latency-optimised inference | First-token and generation | ✓ | ✓ | ✓ | none |
| Cross-region inference profile | Contention | ✗ | ✗ | ✓ | none |
| Provisioned Throughput | Queuing under load | ✗ | ✗ | ✓ | none |
The table reads as a diagnosis tool. If the pain is that answers start too late, streaming and first-token levers own it; if the pain is a slow tail under load, the contention levers own it; if the raw number is just too high everywhere, the model and input-size levers do the heavy lifting.
Before any of that, a picture of where the seconds actually go on this feature’s slow path:
The picks in depth
Start by instrumenting, because everything else is guessing until the stages are measured. Wrap each stage in timing, emit the durations, and aggregate them as percentiles rather than averages; CloudWatch can hold the per-stage p50 and p99, and Bedrock’s own invocation metrics and model-invocation logging give you the model-side numbers including where time-to-first-token sits. The output of this step is a bar per stage at p50 and p99, and it usually settles the argument the team was having by taste. In the picture above, retrieval is not the problem the retrieval camp thought it was; the tail lives in the model call and an occasional slow tool.
With the diagnosis in hand, the levers sort themselves. The p99 first-token wait is the fattest bar, so the input-side levers come first: trim retrieval from twenty passages to the three that rerank highest, cut conversation history to the turns that matter, and cache the stable prefix. Prompt caching is the clean win here because the system prompt and instruction block repeat on every call in a session; marking them cached means the model stops re-reading them each turn, and the first-token wait drops without touching the answer. Trimming context does double duty, shaving first-token time and often improving quality by removing passages that were only distracting the model.
Streaming is the change that most improves how the feature feels, and it is nearly free to add. It does not move the 4.2-second total at all, which is exactly why the team measuring only averages will underrate it, but it moves the first visible token from around a second to a few hundred milliseconds, and for a chat feature that is the difference between responsive and broken. Ship it alongside the input-side cuts, not instead of measuring, because it hides a slow total rather than fixing one.
The model swap is the biggest lever and the one to reach for deliberately. Routing beats a blanket downgrade: send the short, factual questions to a fast small model and reserve the large one for the questions that need it, so the median gets much faster while the hard tail keeps its quality. Every such change has to be checked against an evaluation of answer quality, because a smaller model that is two seconds faster and wrong is not a win. Where the small model still is not fast enough and the quality floor will not allow going smaller, latency-optimised inference buys speed on the same model for a higher price, and it is the right next step rather than sacrificing more quality.
The tail levers are for the p99 spikes that instrumentation ties to load rather than to any one request. If the slow tail correlates with peak traffic and the endpoint is queuing, a cross-region inference profile spreads the load across regions and flattens the contention spikes, and Provisioned Throughput reserves dedicated capacity for steady high-volume traffic so requests stop competing in the on-demand pool. Neither helps a single uncontended slow request, so reach for them only when the data shows contention, not as a reflex. Parallelising the independent work, running the vector search and the metadata lookup at once instead of in series, quietly removes whichever of them was not the slower one from the critical path, and it costs nothing but a code change.
A worked example: cutting the tail without dropping the model
The team instruments the pipeline and gets the picture above: p50 at 3.1 seconds and p99 at 11.8. The first-token wait dominates the tail at 4.5 seconds p99, generation adds 3.8, and a slow tool call adds another 2.6 when it fires. Retrieval, the stage two people wanted to rebuild, is 0.7 seconds at worst.
They attack the biggest bars in order. Reranking retrieval from twenty passages to four cuts roughly 3,000 tokens of input, and the first-token p99 falls from 4.5 to about 2.9 seconds. Marking the system prompt and instruction block as a cached prefix takes another slice off first-token wait on every turn after the first, dropping it to around 2.1. Capping max output tokens and prompting for a tighter answer pulls generation p99 from 3.8 to 2.7. The slow tool call turns out to be running serially after retrieval for no reason; moving it to run in parallel with the vector search removes it from the critical path except when the model genuinely needs its result mid-answer.
Then they add streaming, which does not change any of those totals but moves the first visible token to about 350 milliseconds, so the feature feels responsive even on the slow path. The p99 end-to-end lands near 6 seconds, roughly half what it was, and the model is unchanged, so answer quality is exactly what it was before. Only after all of that, with the quality floor still respected, do they consider latency-optimised inference for the remaining first-token wait, and they leave the model swap on the shelf because they never needed it. The point is the order: measure, attack the fattest stage with the lever it responds to, protect quality, and reach for the drastic lever last.
What’s worth remembering
- Measure each stage as p50 and p99 before touching anything; an average hides the tail, and the tail is the request users are complaining about.
- The model call is two stages, not one: time-to-first-token, set by how much it reads and how contended the endpoint is, and per-token generation, set by model size and output length.
- Streaming cuts perceived latency, not total latency; it moves the first visible token earlier and is the highest-leverage change for a chat feature, and does nothing for a batch job.
- A smaller or distilled model is the biggest single lever on raw latency and the one with the sharpest quality trade; prefer routing easy requests to it over a blanket downgrade.
- Fewer output tokens cut total time linearly, because generation grows with tokens produced; cap max tokens and stop the model restating the question.
- Prompt caching cuts first-token wait by skipping reprocessing of a shared prefix; it helps most when the stable part of the prompt is large relative to the variable part.
- Trimming retrieved context and history shrinks the input the model must read, cutting first-token time and often improving quality by removing distractors.
- Run independent work in parallel so a stage costs the slower of two, not their sum; a tool call that does not depend on retrieval should not sit behind it.
- Latency-optimised inference buys speed on the same model for a higher price, and cross-region inference profiles plus Provisioned Throughput flatten the p99 spikes that come from contention and queuing.
- The fastest option is usually a smaller model, so balance latency against quality; check every speed win against an evaluation of the output, and reach for the drastic lever last.