The situation
A product team runs a customer-facing assistant on Amazon Bedrock. Every request goes to one large capable model, and the bill has grown faster than usage because the model is priced for its hardest job and asked to do its easiest one thousands of times a day. The traffic is a mix. A large share is simple: classify an incoming message into one of a handful of intents, pull an order number out of a sentence, answer a factual question the knowledge base already contains. A smaller share is genuinely hard: reconcile a multi-part complaint, work through a returns policy with three conditions, plan a sequence of steps and explain the reasoning.
When the team samples the logs, the split is roughly eighty-twenty. Four in five requests are the kind a small model answers correctly and in a fraction of the time, at a fraction of the per-token price. One in five actually exercises the large model’s reasoning. Running everything through the flagship means the eighty per cent subsidises the twenty, both in cost and in the extra latency the big model carries on requests that never needed it.
The obvious move is to send easy requests to a cheap small model and hard ones to a large capable model. The obvious risk is getting the split wrong: route a hard request to the weak model and you get an answer that is fast, cheap, and wrong, delivered with the same confidence as a right one. The decision is how to classify each request, how much a misroute costs, and whether to build the router or let Bedrock run it.
What actually matters
The first thing to name is that model size is a cost lever, not a quality dial you turn up for safety. A bigger model is not uniformly better at everything; it is better at the tasks that need what it has, which is depth of reasoning and breadth of knowledge. On a single-step classification, a well-chosen small model matches it and answers quicker. So the question is never “is the big model better” in the abstract, it is “does this particular request need what the big model has”. Where the answer is no, you are paying for capability the request will not use.
The shape of the workload decides how much routing can save. If almost every request is hard, there is little to route away and the machinery earns nothing. If the traffic is genuinely mixed, with a large easy majority and a smaller hard tail, routing captures most of the flagship’s cost as savings on the majority while preserving quality on the tail. The eighty-twenty split is the case routing is built for; a workload that is uniformly hard, or uniformly trivial, does not need a router at all, it needs the one model that fits.
The cost of a misroute is asymmetric and it runs one way harder than the other. Send an easy request to the big model and you overpay by a few tokens and a little latency; the answer is still correct. Send a hard request to the small model and you can get a wrong answer that reads as authoritative, which reaches the customer and costs far more than the tokens you saved. The escalation threshold has to be set with that asymmetry in mind: it is cheaper to occasionally send a borderline-easy request to the big model than to occasionally send a hard one to the small model. When in doubt, route up.
That means the thing you actually have to measure is quality per route, not quality in aggregate. An overall accuracy number hides the failure that matters, because the misroutes are a minority inside a mostly-correct stream. You want to know how often the small model was handed something it got wrong, which means sampling the requests that went to the cheap path and checking them against what the capable model would have produced. Without per-route measurement you cannot tell a healthy router from one that is quietly degrading answers to save money.
And routing is not the only cost lever, so it should not be the only one you pull. A cache in front of the whole thing removes the repeated identical and near-identical requests before either model runs; trimming a bloated prompt cuts the per-call cost on both paths. Routing decides which model a request reaches; caching and prompt trimming decide whether it needs a model at all and how much it costs when it does. They compound, and the cheapest request is the one the cache answers.
What we’ll filter on
- Workload variety, is the traffic a genuine mix of easy and hard, or mostly one kind?
- Misroute cost, how bad is a wrong answer from the weak model, and how asymmetric is it against overpaying on the strong one?
- Classification signal, can the request’s difficulty be told from cheap signals (task type, length, a small classifier) or does it need the model to look?
- Measurability, can quality be measured per route, so a degrading cheap path is visible?
- Build versus managed, does a managed router within a model family fit, or does the split need custom logic across families?
- Stacking, does routing sit alongside caching and prompt trimming rather than replacing them?
The landscape
A rules or heuristic router classifies each request from cheap signals before any model runs. Route by task type when the caller already knows what it is asking for, so a classification endpoint goes to the small model and a reasoning endpoint goes to the large one. Route by input length as a rough proxy, since short inputs are more often simple and very long ones more often need the bigger context and reasoning, though length is a weak signal on its own. Route on the output of a tiny classifier, a small cheap model or a lightweight text classifier whose only job is to label the request easy or hard. Heuristic routing is transparent, free of an extra model call when it keys on task type or length, and easy to reason about; its weakness is that a hand-written rule cannot see difficulty that is not visible in the surface of the request.
API-based model cascading runs the cheap model first and promotes to the capable one when the cheap answer looks weak. The small model attempts every request; if it signals low confidence, or a validator finds the answer malformed or failing a check, the request is retried on the large model. This adapts to difficulty the request’s surface does not reveal, because the cheap model has actually attempted the task. The cost is that hard requests pay twice, once for the failed cheap attempt and again for the capable retry, so it wins when the easy majority is large enough that the double-paid tail stays cheap overall, and it depends on having a reliable signal that the cheap answer was inadequate.
Amazon Bedrock Intelligent Prompt Routing is the managed option. It routes each request within a single model family to the member it predicts will meet your quality bar at the lowest cost, so an easy request goes to the smaller cheaper model in the family and a hard one to the larger. You select a router, either one of the Bedrock-provided defaults or a custom router built from two or more models in the same family, and set the response-quality tolerance, how much difference from the strongest model’s response you are willing to accept; a fallback model catches anything the router cannot confidently place. It works through the Converse and InvokeModel APIs by pointing the request at the router instead of a specific model, and Bedrock predicts per request which model will do. AWS reports cost reductions of up to thirty per cent on suitable workloads without a meaningful quality drop, and there is no separate charge for the routing itself; you pay for whichever model actually serves the request. The constraint is that it routes within a family, not across arbitrary models, so it fits when a single family spans the range of capability you need.
Fanning out and aggregating inverts the trade. Where the routers above pick one model per request, this pattern uses specialized FMs to perform complex tasks in concert: send the same request to two or more of them in parallel and combine what comes back with custom aggregation logic for model ensembles. A majority vote settles a classification, best-of-N returns whichever candidate a judge model scores highest, and a merge takes the field each model is strongest on, one for the sentiment, another for the extracted entities, a third for the policy citation. The cost shape is plain: N models means N times the token spend and the latency of the slowest of them, so fanning out is the one member of this family that spends more rather than less. The extra spend is worth it where being wrong is expensive, a compliance classification that triggers a filing or a medical triage flag that decides who is seen first, and wasted on the intent labels that make up most of this traffic. Ensembles and routers are both model selection frameworks; one spends extra to raise confidence on a narrow slice of traffic, the other spends less to hold quality steady across a wide one.
Intelligent model routing systems have to be implemented somewhere, and where the decision runs is separate from what the decision says. Static routing configurations in application code are the plainest, a map from task type to model id that ships with the release and changes when you deploy. Step Functions for dynamic content-based routing puts the choice in a Choice state, so the state machine reads a field off the request and branches to a different specialized FM on each arm, which fits when the surrounding workflow already runs there. API Gateway with request transformations for routing logic moves it to the edge, rewriting the target model from a header or a path segment before the request reaches your code, which suits a platform whose callers pick their own tier. Intelligent model routing based on metrics reads recent latency, error rates or throttling counts before choosing, so a model that has started timing out sheds traffic without anyone deploying anything. Each home changes who can alter a route and how fast: a config in code needs a release, a Choice state needs a state-machine update, an edge transformation needs neither.
Underneath all of them is the same measurement loop. Whichever router you run, you sample the cheap path and check its answers against the capable model, and you tune the threshold from what you find rather than from a guess.
Evaluation
Side by side
| Approach | Sees hidden difficulty | Extra call on easy path | Hard requests pay twice | Managed | Tuning surface |
|---|---|---|---|---|---|
| Rules by task type | ✗ | ✗ | ✗ | ✗ | Route table |
| Rules by input length | ✗ | ✗ | ✗ | ✗ | Length cut-off |
| Small classifier router | Partly | ✓ (tiny) | ✗ | ✗ | Classifier and threshold |
| API-based model cascading | ✓ | ✗ | ✓ | ✗ | Confidence and validator |
| Fan-out ensemble | ✓ | ✓ (N calls) | ✓ (all of them) | ✗ | Model set, aggregation rule |
| Bedrock Intelligent Prompt Routing | ✓ | ✗ | ✗ | ✓ | Quality tolerance, fallback |
The solution
Start with the workload before touching a router, because the whole case rests on the split. Sample real traffic, sort it into easy and hard by hand, and get the ratio. A genuine eighty-twenty is the sweet spot: a large easy majority to move off the flagship and a real hard tail to protect. If the sample comes back mostly hard, routing saves little and adds a moving part for nothing, and the honest answer is to keep the one model that fits. If it comes back almost all trivial, drop to the small model outright and skip the router. Routing earns its complexity only on a mixed stream.
For a mixed stream where the difficulty is legible from the request itself, a rules or classifier router is the least machinery. When each endpoint already knows its task, a route table by task type is transparent and adds no extra model call: the intent classifier and the extraction job point at the small model, the reasoning endpoint points at the large one. Where task type is not enough, a tiny classifier that labels the request easy or hard gives a cheap signal without running the expensive model first. Length can supplement this as a coarse tiebreak, but do not lean on it alone; a short input can still be a hard reasoning problem and a long one can be a simple extraction from a wall of text.
When difficulty is not visible on the surface, API-based model cascading is worth it, with the escalation threshold set against the asymmetry of a misroute. The cheap model attempts everything; a low-confidence signal or a failed validation promotes the request to the capable model. Because a wrong cheap answer costs more than an unnecessary escalation, tune the threshold to escalate readily: it is better to send some easy requests up than to let hard ones through on the cheap path. The trade is that promoted requests pay for both models, so this pattern depends on the easy majority being large enough that the double-paid tail stays cheap, and on a promotion signal you trust.
Amazon Bedrock Intelligent Prompt Routing is the pick when a single model family spans the capability range and you would rather not build and maintain the router. Point the request at a router instead of a named model, and Bedrock predicts per request which family member will meet your quality tolerance at the lowest cost, falling back to a nominated model when it cannot place the request confidently. You tune one dial, the response-quality tolerance, and set the fallback; there is no separate routing charge, so you pay for the model that actually serves each request, and AWS reports up to thirty per cent savings on suitable workloads. Its boundary is the family: it will not route from one vendor’s small model to another’s large one, so it fits when the family you are on already offers a cheap member and a capable member. When the split you need crosses families, or hangs on business logic Bedrock cannot see, the custom router is the one that fits.
Whichever router runs, the non-negotiable is per-route measurement. Aggregate accuracy hides the misroutes because they are a minority inside a mostly-correct stream, so sample the requests that took the cheap path and check them against what the capable model produces. That sample tells you whether the threshold is set right and catches a router that has started quietly trading quality for cost. Then stack the other levers: a cache in front removes repeated requests before any model runs, and trimming the prompt cuts the per-call cost on both paths. Routing, caching, and trimming are separate cuts at the same bill, and they compound.
Worked example
The team samples a day of traffic and sorts it. Roughly sixty per cent is intent classification and short extraction, twenty per cent is factual questions the knowledge base already answers, and twenty per cent is the hard tail of multi-condition policy reasoning. Three shapes, and the flagship was serving all of them.
The classification and extraction slice is legible from the endpoint, so it takes a rules route straight to the small model with no extra call; the task type is the signal. The factual-question slice goes through the cache and the knowledge base first, and only the residue that needs generation reaches the small model, so most of it never pays for a large-model call at all. The hard policy slice is where difficulty hides inside ordinary-looking questions, so it runs small-model-first with escalation: the cheap model attempts the answer, and a validator that checks the policy conditions were all addressed promotes the weak ones to the capable model. The threshold is set to escalate on any unmet condition, because a wrong policy answer reaching a customer costs far more than a spare capable-model call.
After a fortnight the per-route sample tells the story. The cheap path handles the classification and factual slices with accuracy matching the old flagship-only numbers, and the policy path escalates about a third of its requests, which is the tail that genuinely needed reasoning. The flagship now runs on the twenty-odd per cent of traffic that exercises it, the cache absorbs the repeats, and the bill falls by more than half, most of it from moving the easy majority off the big model rather than from any single clever trick.
What’s worth remembering
- Model size is a cost lever, not a safety dial, so routing pays off on a genuinely mixed workload with a large easy majority and a smaller hard tail, and buys nothing on a stream that is uniformly hard or uniformly trivial.
- The misroute cost is asymmetric: overpaying on an easy request loses a few tokens, but a hard request on the weak model returns a confident wrong answer, so route up when in doubt.
- API-based model cascading adapts to hidden difficulty because the cheap model actually attempts the task, at the cost of hard requests paying for both models.
- Amazon Bedrock Intelligent Prompt Routing routes each request within a single model family to the cheapest member that meets your quality tolerance, with a fallback model and no separate routing charge.
- An ensemble buys agreement rather than speed: N models multiply the bill by N and finish no sooner than the slowest of them, so keep it for the decisions where being wrong is expensive.
- Measure quality per route, not in aggregate; sample the cheap path against the capable model, because misroutes are a minority hidden inside a mostly-correct stream.