Switching Foundation Models Without Shipping Code

The situation

Eight product services call Amazon Bedrock. Each one carries the model id in its source: a constant in a configuration module for most of them, inline in the request builder for two. Sitting next to that id is the inference configuration the service wants (temperature, top-p, maximum tokens, stop sequences) and the prompt text it sends, all of it shipped as code and released the way code is released.

A cheaper model in the same family becomes available, and an offline evaluation says it holds quality on six of the eight workloads. Moving those six means six pull requests, six reviews, six release trains. Two of the six deploy weekly by policy. One belongs to a team halfway through a re-platform that has not shipped anything in five weeks. The saving is real and the calendar swallows it.

Six weeks later a region has a bad afternoon and the same arithmetic runs again under time pressure: eight hotfix branches, eight approvals, eight pipelines, on a Sunday, to change one string. A deprecation notice on the model itself would force the same work again, on a deadline nobody here sets. Written down, what the platform team wants is dynamic model selection and provider switching without requiring code modifications, and that turns out to be several different mechanisms rather than one.

What actually matters

Start with what is actually changing. It is a handful of values: which model id serves this workload, the inference configuration that goes with it, and which prompt version it pairs with. Those values move on a completely different clock from the code around them. A deployment pipeline is built for the risk profile of new code, so it prices every change at that risk: review, tests, staging soak, a canary, a release window. Put a one-string change through it and the change inherits the slowest of the eight pipelines it has to traverse. Pull the values out of the artefact that carries the risk and the price falls away; everything below is a variation on where they go instead.

Speed on its own is not the goal, because the reason those pipelines are slow is that they contain the safety. A change that reaches eight services in ninety seconds, with no way to stage it and no way to get back, is worse than eight deploys. A model swap that degrades answer quality throws no exception. Nothing in the request path notices. Whatever replaces the deploy has to carry the staging and the reversal with it: reach a slice of traffic first, watch a metric, and put the old value back without a human in the loop. Time to switch and blast radius pull against each other, and a mechanism that only improves the first one has moved the risk rather than reduced it.

Then there is what the indirection must not cost you. Today each of the eight services calls Bedrock under its own IAM role, so the invocation is attributable to that service in CloudTrail and its spend lands on that team’s tag. Any layer that sits between the caller and the model risks collapsing all of that into one identity and one bill, and re-establishing per-team attribution afterwards is real work that nobody costs in at design time. Scope goes the same way. An IAM policy naming the exact model a service may invoke is a real control, and it stops working once the service talks to something else holding a broad grant on its behalf.

Finally, latency and request shape. Two of the eight stream tokens to a browser, so time to first token is visible to a person waiting. Sixty milliseconds of extra hop costs a batch job nothing and costs that path a noticeable pause. Shape matters just as much. A configurable model id gets you nowhere if switching family also means rewriting the request body, the stop-sequence field and the way tool schemas are declared. That rewrite is a code change, and you are back where you started. Two problems, then. Where do the values live, and can the call site accept a different model without the payload changing?

What we’ll filter on

1. Time to switch. How long from the decision to every caller using the new model, and does any part of it require a deploy?
2. Blast radius. Can the change reach a slice of traffic first, and does it revert on its own when a metric turns, or does reversal need a person?
3. Attribution and scope. Do per-team cost attribution and per-service IAM scoping survive the indirection, or do they have to be rebuilt?
4. Added latency. What does the mechanism cost per request, and what does it do to time to first token on a streaming path?
5. Provider reach. Could this ever serve a model that does not live in Bedrock?

The landscape

The flexible architecture patterns on offer differ mainly in where the indirection sits. Five places: the caller, a shared library, a configuration service the caller reads, a service the caller talks to instead of Bedrock, or Bedrock itself.

The model id in the code

The starting position, and worth naming rather than skipping, because it is correct for a single service that deploys often and has one model. The id sits beside the prompt it was tuned against, the two are versioned together, and a change is reviewed by the people who own the workload. Nothing is added and nothing is hidden. It fails on exactly one axis, the number of pipelines a change has to cross, and at eight services that axis is the entire problem.

A shared client library

The reflex answer: put the Bedrock call, the model id, the inference configuration and the retry policy into a package that every service depends on, and change it in one place. It genuinely helps with drift, since the retry and timeout behaviour stops being reimplemented eight times. It does not help with the switch, because a new value means a new library version, and a new library version means eight dependency bumps and eight deploys. The change is centralised and the release is not, so the calendar cost is unchanged and there is now a coupling between all eight services and one package’s release cadence.

Dynamic configuration in AWS AppConfig

AWS AppConfig holds the values outside the deployment artefact and hands them to the application at runtime. The service stores a configuration profile, validates it before it goes anywhere, and deploys it to an environment on a chosen strategy. The application reads the current value through the AppConfig Agent, which runs as a Lambda extension, a sidecar, or a local process, and polls and caches on your behalf. The model id, the inference configuration and the prompt version become configuration data read per request out of a local cache rather than constants compiled in, so changing them is a configuration deployment and touches no repository.

Two features are what make this more than a shared parameter store. The first is the deployment strategy, which spreads the change over a growth period rather than flipping every host at once, so a bad value reaches a fraction of traffic before it reaches all of it. The second is the bake time paired with a CloudWatch alarm. AppConfig watches the alarm through the bake window and rolls the configuration back automatically if it fires, which restores the staged rollout and the automatic reversal the deploy pipeline was providing. Validators (a JSON Schema, or a Lambda that checks the model id is one the account actually has access to) reject a malformed configuration before deployment starts.

An internal inference endpoint

The heavier option is to stop letting services call Bedrock at all and give them an internal endpoint instead: Amazon API Gateway in front of an AWS Lambda that resolves the model from configuration and makes the call. Every team points at one URL, and the platform team owns what happens behind it. Its design is a subject of its own. What matters here is what it changes about the five filters, which is that the indirection moves out of the caller’s process and into a network hop somebody has to run.

Bedrock’s own indirection

Bedrock offers two mechanisms that solve parts of this without any configuration layer at all.

The Converse API gives a uniform request and response shape across model families. The same message list, the same inference configuration block, the same tool-specification format and the same streaming interface work against models from different providers, with model-specific parameters confined to an optional additional-fields object. Under the older per-model invocation API, switching family meant rewriting the request body; under Converse, the body stays and the model id changes. That is the shape half of the problem solved in the SDK, and it applies whether or not you ever adopt a configuration service.

An application inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. is an ARN you create in your own account that points at a model, carries your cost-allocation tags, and can be passed anywhere a model id can. The caller holds the profile ARN; the profile decides which model and which region actually serve the request. Repointing the profile changes what runs without the caller knowing, and because the profile is the thing being tagged, per-team cost attribution and IAM scoping ride on it rather than being lost. A profile can also wrap a cross-region inference profile, which is how the same handle also spreads load across regions.

Evaluation

Side by side

Read the table by column rather than by row and the shape of the answer appears. Three mechanisms clear the first column, so “no deploy” does not discriminate. The second column is where the shared library and the bare inference profile fall down, because repointing a profile is instant and total with nothing watching it. The third and fourth columns are where the internal endpoint pays, and the fifth is the only column it wins on its own. Nothing in the table is a complete answer by itself: the profile and Converse fix the request shape and the handle, AppConfig fixes the values and the rollout, and they are solving different halves.

The solution

Take the values out of the code with AWS AppConfig and take the request shape out of the model family with the Converse API. Those two together answer the scenario as posed, and neither of them adds a hop.

The configuration profile holds one entry per workload rather than per service, because a service may run several, and a workload that routes between a cheap and a capable model names both of them in its binding. Each entry names the model id (or better, the application inference profile ARN, so the tags and the region policy come with it), the inference configuration that model wants, and the prompt version to pair with it. The application reads it per request from the AppConfig Agent’s local cache, so the read is a call to localhost rather than a network round trip. A request arriving after a configuration deployment picks up the new value on the next poll, with nothing restarted. Callers use Converse with the same message list and the same tool specifications regardless of which model the configuration names, and model-specific parameters go in the additional-fields object rather than forcing a per-family branch in the caller.

Four things decide whether this works in practice, and three of them are easy to get wrong.

The deployment strategy and the polling interval decide the speed, both ways

AppConfig’s deployment strategy sets the growth type, the growth rate, the deployment duration and the bake time; the agent’s polling interval sets how long a host waits before it sees a change. Add them together and that is the true time to switch, and, more importantly, the true time to unswitch. A linear deployment over thirty minutes with a sixty-second poll keeps a bad value on a slice of traffic for a while before it is everywhere, which is what you want. It also means the rollback is not instantaneous, which is what people forget. Set the pair deliberately for each configuration: a prompt version can afford a slow, wide bake, while a model id you are moving because a region is unhealthy wants a fast strategy and a short poll. Keep the two configurations separate so they can carry different strategies.

The rollback is only automatic if you wire it. Attach a CloudWatch alarm to the deployment and give it a bake time long enough for the signal to appear. The alarm has to watch something a bad model actually moves. Error rate and latency catch a model that is failing or slow. A model returning confident nonsense moves neither, so add a quality proxy: guardrail intervention rate, retrieval-answered rate, or a thumbs-down rate from the product. This is the same instinct as the metric you would use to call an A/B test, running as an automatic stop instead of a decision.

Don’t let the abstraction flatten the models

The tempting shape is one configuration blob shared by every workload, with the model id as the only variable. It fails on the first switch, because inference configuration is not portable. Temperature scales differ between families, maximum-token limits differ, stop sequences are tokenised differently, and tool-schema handling varies even under Converse’s uniform envelope. Pin the inference configuration, the stop sequences and the tool schema per model id. Changing the id then changes the whole block it belongs to, rather than dropping a new model into settings tuned for the old one. It costs a little duplication in the configuration and it removes an entire class of switch that looks fine in staging and is subtly worse in production. Prompt versions belong to the same block for the same reason, which is why prompts and models are versioned together.

The endpoint is a separate decision, taken later

Amazon API Gateway in front of an AWS Lambda earns its extra hop when you need something the configuration layer cannot give you. That list is short: central rate limiting across teams, one place to apply a guardrail so no caller can skip it, per-team API keys and usage plans, or a model from a provider that is not in Bedrock at all. If none of those applies, the hop costs latency, and it lands hardest on the streaming path where it delays the first token. It also costs the per-service IAM scoping and cost attribution you had, because every call now arrives at Bedrock under one execution role and one identity. Request context, tagging and per-team usage plans can rebuild both. That rebuild is work to plan for, not to discover.

The failure mode to plan for is what the endpoint becomes once eight teams depend on it. It is now a single point of failure and a quota funnel. Throttling that used to spread across eight roles and eight sets of account-level limits converges on one path, and one team’s traffic spike becomes everybody’s throttling event. It has to inherit the behaviour the direct callers had, which means backoff, jitter and a circuit breaker on the outbound side, per-team quotas on the inbound side, and its own health as a first-class alarm.

Worked example

The platform team starts with the shape rather than the values. Every one of the eight services already calls Bedrock, so the first change is a mechanical one: move each call from the per-model invocation API to Converse, keeping the model id hardcoded exactly where it was. Nothing switches, nothing improves, and eight deploys go out. What it buys is that the request body is now family-independent, and it is done once while nothing is on fire.

Then the values move. One AppConfig application, one environment per stage, and two configuration profiles: model-bindings and prompt-versions, so each can carry its own deployment strategy. A binding looks like this, one entry per workload:

{
  "support-summariser": {
    "modelId": "arn:aws:bedrock:ap-southeast-2:111122223333:application-inference-profile/support-tier",
    "inferenceConfig": { "temperature": 0.2, "maxTokens": 900, "topP": 0.9 },
    "stopSequences": ["</summary>"],
    "promptVersion": "summariser-v7"
  }
}

A JSON Schema validator rejects a profile with a missing inference configuration, and a Lambda validator rejects a model id the account has no access to, which catches the most common bad deployment before it starts. Each service reads its own workload key through the AppConfig Agent and calls Converse with whatever it finds. The eight services now have no model id in them.

The cheaper model goes out as one configuration deployment against model-bindings for six workloads: linear growth over forty-five minutes, ten-minute bake, a CloudWatch composite alarm on error rate, p95 latency and guardrail intervention rate. Twenty minutes in, one workload’s intervention rate climbs; the new model is more talkative and trips a topic guardrail more often. The alarm fires, AppConfig rolls the whole deployment back, and the on-call engineer reads about it afterwards instead of during. The five clean workloads go out again on their own, the sixth gets its prompt adjusted first. No repository was touched.

Two months later a region has a bad afternoon. The response is one configuration change to point the affected profile ARNs elsewhere, on the fast strategy with a thirty-second poll, live everywhere in under three minutes. The Sunday of eight hotfixes does not happen. The internal endpoint is still not built, because nothing on its list has come up. When a marketing team later asks for a model that is not in Bedrock, the endpoint gets built for that one workload, and it reads its bindings from the same configuration profiles.

What’s worth remembering

1. The model id, the inference configuration and the prompt version change on a different clock from the code around them, so shipping them inside the deployment artefact prices every switch at the cost of the slowest pipeline carrying it.
2. AWS AppConfig moves those values out of the artefact and gives back the pipeline’s safety, because the deployment strategy stages the change and a bake time with a CloudWatch alarm reverses it without a person.
3. The Converse API solves the other half by giving one request and response shape across model families, so a switch changes the id rather than the payload; adopt it before you need it.
4. An application inference profile ARN is a handle you can repoint, and because it carries your cost-allocation tags, per-team attribution and IAM scoping survive the indirection.
5. A shared configuration blob that flattens per-model differences produces switches that pass staging and quietly degrade in production; pin the inference configuration, stop sequences and tool schema to the model id they were tuned for.
6. Amazon API Gateway in front of an AWS Lambda is worth its hop only for central rate limiting, a guardrail nobody can skip, per-team keys, or a non-Bedrock provider; it becomes a quota funnel and a single point of failure unless it inherits the throttling and breaker behaviour the direct callers had.