Switching Foundation Models Without Shipping Code

The situation

Eight product services call Amazon Bedrock. Each one carries the model id in its source: a constant in a configuration module for most of them, inline in the request builder for two. Sitting next to that id is the inference configuration the service sends (temperature, topP, maxTokens, stopSequences) and the prompt text, all of it shipped as code and released the way code is released.

A cheaper model in the same family becomes available, and an offline evaluation says it holds quality on six of the eight workloads. Moving those six means six pull requests, six reviews, six release trains. Two of the six deploy weekly by policy. One belongs to a team halfway through a re-platform that has not shipped anything in five weeks. The saving is real and the release schedule defers it for weeks.

Six weeks later a region has a bad afternoon and the same arithmetic runs again under time pressure: eight hotfix branches, eight approvals, eight pipelines, on a Sunday, to change one string. A deprecation notice on the model itself would force the same work again, on a deadline nobody here sets. Written down, what the platform team wants is dynamic model selection and provider switching without requiring code modifications, and that turns out to be several different mechanisms rather than one.

What actually matters

Start with what is actually changing. It is a handful of values: which model id serves this workload, the inference configuration that goes with it, and which prompt version it pairs with. Those values move on a completely different clock from the code around them. A deployment pipeline is built for the risk profile of new code, so it applies that whole treatment to every change: review, tests, staging soak, a canary, a release window. Put a one-string change through it and the change inherits the slowest of the eight pipelines it has to traverse. Move the values out of the artefact that carries the risk and none of that applies to them; everything below is a variation on where they go instead.

Speed on its own is not the goal, because the reason those pipelines are slow is that they contain the safety. A change that reaches eight services in ninety seconds, with no way to stage it and no way to get back, is worse than eight deploys. A model swap that degrades answer quality throws no exception. Nothing in the request path reports it. Whatever replaces the deploy has to carry the staging and the reversal with it: reach a slice of traffic first, watch a metric, and put the old value back without a human in the loop. Time to switch and blast radius pull against each other, and a mechanism that only improves the first one has moved the risk rather than reduced it.

Then there is what the indirection must not take away. Today each of the eight services calls Bedrock under its own IAM role, so the invocation is attributable to that service in CloudTrail and its spend lands on that team’s tag. Any layer sitting between the caller and the model risks collapsing all of that into one identity and one bill, and re-establishing per-team attribution afterwards is real work that rarely gets planned for. Scope goes the same way. An IAM policy naming the exact model a service may invoke is a real control, and it stops working once the service calls something else holding a broad grant on its behalf.

Finally, latency and request shape. Two of the eight stream tokens to a browser, so time to first token is visible to a person waiting. Sixty milliseconds of extra hop is nothing to a batch job and a noticeable pause on that path. Shape matters just as much. A configurable model id gets you nowhere if switching family also means rewriting the request body, the stop-sequence field and the way tool schemas are declared. That rewrite is a code change, and you are back where you started. Two problems, then. Where do the values live, and can the call site accept a different model without the payload changing?

What we’ll filter on

1. Time to switch. How long from the decision to every caller using the new model, and does any part of it require a deploy?
2. Blast radius. Can the change reach a slice of traffic first, and does it revert on its own when a metric turns, or does reversal need a person?
3. Attribution and scope. Do per-team cost attribution and per-service IAM scoping survive the indirection, or do they have to be rebuilt?
4. Added latency. What does the mechanism add per request, and what does it do to time to first token on a streaming path?
5. Provider reach. Could this ever serve a model that does not live in Bedrock?

The landscape

The flexible architecture patterns on offer differ mainly in where the indirection sits. Five places: the caller, a shared library, a configuration service the caller reads, a service the caller talks to instead of Bedrock, or Bedrock itself.

The model id in the code

The starting position, and worth naming rather than skipping, because it is correct for a single service that deploys often and has one model. The id sits beside the prompt it was tuned against, the two are versioned together, and a change is reviewed by the people who own the workload. Nothing is added and nothing is hidden. It fails on exactly one axis, the number of pipelines a change has to cross, and at eight services that axis is the entire problem.

A shared client library

The reflex answer: put the Bedrock call, the model id, the inference configuration and the retry policy into a package that every service depends on, and change it in one place. It genuinely helps with drift, since the retry and timeout behaviour stops being reimplemented eight times. It does not help with the switch, because a new value means a new library version, and a new library version means eight dependency bumps and eight deploys. The change is centralised and the release is not, so the elapsed time to switch is unchanged and there is now a coupling between all eight services and one package’s release cadence.

Dynamic configuration in AWS AppConfig

AWS AppConfig holds the values outside the deployment artefact and hands them to the application at runtime. The service stores a configuration profile, validates it before it goes anywhere, and deploys it to an environment on a chosen strategy. The application reads the current value through the AppConfig Agent, which runs as a Lambda extension, a sidecar, or a local process, and polls and caches on your behalf. The model id, the inference configuration and the prompt version become configuration data read per request out of a local cache rather than constants compiled in, so changing them is a configuration deployment and touches no repository.

Two features make this more than a shared parameter store. The first is the deployment strategy, which spreads the change linearly or exponentially over a deployment window rather than flipping every host at once, so a bad value reaches a fraction of traffic before it reaches all of it. The second is automatic rollback. An AppConfig environment carries up to five CloudWatch alarms as monitors, and the service monitors them during the deployment and through the bake time that follows, rolling the configuration back if one goes into alarm. Between them that restores the staged rollout and the automatic reversal the deploy pipeline was providing. A configuration profile can also carry up to two validators, a JSON Schema and a Lambda, which run before the deployment starts and reject a bad configuration outright.

An internal inference endpoint

The heavier option is to stop letting services call Bedrock at all and give them an internal endpoint instead: Amazon API Gateway in front of an AWS Lambda that resolves the model from configuration and makes the call. Every team points at one URL, and the platform team owns what happens behind it. Its design is a subject of its own. What matters here is what it changes about the five filters, which is that the indirection moves out of the caller’s process and into a network hop somebody has to run.

Bedrock’s own indirection

Bedrock offers two mechanisms that solve parts of this without any configuration layer at all.

The Converse API gives one request and response shape across every model that supports messages. One message list, one inferenceConfig block holding temperature, topP, maxTokens and stopSequences, one tool-specification format and one streaming operation, with model-specific parameters confined to the optional additionalModelRequestFields object. Under the older per-model invocation API, switching family meant rewriting the request body; under Converse, the body stays and the model id changes. That is the shape half of the problem solved in the SDK, and it applies whether or not you ever adopt a configuration service.

An application inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. is an ARN you create in your own account from a foundation model or a cross-region inference profile, carrying tags of your own. Pass it in the modelId field of InvokeModel or Converse and Bedrock meters usage against those tags, so per-team attribution holds. IAM scoping holds too: a policy can name the profile ARN, and the bedrock:InferenceProfileArn condition key confines a role to reaching the model only through it, as long as the policy also allows the underlying model in each region. What the profile does not give you is a switch. There is no update operation, so it points at whatever it was created from for as long as it exists, and moving a workload means naming a different ARN.

Evaluation

Side by side

Read the table by column rather than by row and the shape of the answer appears. Two mechanisms clear the first column, and the same two clear the second. The shared library and the bare inference profile fall down on both. A new library version is still eight dependency bumps, and a profile that cannot be repointed is still a string sitting in eight repositories. The third and fourth columns are where the internal endpoint loses, and the fifth is the only one it wins on its own. Nothing in the table is a complete answer by itself. The profile and Converse fix the request shape and the handle, AppConfig fixes the values and the rollout, and they are solving different halves.

The solution

Take the values out of the code with AWS AppConfig and take the request shape out of the model family with the Converse API. Those two together cover what this scenario needs, and neither of them adds a hop.

The configuration profile holds one entry per workload rather than per service, because a service may run several, and a workload that routes between a cheap and a capable model names both of them in its binding. Each entry names the model id (or better, the application inference profile ARN, so the tags and the region policy come with it), the inference configuration tuned for that model, and the prompt version to pair with it. The application reads it per request from the AppConfig Agent’s local cache, so the read is a call to localhost rather than a network round trip. A request arriving after a configuration deployment picks up the new value on the next poll, with nothing restarted. Callers use Converse with the same message list and the same tool specifications regardless of which model the configuration names, and model-specific parameters go in additionalModelRequestFields rather than forcing a per-family branch in the caller.

Four things decide whether this works in practice, and three of them are easy to get wrong.

The deployment strategy and the polling interval decide the speed, both ways

AppConfig’s deployment strategy sets the growth type, the step percentage, the deployment time and the bake time; the agent’s polling interval, 45 seconds by default, sets how long a host waits before it sees a change. Add them together and that is the true time to switch, and, more importantly, the true time to unswitch. A linear deployment over thirty minutes with a sixty-second poll keeps a bad value on a slice of traffic for a while before it is everywhere, which is what you want. It also means the rollback is not instantaneous, which is what people forget. Set the pair deliberately for each configuration: a prompt version can take a slow, wide bake, while a model id you are moving because a region is unhealthy needs a fast strategy and a short poll. Keep the two configurations separate so they can carry different strategies.

The rollback is only automatic if you wire it. Register the CloudWatch alarms as monitors on the environment, with an IAM role that lets AppConfig read them, and set a bake time long enough for the signal to appear. The alarm has to watch something a bad model actually moves. Error rate and latency catch a model that is failing or slow. A model returning fluent, wrong answers moves neither, so add a quality proxy: guardrail intervention rate, retrieval-answered rate, or a thumbs-down rate from the product. This is the same instinct as the metric you would use to call an A/B test, running as an automatic stop instead of a decision.

Don’t let the abstraction flatten the models

The tempting shape is one configuration blob shared by every workload, with the model id as the only variable. It fails on the first switch, because inference configuration is not portable. Converse standardises the field names, not the values behind them: each model publishes its own parameter ranges and its own maximum-output-token ceiling, and support for tool use, for vision and for stop sequences is a per-model table rather than a property of the API. Pin the inference configuration, the stop sequences and the tool schema per model id. Changing the id then changes the whole block it belongs to, rather than dropping a new model into settings tuned for the old one. It duplicates a little configuration and removes an entire class of switch that looks fine in staging and is worse in production. Prompt versions belong to the same block for the same reason, which is why prompts and models are versioned together.

The endpoint is a separate decision, taken later

Amazon API Gateway in front of an AWS Lambda is justified when you need something the configuration layer cannot give you. That list is short: central rate limiting across teams, one place to apply a guardrail so no caller can skip it, per-team API keys and usage plans, or a model from a provider that is not in Bedrock at all. If none of those applies, the hop adds latency, and it lands hardest on the streaming path, where it delays the first token and where the Lambda has to use response streaming through a proxy integration rather than returning a buffered body. It also removes the per-service IAM scoping and cost attribution you had, because every call now arrives at Bedrock under one execution role and one identity. Request context, tagging and per-team usage plans can rebuild both. That rebuild is work to plan for, not to discover.

The failure mode to plan for is what the endpoint becomes once eight teams depend on it. It is now a single point of failure and a quota funnel. Throttling that used to spread across eight roles and eight sets of account-level limits converges on one path, and one team’s traffic spike becomes everybody’s throttling event. It has to inherit the behaviour the direct callers had, which means backoff, jitter and a circuit breaker on the outbound side, per-team quotas on the inbound side, and its own health as a first-class alarm.

Worked example

The platform team starts with the shape rather than the values. Every one of the eight services already calls Bedrock, so the first change is a mechanical one: move each call from the per-model invocation API to Converse, keeping the model id hardcoded exactly where it was. Nothing switches and eight deploys go out. At the end of it the request body is family-independent, done once while nothing is on fire.

Then the values move. One AppConfig application, one environment per stage, and two configuration profiles: model-bindings and prompt-versions, so each can carry its own deployment strategy. A binding looks like this, one entry per workload:

{
  "support-summariser": {
    "modelId": "arn:aws:bedrock:ap-southeast-2:111122223333:application-inference-profile/a1b2c3d4e5f6",
    "inferenceConfig": {
      "temperature": 0.2,
      "maxTokens": 900,
      "topP": 0.9,
      "stopSequences": ["</summary>"]
    },
    "promptVersion": "summariser-v7"
  }
}

Two validators is the limit and two is what this needs. The JSON Schema rejects an entry with no inference configuration, and the Lambda, which AppConfig cuts off at fifteen seconds, rejects a profile ARN the account cannot invoke. Both run before the deployment starts, which catches the most common bad deployment. Each service reads its own workload key through the AppConfig Agent and calls Converse with whatever it finds. The eight services now have no model id in them.

The cheaper model goes out as one configuration deployment against model-bindings for six workloads: linear growth over forty-five minutes, ten-minute bake, a CloudWatch composite alarm on error rate, p95 latency and guardrail intervention rate. Twenty minutes in, one workload’s intervention rate climbs; the new model returns longer answers and trips a topic guardrail more often. The alarm goes off, AppConfig rolls the whole deployment back, and the on-call engineer reads about it afterwards instead of during. The five clean workloads go out again on their own, the sixth gets its prompt adjusted first. No repository was touched.

Two months later a region has a bad afternoon. The response is one configuration change naming profile ARNs in another region, on the fast strategy with a thirty-second poll, live everywhere in under three minutes. The Sunday of eight hotfixes does not happen. The internal endpoint is still not built, because nothing on its list has come up. When a marketing team later asks for a model that is not in Bedrock, the endpoint gets built for that one workload, and it reads its bindings from the same configuration profiles.

What’s worth remembering

1. The model id, the inference configuration and the prompt version change on a different clock from the code around them, so shipping them inside the deployment artefact makes every switch as slow as the slowest pipeline carrying it.
2. AWS AppConfig moves those values out of the artefact and gives back the pipeline’s safety, because the deployment strategy stages the change and a CloudWatch alarm registered on the environment reverses it without a person.
3. The Converse API solves the other half by giving one request and response shape across every model that supports messages, so a switch changes the id rather than the payload; adopt it before you need it.
4. An application inference profile is fixed at creation and has no update operation, so a switch means naming a different ARN; keep that ARN in configuration, and its tags keep per-team attribution and IAM scoping intact.
5. A shared configuration blob that flattens per-model differences produces switches that pass staging and degrade in production without raising an error; pin the inference configuration, stop sequences and tool schema to the model id they were tuned for.
6. Amazon API Gateway in front of an AWS Lambda is justified only by central rate limiting, a guardrail nobody can skip, per-team keys, or a non-Bedrock provider; it becomes a quota funnel and a single point of failure unless it inherits the throttling and breaker behaviour the direct callers had.