Building a Deployment Pipeline for a GenAI Feature

The situation

A support assistant runs on Amazon Bedrock across three AWS accounts: dev, staging, production. The feature is five moving pieces. A published prompt version. A published guardrail version. A Bedrock agent alias with two action-group Lambda functions behind it. A knowledge base whose data source syncs from an S3 prefix. An inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. that the application invokes by ARN.

Every one of those pieces changes by clicking. Someone edits the prompt in the console, publishes a version, repoints the alias. Someone else tightens a guardrail filter and publishes that. The Lambda functions go out through a script on a developer laptop with production credentials in the shell. Production changed twice last Thursday afternoon and the change record is a thread in a chat channel.

Three weeks ago answer quality dropped for most of a day. Nobody could reconstruct which of the five pieces had moved, because three of them had moved that week and none of them left anything tying a change to a commit. The team wants what production readiness implies and they never built: one commit produces one release, the release has to prove itself before it reaches customers, and a bad release comes back out in minutes rather than in an afternoon of console archaeology.

What actually matters

The first thing to settle is what a release even is here, because the answer is unusual. Ordinary services deploy one artefact and a rollback restores the previous one. This feature has five versioned things that only make sense together. The prompt was written against a particular guardrail, evaluated against a particular retrieval corpus, and tuned for the behaviour of one model. Promote the prompt without the guardrail and production runs a combination nothing ever measured. Worse, the rollback is then incoherent: repointing the prompt alias returns the feature to a state that is half old and half new, which is a third configuration, also unmeasured. So a release has to be a manifest of resolved version identifiers, produced once from one commit, promoted as a unit, and kept after the promotion so it remains a target you can return to. Everything else in the design follows from that.

The second is that the gate on the release cannot be an assertion. CI/CD pipelines for ordinary software gate on tests that are true or false, and a red build is unambiguous. Generative output has no such assertion at the top layer. What you can measure is a score over a fixed set of examples, so the gate compares a number against a threshold. A run that lands at 0.82 against a floor of 0.85 fails. The same code passes tomorrow at 0.86. That makes the gate itself a versioned artefact with a dataset, a metric, a threshold and a rule for a near miss, which is why the golden dataset has to be pinned by the same commit that pins the prompt. It also makes the gate slow and expensive. A Bedrock evaluation job over a few hundred examples runs for a long time and bills tokens on every run, so where the gate sits in the stage order is a real design decision and not a detail to sort out later.

The third is that a stage can only act on what is declared. Anything created by clicking cannot be diffed, cannot be reviewed before it lands, cannot be scanned, and cannot be reproduced in a second account. Moving the prompt, the guardrail, the agent, the data source and the inference profile into infrastructure as code is the precondition for every other control in the pipeline, and it pays off most sharply on IAM. An agent that calls tools has a permission surface, and the useful moment to notice that a change widened an action-group role is while the change is still a proposed template, not after it has shipped.

The fourth is that promotion crosses an account boundary and rollback does not deploy anything. A stage running in the pipeline account has to act in staging and then in production, which means assuming a deploy role in each target account, scoped to the stacks and resources it manages. And rollback support here means repointing a set of aliases at the versions named in the previous manifest. Those versions still exist, so nothing has to be rebuilt, retrained or re-evaluated. A rollback that requires a build has already lost the argument about how fast it is.

What we’ll filter on

1. Release semantics: does the orchestrator model stages, ordering, artefact hand-off, approvals and failure out of the box, or do I write all of that myself?
2. Long non-deterministic gates: can a stage start an evaluation job that runs for an hour and fail the release on a threshold rather than an exit code?
3. Cross-account promotion: can a stage act in another account through a scoped role without static credentials?
4. Provenance: is there a durable record linking a commit to the exact prompt, guardrail, alias and model versions that went live?
5. Shared code distribution: can the model-client library be published, versioned and pinned so every team gets the same defaults?
6. Rollback support: how fast is it, and does it need a rebuild?

The landscape

The orchestrator is the real choice, and there are three credible ones.

AWS CodePipeline with AWS CodeBuild stages. CodePipeline models a release as ordered stages of actions with artefacts passed between them. It already understands what a release needs: source triggers, parallel actions, manual approval actions, and per-action IAM roles including roles in other accounts. AWS CodeBuild is where the work happens, running whatever container image and commands a stage needs, which covers synthesising templates, running unit tests, running security scans and calling the evaluation APIs. Deployment of the CloudFormation stacks is a native action type, so the deploy stage does not need bespoke scripting. AWS CodeDeploy handles the traffic-shifting side for the action-group Lambda functions, moving an alias from the old version to the new one in a canary or linear pattern with an automatic rollback on a CloudWatch alarm.

A Step Functions state machine as the release orchestrator. Step Functions is very good at exactly the part CodePipeline is weakest at: waiting. A state machine can start a Bedrock evaluation job, poll or wait on a task token for as long as it takes, branch on the resulting score, and fan out to run several evaluation dimensions in parallel. It also expresses conditional promotion cleanly. What it does not give you is release semantics. Stage ordering, artefact versioning, approvals and the notion of a pipeline execution are things you build in Amazon States Language, and you end up maintaining an orchestrator instead of using one.

An external CI runner calling AWS. Most teams already have a runner attached to their repository. GitHub Actions authenticating through GitHub OIDC into an IAM role removes the worst problem with that pattern, which is long-lived access keys sitting in a CI secret store. The runner assumes a role per environment, gets short-lived credentials, and calls AWS the same way any other client does. The trade is that the release record lives outside AWS, cross-account promotion is a set of role assumptions you script and audit yourself, and the runner is a second control plane your security review has to cover.

Underneath any of the three sits the infrastructure as code layer, and that choice is independent of the orchestrator. AWS CDK synthesises CloudFormation from application code, which suits this workload. The prompt text, the guardrail configuration, the agent action-group schemas and the data-source definition all want to live next to the code that reads them, assembled with loops and constructs rather than copied. Plain CloudFormation templates work equally well if the team prefers declarative source. Either way the deploy stage submits a change set and CloudFormation performs the change, which is what makes the promotion reviewable.

Two supporting pieces are worth naming. AWS CodeArtifact holds the shared model-client library, the internal package that wraps invocation with the agreed retry policy, timeout, guardrail identifier and logging fields. Publishing it from the build stage and pinning it by version in every consumer means a change to the retry defaults reaches every team through a version bump rather than through a message asking people to copy a snippet. The automated testing frameworks inside the stages come in two kinds. The deterministic layer takes the usual unit and contract test runners over the Lambda functions and the API. The non-deterministic layer takes an evaluation harness, either Bedrock evaluation jobs or a library run such as fmeval invoked from a CodeBuild stage.

Evaluation

Side by side

The two weaknesses in the top row and the middle row are complementary, which is what decides this. CodePipeline knows what a release is and does not want a stage sitting idle for an hour. Step Functions will happily sit idle for an hour and does not know what a release is. Combining them costs nothing: a CodeBuild action starts a state machine execution and waits for it, or the state machine calls back with a task token, and each service does the part it is built for. The external runner is a reasonable answer for an organisation that has standardised on it, but it puts the release record and the cross-account trust outside the accounts being deployed to, which is a harder story at review time.

The stage sequence

The solution

AWS CodePipeline is the spine, AWS CodeBuild does the work in each stage, a Step Functions state machine owns the long evaluation, and AWS CDK defines every resource the feature is made of. That combination gives continuous deployment and testing of GenAI components under a single release identity, which is the property the team was missing.

The release manifest

The source stage triggers on a commit to one repository holding all of it: the prompt text, the guardrail configuration, the agent action-group schemas, the knowledge base data-source definition, the Lambda source, the CDK app, the golden dataset reference, and the thresholds each metric has to clear. The build stage resolves that commit into a manifest, which is the durable answer to the provenance question:

{
  "release": "2026-08-18.7",
  "commit": "9f2c41a",
  "prompt_version": "12",
  "guardrail_version": "5",
  "agent_alias": "prod-2026-08-18-7",
  "inference_profile": "arn:aws:bedrock:ap-southeast-2:111122223333:application-inference-profile/support-assistant",
  "kb_data_source_sync": "ingestion-job-8841",
  "client_library": "1.9.3",
  "golden_dataset": "s3://evals/support/golden-v14.jsonl"
}

Every later stage reads the manifest instead of resolving anything itself, so staging and production are given identical inputs rather than trusted to look them up at the same moment. The manifest survives the pipeline execution, which is what makes rollback a lookup. This is the same set of pinned identifiers that versioned release artefacts require, now produced automatically rather than recorded by hand.

Build, and the shared library

The build stage runs CDK synth to produce templates, runs the unit and contract tests over the Lambda handlers, and publishes the model-client library to AWS CodeArtifact under a new version. The library is where the retry policy, the timeout, the default guardrail identifier and the structured logging fields live, so a team consuming it gets those behaviours without deciding them. Point the CodeArtifact repository upstream at the public registry so consumers resolve internal and external packages through one endpoint. Pin the internal version in every consumer, so an upgrade is a reviewed change rather than a surprise.

Security scans

Before anything deploys, a CodeBuild stage runs the security scans against what the build produced. Dependency scanning over the resolved lockfile, template scanning over the synthesised CloudFormation for public buckets, unencrypted stores and missing logging, a secrets check across the diff, and an IAM policy diff. That last one deserves the attention. The stage renders the policies the change set would create and runs IAM Access Analyzer policy validation over them. A change that widens an action-group Lambda role fails the stage instead of reaching a reviewer’s inbox after the fact. This is where declared resources pay for themselves: a console-clicked policy has nothing to scan.

Two gates, not one

Gating twice is a response to cost: automated quality gates for deployments have to be affordable enough to run on every commit and thorough enough to be worth trusting, and no single run is both. The dev stage deploys the stack and then scores a smoke set of roughly thirty examples, chosen to cover the answer shapes the assistant handles most and the two failure modes it has actually shipped. It runs in minutes, costs very little, and catches the broken prompt, the guardrail that now blocks every legitimate question, and the agent whose tool schema no longer parses. Failing there is cheap.

The staging stage runs the full golden set, and this is the gate with teeth. A CodeBuild action starts a Step Functions execution and waits. The state machine starts a Bedrock evaluation job or an fmeval run against the deployed staging endpoint, waits for completion, compares each metric against its threshold, and returns a pass or fail with the numbers attached. Two details matter in the wait. The knowledge base data-source sync is not transactional with the stack deployment, so the state machine has to poll the ingestion job to completion before evaluating; scoring against a half-synced index produces a failure that has nothing to do with the change. And a near miss needs a rule decided in advance, because a metric that sits one point under its floor will otherwise be argued about at four in the afternoon by whoever wants the release out.

Promotion and rollback

The production deploy is a cross-account action. The pipeline role assumes a deploy role in the production account that is scoped to the specific stacks and the specific Bedrock resources it manages, not an administrator role with a wildcard. Do this per environment, so a compromised pipeline execution reaches one account’s declared resources rather than the estate. What the deploy stage actually does when the release includes a customised model is a separate matter, handled by the same registry and endpoint mechanics as any other model promotion; the pipeline supplies the trigger and the approval record, not a second version of that process.

Traffic then shifts rather than switching. AWS CodeDeploy moves the action-group Lambda aliases in a canary pattern with CloudWatch alarms attached. Alongside it, the assistant runs the split described in running two variants against live traffic, so a share of requests exercise the new manifest under production conditions while the rest stay on the old one. Rollback is then the cheapest operation in the whole design: read the previous manifest, repoint the prompt alias, the guardrail version reference, the agent alias and the inference profile at the identifiers it names, and let CodeDeploy return the Lambda aliases. No build runs, no evaluation runs, and the state you land in is one that passed a gate.

Worked example

A guardrail change that lands

A commit tightens the guardrail’s denied-topics list and adjusts two lines of the system prompt to explain the new refusal. The build stage synthesises, tests, and publishes client library 1.9.4. Security scans pass; the IAM diff is empty. Dev deploys and the smoke set scores 0.91 on answer relevance against a floor of 0.88, in four minutes. Staging deploys through its cross-account role, the state machine waits eleven minutes for the ingestion job, then runs the full golden set: groundedness 0.93 against 0.90, refusal correctness 0.97 against 0.95, harmful-output rate zero. The approval action carries those numbers and the empty IAM diff. Production deploys, the canary runs ten percent of traffic for thirty minutes with no alarm, and the manifest for release 2026-08-18.7 goes into the record.

A prompt change that stops at the gate

A commit rewrites the retrieval instructions to make answers more concise. Smoke passes at 0.89, one point above the floor, which is the sort of margin that reads as fine. The full golden set disagrees: groundedness drops to 0.84 against a floor of 0.90, because the shorter answers stopped quoting the policy text they were drawing on. The state machine returns a failure with the per-example deltas, the staging stage goes red, and production never receives the change. Total cost is one full evaluation run and about twenty minutes. The alternative, which is what the team used to do, was a day of degraded answers and no way to attribute them.

A release that has to come back

A change passes both gates and then trips a CloudWatch alarm on tool-call error rate eight minutes into the canary. CodeDeploy returns the Lambda aliases automatically. An operator runs the rollback action. It reads the manifest for the previous release and repoints the prompt alias to version 11, the guardrail reference to version 4, the agent alias to the previous one, and the inference profile to the ARN that manifest names. Four minutes, no build, no evaluation, and the running configuration is one that has a passing gate record behind it.

What’s worth remembering

1. A generative release is a manifest of resolved versions, not a binary: prompt, guardrail, agent alias, data-source sync and inference profile promote together, and a rollback that returns only some of them lands on a configuration nothing has measured.
2. The gate in automated deployment pipelines for this workload compares a score against a threshold rather than checking a passing assertion, so the golden dataset, the metric and the floor are versioned by the same commit as the code.
3. Run a cheap smoke evaluation in dev and the full golden set in staging, because evaluation jobs are slow and bill tokens on every run, and most broken changes die on thirty examples.
4. Security scans belong before the first deploy and want declared resources to work on: dependency and template scanning, a secrets check, and an IAM policy diff that fails the run when an action-group role widens.
5. Cross-account promotion is a scoped deploy role per environment assumed by the pipeline, not a wildcard administrator, and an external runner needs OIDC federation rather than stored access keys.
6. Rollback support means repointing aliases at the previous manifest in minutes with no rebuild, which only works if you kept the manifest and the old versions still exist.