Building a Deployment Pipeline for a GenAI Feature

The situation

A support assistant runs on Amazon Bedrock across three AWS accounts: dev, staging, production. The feature is five moving pieces. A published prompt version. A published guardrail version. A Bedrock agent alias with two action-group Lambda functions behind it. A knowledge base whose data source syncs from an S3 prefix. An inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. that the application invokes by ARN. The agent predates 30 July 2026, when Amazon Bedrock Agents became Agents Classic and closed to accounts with no prior use of it; existing agents and aliases keep running, and a team building this today would put the agent on Amazon Bedrock AgentCore instead.

Every one of those pieces changes by clicking. Someone edits the prompt in the console, publishes a version, and changes what the application points at. Someone else tightens a guardrail filter and publishes that. The Lambda functions go out through a script on a developer laptop with production credentials in the shell. Production changed twice last Thursday afternoon and the change record is a thread in a chat channel.

Three weeks ago answer quality dropped for most of a day. Nobody could reconstruct which of the five pieces had moved, because three of them had moved that week and none of them left anything tying a change to a commit. The team wants what production readiness implies and they never built: one commit produces one release, the release has to prove itself before it reaches customers, and a bad release comes back out in minutes rather than in an afternoon of console archaeology.

What actually matters

The first thing to settle is what a release even is here, because the answer is unusual. Ordinary services deploy one artefact and a rollback restores the previous one. This feature has five versioned things that only make sense together. The prompt was written against a particular guardrail, evaluated against a particular retrieval corpus, and tuned for the behaviour of one model. Promote the prompt without the guardrail and production runs a combination nothing ever measured. Worse, the rollback is then incoherent: pointing the application back at the old prompt version returns a state that is half old and half new, a third configuration, also unmeasured. So a release has to be a manifest of resolved version identifiers, produced once from one commit, promoted as a unit, and kept after the promotion so it remains a target you can return to. Everything else in the design follows from that.

The second is that the gate on the release cannot be an assertion. CI/CD pipelines for ordinary software gate on tests that are true or false, and a red build is unambiguous. Generative output has no such assertion at the top layer. What you can measure is a score over a fixed set of examples, so the gate compares a number against a threshold. A run that lands at 0.82 against a floor of 0.85 fails. The same code passes tomorrow at 0.86. That makes the gate itself a versioned artefact with a dataset, a metric, a threshold and a rule for a near miss, which is why the golden dataset has to be pinned by the same commit that pins the prompt. It also makes the gate slow and expensive. A Bedrock evaluation job over a few hundred examples runs for a long time and bills tokens on every run, so where the gate sits in the stage order is a real design decision and not a detail to sort out later.

The third is that a stage can only act on what is declared. Anything created by clicking cannot be diffed, cannot be reviewed before it lands, cannot be scanned, and cannot be reproduced in a second account. Moving the prompt, the guardrail, the agent, the data source and the inference profile into infrastructure as code is the precondition for every other control in the pipeline, and IAM is where that shows most sharply. An agent that calls tools has a permission surface, and the useful moment to notice that a change widened an action-group role is while the change is still a proposed template, not after it has shipped.

The fourth is that promotion crosses an account boundary and rollback does not deploy anything. A stage running in the pipeline account has to act in staging and then in production, which means assuming a deploy role in each target account, scoped to the stacks and resources it manages. And rollback support here means pointing each reference back at the identifiers the previous manifest names. Those versions still exist, so nothing has to be rebuilt, retrained or re-evaluated. A rollback that triggers a build takes as long as a deploy, which defeats it.

What we’ll filter on

1. Release semantics: does the orchestrator model stages, ordering, artefact hand-off, approvals and failure out of the box, or do I write all of that myself?
2. Long non-deterministic gates: can a stage start an evaluation job that runs for an hour and fail the release on a threshold rather than an exit code?
3. Cross-account promotion: can a stage act in another account through a scoped role without static credentials?
4. Provenance: is there a durable record linking a commit to the exact prompt, guardrail, alias and model versions that went live?
5. Shared code distribution: can the model-client library be published, versioned and pinned so every team gets the same defaults?
6. Rollback support: how fast is it, and does it need a rebuild?

The landscape

The orchestrator is the real choice, and there are three credible ones.

AWS CodePipeline with AWS CodeBuild stages. CodePipeline models a release as ordered stages of actions with artefacts passed between them, and it already covers what a release needs: source triggers, parallel actions, manual approval actions that hold for seven days by default, and a per-action role ARN that can point into another account. AWS CodeBuild is where the work happens, running whatever container image and commands a stage needs, which covers synthesising templates, running unit tests, running security scans and calling the evaluation APIs. A build action runs for up to 36 hours, so a slow evaluation does not need a separate waiting service. Deployment of the CloudFormation stacks is a native action type, so the deploy stage does not need bespoke scripting. AWS CodeDeploy handles the traffic-shifting side for the action-group Lambda functions, moving an alias from the old version to the new one in a canary or linear pattern with an automatic rollback on a CloudWatch alarm.

A Step Functions state machine as the release orchestrator. A state machine starts a Bedrock evaluation job, polls it or waits on a task token, branches on the resulting score, and fans out to run several evaluation dimensions in parallel. That branching and fan-out is what a shell script in a build action would have to reimplement. What Step Functions does not supply is release semantics. Stage ordering, artefact versioning, approvals and the notion of a pipeline execution are all things you write in Amazon States Language, which leaves you maintaining an orchestrator instead of using one.

An external CI runner calling AWS. Most teams already have a runner attached to their repository. GitHub Actions authenticating through GitHub OIDC into an IAM role removes the worst problem with that pattern, which is long-lived access keys sitting in a CI secret store. The runner assumes a role per environment, gets short-lived credentials, and calls AWS the same way any other client does. The trade is that the release record lives outside AWS, cross-account promotion is a set of role assumptions you script and audit yourself, and the runner is a second control plane your security review has to cover.

Underneath any of the three sits the infrastructure as code layer, and that choice is independent of the orchestrator. AWS CDK synthesises CloudFormation from application code, which suits this workload. The prompt text, the guardrail configuration, the agent action-group schemas and the data-source definition all belong next to the code that reads them, assembled with loops and constructs rather than copied. Plain CloudFormation templates work equally well if the team prefers declarative source. Either way the deploy stage submits a change set and CloudFormation performs the change, which is what makes the promotion reviewable.

Two supporting pieces are worth naming. AWS CodeArtifact holds the shared model-client library, the internal package that wraps invocation with the agreed retry policy, timeout, guardrail identifier and logging fields. Publishing it from the build stage and pinning it by version in every consumer means a change to the retry defaults reaches every team through a version bump rather than through a message asking people to copy a snippet. The automated testing frameworks inside the stages come in two kinds. The deterministic layer takes the usual unit and contract test runners over the Lambda functions and the API. The non-deterministic layer takes an evaluation harness, either Bedrock evaluation jobs or a library run such as fmeval invoked from a CodeBuild stage.

Evaluation

Side by side

The gaps in the top row and the middle row line up, which is what decides this. CodePipeline supplies release semantics and no branching; Step Functions supplies branching and no release semantics. The hour itself is not the constraint for either: a build action runs up to 36 hours, and CodePipeline’s own Step Functions invoke action polls a standard state machine to a terminal status with a seven-day default timeout. So the pipeline keeps the stages and hands the evaluation to a state machine that scores several metrics in parallel and returns one verdict. The external runner is a reasonable answer for an organisation that has standardised on it, but it puts the release record and the cross-account trust outside the accounts being deployed to, which is a harder story at review time.

The stage sequence

The solution

AWS CodePipeline is the spine, AWS CodeBuild does the work in each stage, a Step Functions state machine owns the long evaluation, and AWS CDK defines every resource the feature is made of. That combination gives continuous deployment and testing of GenAI components under a single release identity, which is the property the team was missing.

The release manifest

The source stage triggers on a commit to one repository holding all of it: the prompt text, the guardrail configuration, the agent action-group schemas, the knowledge base data-source definition, the Lambda source, the CDK app, the golden dataset reference, and the thresholds each metric has to clear. The build stage resolves that commit into a manifest, which is the durable answer to the provenance question:

{
  "release": "2026-08-18.7",
  "commit": "9f2c41a",
  "prompt_version": "7",
  "guardrail_version": "5",
  "agent_alias": "prod-2026-08-18-7",
  "inference_profile": "arn:aws:bedrock:ap-southeast-2:111122223333:application-inference-profile/tq8mz41v7kd3",
  "kb_ingestion_job": "T7KQ2XM9PA",
  "client_library": "1.9.3",
  "golden_dataset": "s3://evals/support/golden-v14.jsonl"
}

Every later stage reads the manifest instead of resolving anything itself, so staging and production are given identical inputs rather than trusted to look them up at the same moment. The manifest survives the pipeline execution, which is what makes rollback a lookup. It also outlives the version history it names: Prompt management keeps ten versions per prompt by default, so an old release stays reachable only while something has written down what it was made of. This is the same set of pinned identifiers that versioned release artefacts require, now produced automatically rather than recorded by hand.

Build, and the shared library

The build stage runs CDK synth to produce templates, runs the unit and contract tests over the Lambda handlers, and publishes the model-client library to AWS CodeArtifact under a new version. The library is where the retry policy, the timeout, the default guardrail identifier and the structured logging fields live, so a team consuming it gets those behaviours without deciding them. Add the matching -store repository as an upstream so consumers resolve internal and public packages through one endpoint; a repository carries one external connection, so the store pattern is how the rest of the domain reaches npm or PyPI. Pin the internal version in every consumer, so an upgrade is a reviewed change rather than a surprise.

Security scans

Before anything deploys, a CodeBuild stage runs the security scans against what the build produced. Dependency scanning over the resolved lockfile, template scanning over the synthesised CloudFormation for public buckets, unencrypted stores and missing logging, a secrets check across the diff, and an IAM policy diff. That last one deserves the attention. The stage renders the policies the change set would create and runs them through IAM Access Analyzer. Policy validation reports grammar errors and overly permissive statements; the custom policy check CheckNoNewAccess is the one that compares the new policy against the old, so a change that widens an action-group Lambda role fails the stage instead of reaching a reviewer’s inbox after the fact. Declared resources are what make that possible: a console-clicked policy has nothing to scan.

Two gates, not one

Gating twice is a response to the token bill: an automated quality gate has to be quick enough to run on every commit and thorough enough to be worth trusting, and no single run is both. The dev stage deploys the stack and then scores a smoke set of roughly thirty examples, chosen to cover the answer shapes the assistant handles most and the two failure modes it has actually shipped. It runs in minutes and catches the broken prompt, the guardrail that now blocks every legitimate question, and the agent whose tool schema no longer parses. Failing there takes four minutes.

The staging stage runs the full golden set, and that is the gate that stops releases. A Step Functions invoke action starts the execution and polls it to a terminal status. The state machine starts a Bedrock evaluation job or an fmeval run against the deployed staging endpoint, waits for completion, compares each metric against its threshold, and returns a pass or fail with the numbers attached. A custom prompt dataset holds up to 1,000 prompts, which sets the ceiling on how large a golden set one job can score. Two details matter in the wait. The knowledge base data-source sync is not transactional with the stack deployment, so the state machine has to poll the ingestion job to completion before evaluating; scoring against a half-synced index produces a failure that has nothing to do with the change, and a knowledge base runs one ingestion job at a time, so two executions cannot sync in parallel. And a near miss needs a rule decided in advance, because a metric that sits one point under its floor will otherwise be argued about at four in the afternoon by whoever wants the release out.

Promotion and rollback

The production deploy is a cross-account action. The pipeline role assumes a deploy role in the production account that is scoped to the specific stacks and the specific Bedrock resources it manages, not an administrator role with a wildcard. Do this per environment, so a compromised pipeline execution reaches one account’s declared resources rather than the estate. The artefact bucket needs a customer managed KMS key shared with each target account as well, because the default pipeline key does not cross an account boundary. What the deploy stage actually does when the release includes a customised model is a separate matter, handled by the same registry and endpoint mechanics as any other model promotion; the pipeline supplies the trigger and the approval record, not a second version of that process.

Traffic then shifts rather than switching. AWS CodeDeploy moves the action-group Lambda aliases in a canary pattern with CloudWatch alarms attached. Alongside it, the assistant runs the split described in running two variants against live traffic, so a share of requests exercise the new manifest under production conditions while the rest stay on the old one. Rollback is then the shortest operation in the whole design: read the previous manifest, point the prompt version reference, the guardrail version reference, the agent alias and the inference profile at the identifiers it names, and let CodeDeploy redeploy the previous Lambda revision. No build runs, no evaluation runs, and the state you land in is one that passed a gate.

Worked example

A guardrail change that lands

A commit tightens the guardrail’s denied-topics list and adjusts two lines of the system prompt to explain the new refusal. The build stage synthesises, tests, and publishes client library 1.9.3. Security scans pass; the IAM diff is empty. Dev deploys and the smoke set scores 0.91 on answer relevance against a floor of 0.88, in four minutes. Staging deploys through its cross-account role, the state machine waits eleven minutes for the ingestion job, then runs the full golden set: groundedness 0.93 against 0.90, refusal correctness 0.97 against 0.95, harmful-output rate zero. The approval action carries those numbers and the empty IAM diff. Production deploys, the canary runs ten percent of traffic for thirty minutes with no alarm, and the manifest for release 2026-08-18.7 goes into the record.

A prompt change that stops at the gate

A commit rewrites the retrieval instructions to make answers more concise. Smoke passes at 0.89, one point above the floor, which is the sort of margin that reads as fine. The full golden set scores it differently: groundedness drops to 0.84 against a floor of 0.90, because the shorter answers stopped quoting the policy text they were drawing on. The state machine returns a failure with the per-example deltas, the staging stage goes red, and production never receives the change. That took one full evaluation run and about twenty minutes. The alternative, which is what the team used to do, was a day of degraded answers and no way to attribute them.

A release that has to come back

A change passes both gates and then trips a CloudWatch alarm on tool-call error rate eight minutes into the canary. CodeDeploy rolls back automatically, redeploying the previous Lambda revision. An operator runs the rollback action. It reads the manifest for the previous release and points the application at prompt version 6, guardrail version 4, the previous agent alias, and the inference profile ARN that manifest names. Four minutes, no build, no evaluation, and the running configuration is one that has a passing gate record behind it.

What’s worth remembering

1. A generative release is a manifest of resolved versions, not a binary: prompt, guardrail, agent alias, data-source sync and inference profile promote together, and a rollback that returns only some of them lands on a configuration nothing has measured.
2. The gate in automated deployment pipelines for this workload compares a score against a threshold rather than checking a passing assertion, so the golden dataset, the metric and the floor are versioned by the same commit as the code.
3. Run a short smoke evaluation in dev and the full golden set in staging, because evaluation jobs are slow and bill tokens on every run, and most broken changes die on thirty examples.
4. Security scans belong before the first deploy and need declared resources to work on: dependency and template scanning, a secrets check, and an Access Analyzer check for new access that fails the run when an action-group role widens.
5. Cross-account promotion is a scoped deploy role per environment assumed by the pipeline, not a wildcard administrator, and an external runner needs OIDC federation rather than stored access keys.
6. Rollback support means pointing every reference back at the previous manifest in minutes with no rebuild, which only works if you kept the manifest and the old versions still exist.