How to Build Multi-Account Multi-Region Pipelines With CDK Pipelines

The situation

A platform team runs a payments microservice in three stages: dev in account 111111111111 (us-east-1), staging in 222222222222 (us-east-1), prod in 333333333333 deployed to both us-east-1 and eu-west-1. The pipeline lives in a central tools account 999999999999 in us-east-1, the CI/CD account pattern where a single account owns all pipelines and deploys outwards.

The current setup is CodePipeline defined in a hand-written CloudFormation template: CodeCommit source, CodeBuild running npm test && npm run build, then five CLOUDFORMATION_CHANGE_SET_REPLACE actions, one per target environment, each with a hand-authored RoleArn pointing at a deployment role in the target account. A ManualApprovalAction between staging and prod.

It works. The problems surface when anyone tries to change it: adding a fourth stage means edits across three repos and three cross-account IAM roles; the pipeline has no idea what it’s deploying beyond “an S3 artefact and a parameter file”; mapping a pipeline execution to a git SHA and a target region takes three console tabs; and updating the pipeline itself requires a separate cloudformation deploy from a workstation, which means the local definition constantly drifts from the live one.

The team wants the pipeline’s shape to live in the same repo as the microservice and evolve with it; stages parameterised on account and region so adding a region is one argument; self-mutation so a pipeline change propagates on the next run; manual approval gates; and no parallel control plane.

What actually matters

Before reaching for a service, it’s worth being explicit about what “pipeline as code” has to give us that the current setup doesn’t.

The first thing is the pipeline is an artefact, not a deployment. Today the pipeline template lives in a repo but its deploy happens out-of-band, from a workstation, when someone remembers. That’s not “pipeline as code”; that’s “a YAML file with a manual deploy step.” We want the pipeline to redeploy itself from source on every run, so the live definition is always the one in main. Anything less and we’re back to “someone meant to update the pipeline before they left.”

The second is first-class account and region awareness. Cross-account CloudFormation actions work today but every role ARN is copy-pasted prose in a template. Adding prod’s third region means writing the ARN again, testing the trust relationship, remembering to update the IAM role’s trust policy in the target account. We want a definition where “this stage runs in account X, region Y” is a parameter on a function call, and everything that follows, roles, trust, asset publishing, is generated.

The third is blast radius on a pipeline-definition change. A bad template can break deployment for everyone. We want the pipeline to fail the definition-change step before any application stage runs, so a malformed shape never stops the real work. The self-mutation step runs first for a reason: if it fails, the previous pipeline shape stays live and the application stages simply don’t execute that run. Fix the definition, push, the next run recovers.

The fourth is observability of “what’s where”. Today, tracing a commit through five stages means matching pipeline execution IDs against git SHAs against CloudFormation changeset IDs. Native integration where the source metadata travels with the execution collapses this to a single glance per stage. The mapping doesn’t need new tooling; it needs the pipeline to know it’s the pipeline.

The fifth is cost shape. CodePipeline is per-active-pipeline-per-month plus CodeBuild minutes. CDK Pipelines sits on top of those primitives, there is no new service bill, just the same underlying primitives wrapped in generated templates. Nothing to forecast that wasn’t already being forecast.

The sixth is not adopting a parallel control plane just to get pipeline-as-code. AWS ships several higher-level answers (Proton, CodeCatalyst) that would give most of what we want, but each comes with its own console, its own permissions model, and its own upgrade cadence. For one microservice’s pipeline, that trade is expensive. The right abstraction is one that generates CodePipeline under the hood and leaves us to run CodeBuild, CloudFormation, and IAM, which we already do.

What we’ll filter on

1. Declarative in code, pipeline shape lives as source, not as console state or verbose hand-written YAML.
2. Account and region awareness, stages name their target { account, region }; cross-account roles are generated.
3. Self-mutation, pipeline-definition change on main propagates to the live pipeline before application stages run.
4. Manual approval gates, named approval between stages or around waves.
5. Low operational overhead, serverless primitives under a thin abstraction, not a new managed service.

The pipeline-as-code landscape

AWS has four answers for declarative deployment pipelines.

CodePipeline with hand-written CloudFormation. The status quo. The pipeline is a YAML template enumerating stages, actions, and deployment roles. Declarative if you count 600 lines as declarative. Multi-account is possible but every RoleArn is hand-authored; target-account roles are separate templates deployed separately. No self-mutation, a pipeline change needs a separate deploy. Manual approval via ManualApprovalAction.

CDK Pipelines (aws-cdk-lib/pipelines). A CDK construct library that builds a CodePipeline from a higher-level CodePipeline class. Inputs are a Source (CodeCommit, GitHub via CodeStar Connections, S3) and one or more Stage instances, each targeting a specific { account, region }. CDK generates pipeline stages, cross-account deployment roles, asset-publishing infrastructure, and an UpdatePipeline stage that reruns cdk deploy on the pipeline stack before application stages. Ticks everything.

AWS Proton. A managed service for platform-team-curated templates. Environment templates and service templates; developers pick a template and Proton provisions it, pipeline included. Declarative, multi-account-capable, managed versioning. But the abstraction is environments and services from a catalogue, not “deploy this microservice through these stages.” Standing up Proton as a second control plane for one microservice is disproportionate.

AWS CodeCatalyst. A newer all-in-one developer service: source, issues, workflows (YAML pipelines), managed dev environments, blueprints. Declarative with account connections for multi-account deploys. But CodeCatalyst lives in its own console and wants to own source, issues, and dev environments too. Adopting it for the pipeline alone is a heavier change than adopting CDK Pipelines.

Side by side

Matching the wishlist to an abstraction

CDK Pipelines in depth

The construct library is at aws-cdk-lib/pipelines in CDK v2; the modern class is CodePipeline. (An older CdkPipeline class is deprecated.) The pipeline has a fixed top-level shape.

Source. A CodePipelineSource – codeCommit(...), connection(...) for GitHub/Bitbucket via CodeStar Connections, or s3(...). Produces a FileSet that flows to synth.

Synth. A ShellStep (or CodeBuildStep for build-environment customisation) that runs cdk synth and outputs cdk.out. For Python, the conventional commands are pip install -r requirements.txt and cdk synth.

UpdatePipeline. A CDK-generated stage that runs cdk deploy <PipelineStack> before any application stage executes. Each run rebuilds the pipeline itself first; if the definition in source has changed, the new shape takes effect on this run. This is the self-mutation loop.

Assets. An auto-generated stage that publishes CDK assets. Docker images to ECR, Lambda bundles and file assets to S3, using the bootstrap stack’s publishing roles per target account and region.

Stages. addStage(stage) adds an application stage. A CDK Stage is a group of Stacks sharing a { account, region } environment. Passing a stage generates a CodePipeline stage that deploys every stack via the bootstrap stack’s DeploymentActionRole and CloudFormationExecutionRole.

Waves. addWave(name) groups stages that run in parallel. Two prod regions sit in one Wave and deploy concurrently.

Approvals and steps. addStage(..., { pre: [...], post: [...] }) inserts Steps around a stage. ManualApprovalStep('PromoteToProd') is the approval gate; ShellStep or CodeBuildStep is a test runner.

The Python for this pipeline is shorter than the CloudFormation it replaces:

pipeline = pipelines.CodePipeline(
    self, "Pipeline",
    synth=pipelines.ShellStep(
        "Synth",
        input=pipelines.CodePipelineSource.code_commit(repo, "main"),
        commands=["pip install -r requirements.txt", "cdk synth"],
    ),
    cross_account_keys=True,
)

pipeline.add_stage(AppStage(
    self, "Dev",
    env=Environment(account="111111111111", region="us-east-1"),
))

pipeline.add_stage(AppStage(
    self, "Staging",
    env=Environment(account="222222222222", region="us-east-1"),
))

prod_wave = pipeline.add_wave(
    "Prod",
    pre=[pipelines.ManualApprovalStep("PromoteToProd")],
)

prod_wave.add_stage(AppStage(
    self, "ProdUsEast1",
    env=Environment(account="333333333333", region="us-east-1"),
))

prod_wave.add_stage(AppStage(
    self, "ProdEuWest1",
    env=Environment(account="333333333333", region="eu-west-1"),
))

Adding a third prod region is one line. Adding a pre-deploy integration test on staging is one line. Adding a post-deploy smoke test on each prod region is one line in each stage’s post.

Bootstrapping, and why cross-account works

The pipeline account and every target account-and-region pair must be bootstrapped with the CDK modern bootstrap stack (CDKToolkit). Bootstrapping creates:

• DeploymentActionRole, assumed by the pipeline to execute the deployment.
• CloudFormationExecutionRole, passed to CloudFormation as execution role for each stack.
• FilePublishingRole and ImagePublishingRole, assumed by the Assets stage to publish to S3 and ECR.
• LookupRole, used during cdk synth for context lookups.
• A staging S3 bucket and ECR repository in each bootstrapped region.

Every target account is bootstrapped with --trust 999999999999 so the pipeline account can assume the deployment role. --cloudformation-execution-policies grants the execution role its power, typically arn:aws:iam::aws:policy/AdministratorAccess as a default, narrowed in regulated environments.

One bootstrap per account-and-region pair: account 333 in us-east-1 is a separate bootstrap from account 333 in eu-west-1. “Adding a region” means both “add a stage in CDK” and “run cdk bootstrap aws://333333333333/eu-west-1 before the first deploy.”

A worked pipeline run

A developer merges a commit that changes a Lambda handler and adds ProdApSoutheast2 as a third prod stage.

T+0. CodeCommit push on main. Source action picks up the commit; source metadata includes commit ID, author, message, visible in the console on every subsequent transition.

T+~30s. Synth runs in CodeBuild. cdk.out produced with the CloudAssembly, generated CloudFormation for every stage, a manifest describing which stack goes to which environment, and asset manifests.

T+~3min. UpdatePipeline runs. CodeBuild executes cdk -a . deploy PipelineStack. The generated template now includes ProdApSoutheast2. CloudFormation applies the diff against the live pipeline. The execution then continues on the updated pipeline, this is the self-mutation.

T+~6min. Assets stage publishes any new Lambda bundles, Docker images, or file assets to the cdk-<qualifier>-assets-<account>-<region> buckets and ECR repositories. For the new ap-southeast-2 region, CDK needs that region’s bootstrap stack in place, if it isn’t, this stage fails with “cannot find bootstrap stack.” That failure is recoverable: bootstrap the region, retry.

T+~9min. Dev stage deploys: one CloudFormation CHANGE_SET_REPLACE in 111/us-east-1, using the deployment role assumed cross-account from the pipeline account.

T+~14min. Staging stage deploys in 222/us-east-1.

T+~18min. ManualApprovalStep pauses the pipeline. Console shows the commit, message, and queued changesets. A team member approves.

T+~18min +human. Prod wave starts. Three stages run in parallel, us-east-1, eu-west-1, ap-southeast-2, each executing its own changeset against 333 in its region.

T+~23min. Wave completes. “Commit abc1234 deployed to prod across three regions.”

Self-mutation, in detail

Instantiating CodePipeline causes CDK to inject UpdatePipeline when the pipeline is built. That stage runs cdk deploy on the pipeline stack itself, using the CloudAssembly produced by the current run’s Synth. Whatever cdk synth just produced for the pipeline stack becomes the live pipeline before application stages run.

You cannot permanently break the pipeline by committing a bad definition, because UpdatePipeline runs before any application deploy. A broken definition fails UpdatePipeline, application stages don’t run, and the previous shape remains live. Fix, push, the next run self-mutates back.

One edge case: changes to the Source or Synth steps themselves apply on the run after the one that merges them. The current run is already committed to its source and synth configuration; UpdatePipeline updates the definition, but the current run’s source and synth have already happened. Land source/synth changes in a separate commit whose only job is to update the pipeline shape.

When the hand-written CodePipeline is still right

CDK Pipelines has one prerequisite: the stacks deployed via the pipeline are CDK stacks. A team that hand-authors CloudFormation and has no interest in adopting CDK will find CDK Pipelines awkward, synth has nothing to synth. Technically it works via CfnInclude, but the value is diluted.

CDK Pipelines is also firmly CodePipeline-backed. Teams standardised on GitHub Actions or GitLab CI will find it hard to integrate; run cdk deploy from the external CI system and accept that self-mutation is not on offer.

What’s worth remembering

1. aws-cdk-lib/pipelines is the CDK v2 module; CodePipeline is the class. The older CdkPipeline is deprecated.
2. A CDK Stage is a group of Stacks that share an environment ({ account, region }). Passing a stage to addStage targets a specific account and region.
3. Waves deploy stages in parallel; stages inside a wave run concurrently.
4. The pipeline has a fixed top-level shape: Source, Synth, UpdatePipeline, Assets, then application stages and waves.
5. Self-mutation runs the pipeline stack’s cdk deploy before application stages. A pipeline-definition change in source takes effect on the same run.
6. ManualApprovalStep is the approval gate; use it in pre: on a stage or wave. ShellStep and CodeBuildStep are for test runners.
7. Every target account and region must be bootstrapped with the modern bootstrap stack. --trust <pipeline-account> grants the pipeline account permission to assume the deployment role.
8. Asset publishing uses the bootstrap stack’s publishing roles per account and region; assets are published once per run.
9. Cross-account deployment uses the bootstrap stack’s DeploymentActionRole and CloudFormationExecutionRole. CDK adds the cross-account role ARNs to the generated pipeline.
10. Source/synth changes take effect on the run after the one that commits them; everything else takes effect on the same run via UpdatePipeline.
11. CodePipeline’s source metadata tracks the git commit per execution, “which commit is in which stage” is native.