The situation
A logistics company deploys a containerised application through four environments: development, staging, pre-production and production, each in its own AWS account. The pipeline runs docker build in each account, from the same commit, with environment-specific build arguments baking in the API endpoints and the log level.
Two incidents have come from this. A base image tag moved between the staging build and the production build, so production ran a different operating system layer than anything that had been tested. And a transitive dependency published a new patch version between builds, which staging never saw.
The team knows the answer is to build once. What has stopped them is that the environment-specific values are currently baked in at build time, and nobody has decided where they should live instead. There is also an unresolved argument about whether production should pull an image from a registry in a different account.
What actually matters
The first thing that matters is that the thing being tested has to be the thing being shipped, byte for byte. Any step between the test and the deployment that can produce a different artefact makes the test a statement about something else. Rebuilding is the obvious such step and it is not the only one: a re-tag, a re-push, or a rebuild of a base layer all break the identity.
The second is that configuration baked at build time is what forces the rebuild. Moving it to runtime is the change that makes promotion possible, and it is the real work, since it means deciding for each value whether it is configuration, a secret, or genuinely part of the artefact.
The third is that promotion across accounts is an identity and permissions problem more than a pipeline one. Production pulling an image built in a shared account needs a repository policy, an execution role with the right permissions, and, where images are encrypted, a KMS key policy allowing the production account to decrypt. That last one is the step teams miss.
The fourth is that the artefact needs to be identifiable and immutable. A mutable tag reintroduces the problem: :latest in production is not a promotion, it is a rebuild with extra steps. Immutable tags or digest references are what make “the same bytes” verifiable.
Underneath it, provenance matters once one artefact travels through four environments. Being able to say which commit, which build, and which scan result produced this digest is what makes an approval meaningful.
What we’ll filter on
- Can this step change the artefact’s bytes?
- Is the value configuration, a secret, or part of the artefact?
- How is the artefact referenced: mutable tag, immutable tag, or digest?
- What permissions does the consuming account need, including on the encryption key?
- Where does the approval sit, and what is it approving?
- Can we prove which source produced this artefact?
The landscape
Build once, promote the artefact. The pattern: one build produces one artefact, and each environment deploys that artefact with its own configuration supplied at runtime. Everything else on this list is a mechanism for making it work.
Amazon ECR with a shared registry. A repository in a shared services account, with a repository policy granting pull to the workload accounts. Cross-account pull needs the repository policy and the consuming task execution role’s permissions, and where the repository uses a customer-managed KMS key, the key policy must allow the consuming account to decrypt.
ECR replication. Automatic cross-region and cross-account replication of images, so each account has a local copy. It removes the cross-account pull path at the cost of storing copies, and it is worth considering when a region’s availability during a deployment matters.
Immutable tags. An ECR repository setting preventing a tag from being overwritten. This is what stops v1.4.2 meaning something different next week, and it is a one-field change that removes a whole class of confusion.
Digest references. Referencing an image by sha256: digest rather than by tag. The strongest form of identity, and the one to use in the deployment manifests, since a digest cannot be moved.
Runtime configuration: Parameter Store and AppConfig. Where the environment-specific values go instead of build arguments. Parameter Store holds the values per environment; AppConfig adds validation, deployment strategies and rollback for configuration changes, which matters when a bad configuration value can break production as thoroughly as a bad build.
Secrets Manager. For the values that are secrets rather than configuration, with rotation. The distinction matters because a secret in Parameter Store as a plain string is a finding, and a configuration value in Secrets Manager is paying for rotation nobody needs.
Task definitions and environment injection. ECS task definitions reference secrets and parameters by ARN, so the container receives values at start-up without them being in the image or the definition. The task definition then differs per environment while the image does not.
CodePipeline across accounts. A pipeline in a tooling account deploying into workload accounts, using cross-account roles. The artefact bucket needs a KMS key shared with the target accounts, which is the same encryption-permissions issue as ECR and the one that produces most of the “access denied” tickets.
Signing and provenance. Signing images so a deployment can verify what it is running, and recording the build’s inputs. This is what makes an approval in pre-production mean something in production.
Evaluation
Side by side
| Approach | Same bytes everywhere | Config source | Cross-account complexity | Failure mode |
|---|---|---|---|---|
| Rebuild per environment | ✗ | Build args | None | Ships something untested |
| Build once, mutable tag | Partly | Runtime | Repository policy | Tag moves underneath you |
| Build once, immutable tag | ✓ | Runtime | Repository policy, KMS | Tag typo deploys the wrong version |
| Build once, digest reference | ✓ verifiable | Runtime | Repository policy, KMS | Digest is unreadable to humans |
| Build once, ECR replication | ✓ | Runtime | Replication config, KMS per region | Copies to keep in step |
The rows differ mostly in how the artefact is named, and the naming is what determines whether “the same bytes” is a claim or a fact. The digest row is the one that makes it verifiable, and the readability objection is answered by keeping the tag for humans and the digest for the deployment.
The solution
Build once in the tooling account, sort the baked-in values into configuration and secrets, promote by digest, and get the KMS permissions right before anything else.
Start by classifying what is currently baked in. API endpoints and log level are configuration and belong in Parameter Store per environment. Anything credential-shaped belongs in Secrets Manager. Anything genuinely invariant across environments stays in the image. This classification is the work that unblocks everything, and it is usually a shorter list than the team expects once each value is examined.
Change the task definitions to inject those values at start-up by ARN, so the definition differs per environment and the image does not. Where a configuration change can break the service, put it behind AppConfig rather than Parameter Store, so a bad value rolls back on an alarm the way a bad deployment does.
Move the build into the tooling account, producing one image pushed to a shared ECR repository with immutable tags enabled. Tag it with the commit SHA for humans and record the digest, which is what the deployments will actually reference.
Sort the cross-account permissions in one pass, because they are the source of most of the friction. The repository policy grants pull to each workload account; each task execution role gets ecr:GetAuthorizationToken and the pull actions; and if the repository or the pipeline’s artefact bucket uses a customer-managed KMS key, the key policy grants decrypt to the workload accounts. That last item is the one that produces an “access denied” nobody can explain, because the ECR permissions look correct.
Then make promotion a deployment of a digest rather than a build. The pipeline’s staging stage deploys the digest, the tests run against it, and the pre-production and production stages deploy the same digest with a manual approval between. Nothing is rebuilt and nothing is re-tagged, so the approval is approving an artefact rather than a commit.
Add scanning once, at push, rather than per environment. ECR enhanced scanning with Inspector reports findings against the image, and the pipeline gates on the result of that single scan. Scanning the same digest four times produces four identical reports.
Finally, record provenance. Which commit, which build, which scan, which digest, kept somewhere queryable, so an approval and an incident investigation can both answer “what is actually running” without archaeology.
Why not keep rebuilding but pin every dependency. Pinning helps and does not make the build deterministic, since base image layers, build tooling and the network all still vary. It reduces the chance of the incident without removing the class.
Why not replicate the image into each account so nothing is cross-account. It works and it is more moving parts, and it does not remove the KMS question, it multiplies it per region. Worth it where deployment must not depend on another account’s availability, and not as the default.
Worked example
The classification finds eleven baked-in values. Six are endpoints, two are log levels and feature toggles, two are credentials that should never have been in an image, and one is a genuinely invariant build constant that stays.
Two of the credentials being in the image is the finding nobody expected, and it turns the project from a pipeline improvement into a security remediation with its own urgency. The images are rebuilt and the old ones expired.
The cross-account permissions take two attempts. The first grants ECR pull and fails on the KMS key, producing an access-denied error that names ECR rather than KMS and costs an afternoon. The second gets the key policy right, and the resulting runbook entry is the most-referenced part of the whole project.
Switching deployments to digest references produces one moment of friction: the deployment logs become unreadable to humans. Keeping the commit-SHA tag alongside the digest in the pipeline’s output resolves it.
The first promotion under the new pipeline is uneventful, which is the intent. The measurable difference appears three weeks later, when a base image tag moves again. Under the old pipeline that would have been the first incident repeating; under the new one the production deployment references a digest built before the move, and the change surfaces on the next build in development where it belongs.
Scanning once instead of four times reduces the pipeline duration by about seven minutes and removes three duplicate findings tickets a week.
What’s worth remembering
- If the artefact can change between test and deployment, the test is a statement about something else; rebuilding per environment is the obvious such step and re-tagging is a quieter one.
- Configuration baked at build time is what forces the rebuild, so the unblocking work is classifying each value as configuration, secret, or genuinely part of the artefact.
- Reference the artefact by digest and enable immutable tags, because a mutable tag makes “the same bytes” a claim rather than a fact.
- Cross-account promotion needs the repository policy, the consuming role’s permissions, and the KMS key policy where a customer-managed key is used; the missing key permission produces an access-denied error that names the wrong service.
- Put environment-specific values in Parameter Store, secrets in Secrets Manager, and anything whose bad value could break production behind AppConfig so it rolls back on an alarm.
- Scan once at push and gate on that result; scanning the same digest in every environment produces identical reports and slower pipelines.