The situation
A healthcare software company deploys 34 CloudFormation stacks across six accounts. Every one of them names an AMI ID in a mapping, keyed by region. The mappings were populated by hand and the newest entry is nine months old.
A vulnerability scan has flagged 140 unpatched packages across the fleet, all of them fixed in images published since. The remediation ticket says “use current AMIs”, which is the correct instruction and not a design.
Two things complicate it. The images are not stock: a golden image build adds an agent, a CA bundle, and a hardened configuration, so “the latest Amazon Linux” is not what the stacks should use. And two of the 34 stacks run a regulated workload where an unreviewed image change is itself a compliance problem, so whatever mechanism is chosen cannot silently change what those stacks deploy.
What actually matters
The first thing that matters is that a hardcoded AMI ID fails quietly. Nothing breaks, no alarm fires, and the only signal is a scan months later. Any replacement should make staleness visible rather than merely possible to avoid.
The second is that resolving the newest image at deployment time trades one problem for another. A template that always takes the latest is a template where redeploying an unchanged stack can produce a different result, which is exactly what a reproducible build is meant to prevent. That is acceptable for a stateless web tier and unacceptable for the regulated workload, and a single mechanism applied to both will be wrong for one of them.
The third is that pinning and updating are both needed, and the useful design is one where the pin is explicit and moving it is easy. A version that can only be changed by hand-editing 34 templates will not be changed; a version that changes on its own cannot be reviewed. What works is a pointer that is updated deliberately by a pipeline.
The fourth is that the image itself needs a pipeline before any of this matters. A golden image built by somebody running a script on a laptop is not a thing that can be kept current, whatever the templates reference.
Underneath it, updating the reference does nothing to running instances. A new AMI changes what launches next, and getting the change onto the fleet is a separate operation with its own risk.
What we’ll filter on
- Does a stale image become visible, or stay silent?
- Is the version pinned, and can the pin be moved without editing every template?
- Does redeploying an unchanged stack produce the same result?
- Who reviews an image change, and can it be blocked for specific workloads?
- Does the mechanism update running instances, or only future launches?
- What builds and tests the image in the first place?
The landscape
Hardcoded AMI IDs in mappings. Fully reproducible and fully manual. A redeploy always produces the same image, and keeping the mapping current is somebody’s job that nobody does. The failure is silent, which is why nine months passed.
SSM Parameter Store public parameters. AWS publishes the latest AMI IDs for its own images as public SSM parameters, and CloudFormation can resolve a parameter of type AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> at deployment. It removes the mapping entirely for stock images, and it resolves whatever is current at deploy time. It does not help with a golden image, which AWS does not publish.
A private SSM parameter as a pointer. The company’s own parameter, say /golden/al2023/latest, holding the current golden AMI ID. Templates resolve it; the image pipeline writes it. This is the mechanism that separates the pin from the templates: 34 stacks reference one pointer, and moving the pointer is one write. Parameter Store keeps a version history, so rolling back is setting the value to a previous one.
Parameter versions for pinning. Referencing a specific parameter version rather than the latest gives a stack a pin that does not move, while still living in the same mechanism. This is how the regulated stacks get reproducibility without being excluded from the design.
EC2 Image Builder. A pipeline that builds the golden image from a base, applies components, tests the result, and distributes it to accounts and regions. It can write the resulting AMI ID to an SSM parameter as a distribution step, which is the join between the image pipeline and the templates. It also supports scheduled builds with a dependency-update trigger, so a new base image from AWS produces a new golden image without anyone starting it.
Instance refresh and launch templates. Updating the AMI reference changes what launches next and nothing that is running. An Auto Scaling group picks it up on the next scale-out, or immediately via an instance refresh, batched with a minimum healthy percentage.
Image deprecation and recycle bin. AMIs can be deprecated with a date, so an old image stops being selected by “latest” style lookups while remaining usable if referenced explicitly. This is how an ageing image becomes visible rather than just old.
AWS Config and Inspector. The detection layer: Inspector reports CVEs on running instances, and a Config rule can check whether instances were launched from an approved image. Together they turn “are we current” into a number rather than a scan somebody runs quarterly.
Evaluation
Side by side
| Mechanism | Reproducible redeploy | Staleness visible | Update effort | Works for golden images |
|---|---|---|---|---|
| Hardcoded mapping | ✓ | ✗ silent | Edit every template | ✓ |
| Public SSM parameter (latest) | ✗ | n/a | None | ✗ stock only |
| Private SSM pointer (latest) | ✗ | Via pipeline | One write | ✓ |
| Private SSM pointer, pinned version | ✓ | Via pipeline | One reference change | ✓ |
| Image Builder writing the pointer | n/a | ✓ build history | Automatic | ✓ |
| Inspector + Config rule | n/a | ✓ measured | n/a | ✓ |
The two rows in the middle are the same mechanism used two ways, which is what makes it the answer: the ordinary stacks resolve the pointer’s latest value, the regulated ones pin a version, and both live in one system that one pipeline updates.
The solution
Build the image on a pipeline, publish the result to an SSM parameter, have templates resolve that parameter, and pin the version for the workloads that need review.
The image pipeline comes first, because nothing else works without it. Move the golden image build into EC2 Image Builder: a recipe naming the base image and the components that add the agent, the CA bundle and the hardening, a test stage that verifies the result boots and the agent reports, and a distribution configuration that copies the AMI to the six accounts and the regions in use. Schedule it, with the dependency-update trigger, so a new AWS base image produces a new golden image without anybody remembering.
Have the distribution step write the resulting AMI ID to an SSM parameter per image family, /golden/al2023/latest and its siblings. That parameter is now the single place the fleet’s image version lives, and Parameter Store’s version history is the rollback mechanism.
Change the 34 templates to take a parameter of type AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> with a default of the pointer’s name, and delete the mappings. This is a mechanical change and the largest single reduction in template surface in the whole exercise: a per-region mapping in every template becomes one parameter.
For the two regulated stacks, reference a specific parameter version rather than the latest. Those stacks then deploy the same image every time, the version they use is visible in the template, and moving it is a reviewable pull request rather than an invisible consequence of the image pipeline running. That is the requirement satisfied without carving those workloads out of the design.
Getting the new image onto running instances is the separate operation. For Auto Scaling groups, an instance refresh with a minimum healthy percentage, scheduled rather than manual, so the fleet rolls onto each new image within a known interval. For anything not in a group, the image change does nothing and patching is a Systems Manager problem instead.
Then make staleness visible, because the original failure was silence. Deprecate the previous AMI when a new one is published, so an old image is marked rather than merely old. Run Inspector continuously and alarm on findings above a severity threshold. Add a Config rule checking that running instances were launched from an approved image, which catches the instance somebody launched from a snapshot in 2024.
Why not point everything at the public latest parameter. It removes the mapping and it deploys stock images, which lack the agent, the CA bundle and the hardening. The golden image exists for reasons, and this would quietly discard them.
Why not keep the hardcoded IDs and add a reminder to update them. The nine months are the evidence. A mechanism that depends on somebody remembering has already been tested here.
Worked example
Moving the build into Image Builder takes three weeks, most of it turning a 400-line shell script into components and discovering two steps that only worked because of state left over from an earlier run on the build machine.
The test stage catches something on the second build: the hardening component disables a service the monitoring agent depends on, which had been true for months and was invisible because nothing tested the image. Instances had been launching without monitoring, and the fleet’s dashboards had a gap nobody had connected to the image.
The template change takes two days. Thirty-four templates lose their mappings, and the diff removes about 900 lines of region-keyed AMI IDs.
The regulated stacks pin parameter version 14. Six weeks later the pipeline has published versions 15 through 19, the ordinary stacks have moved with them, and the regulated ones have not moved at all. Their upgrade to version 19 is a pull request with the image’s build report attached, reviewed and approved, which is the process the compliance requirement was asking for and which nobody had a way to run before.
The scheduled instance refreshes take the CVE count from 140 to 6 over three weeks. The remaining six are on the long-lived instances that are not in Auto Scaling groups, which is a Patch Manager problem and gets its own ticket.
Four months on, a new base image with a kernel fix is published by AWS on a Tuesday. The pipeline builds and tests overnight, the parameter moves, the scheduled refresh rolls it out over the following two days, and nobody does anything. The regulated stacks stay where they are until somebody reviews them, which is correct.
What’s worth remembering
- A hardcoded AMI ID fails silently: the stack keeps deploying and the image keeps ageing, so the replacement mechanism should make staleness visible rather than only avoidable.
- Resolving “latest” at deployment time costs reproducibility, since redeploying an unchanged stack can produce a different result. That is fine for a stateless tier and wrong for a workload where the image is reviewed.
- An SSM parameter as a pointer separates the pin from the templates: many stacks reference one parameter, and the image pipeline moves it with a single write.
- Referencing a specific parameter version gives reproducibility inside the same mechanism, so regulated workloads pin and ordinary ones float without needing two designs.
- Image Builder tests the image before distributing it and can write the resulting AMI ID to the parameter, which is the join between the image pipeline and the templates.
- Changing the AMI reference affects future launches only; getting it onto running instances is a scheduled instance refresh, and anything outside an Auto Scaling group is a patching problem instead.