Exam Room · Solutions Architect

Lab: Bake an Image and Replace a Fleet Without Logging In

· 0 min read

Cloud Architecture · part of The Exam Room

This is a lab in the Solutions Architect Associate hands-on track. The reference posts argue the decisions; this one stands a decision up and makes it behave. The full lab is in lab-saa-02-immutable-fleet.zip.

If this is your first lab, do the one-time, once-per-account setup first: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy. This one bakes two AMIs and runs six instances across its life, so tear down when you are finished.

The scenario

An Auto Scaling group runs two instances. A package on them needs updating by Friday. The quick way is Systems Manager: Patch Manager compares each node against a patch baseline, installs what the baseline approves, and reports compliance afterwards, on a schedule or on demand through Patch now. Nothing is replaced, nothing restarts unless the package says so, and by Friday both instances have the update.

They are also now two instances whose contents are the sum of every change anyone ever made to them. Patch compliance data is a point-in-time snapshot from the last successful patching operation, which tells you what was true when that operation ran. It does not tell you that the two disks match, and if one of them was touched by hand in between, they do not.

The other way is to put the change in the image and replace the instances. The fleet then runs software you can name by AMI ID, and a drifted server gets terminated rather than repaired. This lab builds that. An EC2 Image Builder recipe bakes a marker file into an AMI, and a launch template names the AMI. An Auto Scaling group runs from the template, and an instance refresh rolls the group from one image to the next. Along the way you edit one instance by hand, so there is real drift on the fleet when the refresh runs.

What you’re given

CloudFormation builds the whole world: a VPC with two public subnets, security groups with no inbound rules at all, the Image Builder resources, the launch template, and the Auto Scaling group.

The Image Builder half is four resources and a build. A component is a plain-text YAML document of steps. This one writes a release marker and makes it read-only:

Data: !Sub |
  name: app-release
  schemaVersion: 1.0
  phases:
    - name: build
      steps:
        - name: WriteRelease
          action: ExecuteBash
          inputs:
            commands:
              - set -euo pipefail
              - printf '%s\n' '${AppRelease}' > /etc/app-release
              - chmod 0444 /etc/app-release

An image recipe pins a base image and the components applied to it. The parent image here is the versionless x.x.x form of the Amazon Linux 2023 managed image. Every bake then starts from the newest published version rather than from whatever was current the day the recipe was written:

ParentImage: !Sub 'arn:aws:imagebuilder:${AWS::Region}:aws:image/amazon-linux-2023-x86/x.x.x'

An infrastructure configuration says where the build runs: instance type, subnet, security group, and the instance profile the build instance carries. That profile needs two managed policies, not one. EC2InstanceProfileForImageBuilder grants the minimum an instance needs to work with Image Builder, and AWS documents that it deliberately leaves out the permissions for the Systems Manager Agent. Image Builder runs components through Run Command, so AmazonSSMManagedInstanceCore goes on alongside it. Grant only the first and the build sits there until it times out.

Then AWS::ImageBuilder::Image runs an actual build during stack creation. Image Builder launches an instance, applies the component, validates, stops the instance, registers an AMI, and terminates the instance. Allow fifteen minutes. The AMI ID comes back as an attribute, which is what the launch template uses:

FleetLaunchTemplate:
  Type: AWS::EC2::LaunchTemplate
  Properties:
    LaunchTemplateData:
      ImageId: !GetAtt BakedImage.ImageId

Fleet:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    LaunchTemplate:
      LaunchTemplateId: !Ref FleetLaunchTemplate
      Version: !GetAtt FleetLaunchTemplate.LatestVersionNumber

That Version line is worth reading twice. CloudFormation creates a new launch template version when the template data changes, and AWS documents that the default version of a launch template cannot be specified in CloudFormation at all. A group pinned to $Default would therefore sit on version 1 for ever while CloudFormation stacked up versions 2, 3 and 4 beside it, with nothing in the diff to show for it. $Latest tracks correctly and gives up something else, which the next section gets to.

There is no UpdatePolicy on the group, so redeploying the stack with a new AMI changes the launch template and leaves the running instances alone. New launches get the new image. The two instances already up do not.

The gap is the replacement.

Your task

Write src/refresh.json, the input to start-instance-refresh. test.sh substitutes the group name, launch template ID and version from the stack outputs before it calls the API, so use the placeholders as they stand:

{
  "AutoScalingGroupName": "AUTO_SCALING_GROUP",
  "DesiredConfiguration": {
    "LaunchTemplate": {
      "LaunchTemplateId": "LAUNCH_TEMPLATE_ID",
      "Version": "LAUNCH_TEMPLATE_VERSION"
    }
  },
  "Preferences": {
    "MinHealthyPercentage": 100,
    "MaxHealthyPercentage": 200,
    "InstanceWarmup": 60,
    "SkipMatching": true,
    "AutoRollback": true,
    "StandbyInstances": "Terminate",
    "ScaleInProtectedInstances": "Ignore"
  }
}

Four of those lines carry the decision.

DesiredConfiguration with a numbered version is what makes rollback available. AWS is explicit: the rollback option needs a desired configuration, and it needs a specific numbered launch template version. A group configured to use $Latest or $Default, or a launch template whose ImageId is an AMI alias from Parameter Store, cannot roll an instance refresh back at all. That last restriction goes further than rollback. An Auto Scaling group whose launch template resolves its AMI through a Systems Manager parameter cannot start an instance refresh with a desired configuration or with skip matching. Warm pools are out too. The resolve:ssm: indirection is a real pattern, and it rules out the two features this lab is built on.

SkipMatching with a desired configuration replaces only the instances that do not match it. Set it without a desired configuration and the comparison runs against the configuration last saved on the group. After the redeploy that is already the new version, so every instance matches and nothing gets replaced. AWS also warns that skip matching cannot tell whether a user data script pulls fresh code at boot, so a fleet that installs from a repository at launch should leave it off.

MinHealthyPercentage: 100 with MaxHealthyPercentage: 200 launches replacements before terminating anything, which keeps both instances serving throughout. The defaults are different, and they are also different between the console and the CLI. From the CLI, with no instance maintenance policy on the group, minimum healthy defaults to 90 percent and maximum healthy to 100, so the group terminates first and replaces after. Skip matching defaults to disabled from the CLI and enabled in the console. Standby and scale-in-protected instances default to Ignore in the console and Wait from the CLI, and Wait means an hour to deal with them before the refresh fails. Auto rollback and bake time are off everywhere.

AutoRollback: true reverses the deployment if replacements fail to come up. With it enabled, the refresh keeps trying for an hour before it gives up and rolls back. You can add up to ten CloudWatch alarms through AlarmSpecification, with two conditions attached. An alarm sitting in ALARM or INSUFFICIENT_DATA when you start returns an error rather than being ignored. And alarms specified without auto rollback enabled give you a refresh that fails on the alarm and leaves the half-replaced fleet where it is.

Run it

./scripts/deploy.sh          # bakes image one, starts the fleet
./scripts/test.sh            # drifts an instance, bakes image two, refreshes
./scripts/teardown.sh        # deletes the stack, AMIs and snapshots

Set aside about forty minutes. Two Image Builder builds run, at roughly fifteen minutes each, and almost all of the rest is waiting for instances.

test.sh reads the fleet, then edits one instance through Run Command, writing hand-edited over the marker the image put there. That is the drift. Nothing in AWS objects to it, no alarm fires, and the instance keeps serving.

Then it redeploys the stack at recipe version 1.0.1 with the release string green, which bakes a second AMI and creates launch template version 2. It checks the fleet again before going near the refresh, and the same two instance IDs are still there on the first AMI. A new image and a new launch template version replace nothing by themselves.

Then the refresh runs, and the script polls until it lands:

Starting the instance refresh ...
  InProgress              0%
  InProgress             50%
  Successful            100%

Fleet after the refresh:
  i-0a4c91f2e7b6d3c08  ami-04b17e9d3c2a85f61  release=green
  i-0f38b7c1a95e2d40b  ami-04b17e9d3c2a85f61  release=green

PASS: every instance is new, every instance is on ami-04b17e9d3c2a85f61, and
the hand edit on i-07c2e5a8f14d9b630 is gone with the instance that carried it.

Then break it on purpose. Change the version in your refresh.json from the placeholder to "$Default" and run test.sh again: the API returns an error instead of starting, because auto rollback has no numbered version to roll back to. Take SkipMatching out next, and watch the refresh replace instances that were already running the image it is rolling out.

If it fails

  • The stack sits in CREATE_IN_PROGRESS on BakedImage for half an hour, then fails. The build instance cannot reach Systems Manager. Check that the build instance profile carries AmazonSSMManagedInstanceCore as well as EC2InstanceProfileForImageBuilder, and that the subnet has a route to the internet or interface endpoints for ssm, ssmmessages and ec2messages.
  • The refresh is rejected before it starts. The launch template version in the desired configuration is $Latest or $Default while AutoRollback is true. Rollback needs a specific numbered version to roll back to.
  • The refresh succeeds in seconds and replaces nothing. SkipMatching is on without a DesiredConfiguration. The comparison ran against the group’s own saved configuration, which the redeploy had already updated.
  • The refresh fails after about an hour with instances stuck launching. The replacements are not passing health checks inside the warmup window. Raise InstanceWarmup, or set a default instance warmup on the group, which is what the refresh falls back to along with the health check grace period.
  • A second deploy.sh at the same recipe version does nothing. Components and recipes are versioned resources, and a change goes in as a new version. The second bake needs a higher BuildVersion.
  • Teardown leaves AMIs behind. Image Builder registers the AMI in your account and it outlives the stack. teardown.sh deregisters both images and deletes their snapshots; check the EC2 console if you deleted the stack by hand.

Reveal the solution

SRC=solution ./scripts/test.sh

What you just learned

  • Immutable infrastructure is a rule about where change enters: the image changes, and instances are replaced rather than edited. What makes it hold is that the fleet has no other path in. This lab’s security groups have no inbound rules. The only write to a running instance is the one the test script makes on purpose, so the refresh has something to erase.
  • An instance refresh is the replacement mechanism for an Auto Scaling group, and its defaults are conservative and inconsistent between the console and the CLI. From the CLI: 90 percent minimum healthy, 100 percent maximum, rollback off, skip matching off, standby and scale-in-protected instances set to Wait, checkpoint delay one hour.
  • Rollback is the reason to pin a numbered launch template version. $Latest, $Default and a resolve:ssm: AMI alias each rule it out, and the parameter alias also rules out desired configurations and skip matching entirely.
  • Image Builder itself is free; you pay for the EC2 instance it launches, the EBS snapshots behind the AMIs, the S3 logs and any Inspector scanning. The cost of baking is mostly the AMIs you forget to delete, which is what its lifecycle policies exist to deprecate, disable or delete for you.
  • The pipeline can close the loop without a human in it. A distribution configuration can create a new launch template version carrying the new AMI ID and optionally set it as the default. It can also write the AMI ID into a Parameter Store parameter of type aws:ec2:image, as part of the build.
  • Patch Manager is still the answer for fleets you cannot rebuild, and for reporting. Its baselines, scan-only mode and compliance reports tell you what is missing across nodes you do not control. What it cannot give you is two disks you can prove are identical.

Next

The rest of the Solutions Architect Associate lab track is in the README.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.