The situation
A payments company runs 900 EC2 instances across four accounts, in a mix of Auto Scaling groups and long-lived single instances. A compliance deadline requires a configuration change on all of them: a new audit agent installed, configured against a central collector, and running.
The agent has been tested on twenty instances in staging and works. Nobody is confident that staging represents production, because production carries three operating system versions and two instance families that staging does not.
Three constraints. The change must reach every instance including ones launched after the rollout, because a compliance requirement met on Tuesday and missed on Wednesday is not met. Roughly 200 instances are in Auto Scaling groups that replace members regularly. And the payments processing instances have a change freeze during business hours, which is the majority of the fleet’s traffic.
What actually matters
The first thing that matters is the difference between applying a change once and holding a state. A command that installs the agent on 900 instances satisfies the requirement today and fails it tomorrow, when the Auto Scaling group launches a replacement without it. A mechanism that reapplies is the only kind that meets a standing requirement.
The second is that batching is what converts a fleet-wide failure into a small one. The change is going to break something on a subset nobody predicted, and the design question is whether that subset is twenty instances or nine hundred. Rate control and an error threshold are what bound it, and a rollout without them is a bet that the testing was complete.
The third is that “it ran” and “it worked” are different signals. A command that exits zero has proved the script ran, not that the agent is reporting to the collector. A rollout that advances on exit status will happily march a broken configuration across the fleet, and the verification step is what makes batching meaningful.
The fourth is that instances in Auto Scaling groups and long-lived instances want different treatment. A group member should get the change baked into the image or the launch template, so replacements arrive with it; a long-lived instance has to be changed in place. Applying one mechanism to both means either mutating instances that should be immutable, or leaving the immutable ones out.
Underneath it, the change freeze is a scheduling constraint that the mechanism should respect automatically rather than by somebody remembering. A maintenance window is the mechanism for that.
What we’ll filter on
- Does it apply once, or hold the state and reapply?
- Does it reach instances launched after the rollout?
- Can it batch, and does it stop when a batch fails?
- Does it verify the outcome, or only the exit status?
- Does it respect a change freeze without a human enforcing it?
- Is it right for group members, long-lived instances, or both?
The landscape
Systems Manager Run Command. A command against instances targeted by tag or resource group, with rate control (concurrency and error threshold) built in. It applies once. It is the right tool for an ad-hoc action across a fleet and the wrong one for a standing requirement, because nothing reapplies it to a new instance.
Systems Manager State Manager. An association binding a document to targets on a schedule, reapplying so configuration converges. Targeting by tag means an instance launched later and carrying the tag is configured without anyone acting, which is the property the compliance requirement needs. It reports compliance per association, and it supports rate control the same way Run Command does.
Systems Manager Patch Manager. The specialised case of the same idea for OS patching, with baselines and compliance reporting. Named here to keep the distinction clear: State Manager is configuration generally, Patch Manager is patch baselines.
Maintenance windows. A schedule with registered targets and tasks, so work runs inside an approved period and stops at the end of it. This is how a change freeze becomes a property of the mechanism rather than a note in a runbook.
EC2 Image Builder. Builds and tests AMIs on a pipeline, producing a new image with the agent already installed, distributed to the accounts and regions that need it. For Auto Scaling group members this is the durable answer: the change is in the image, so every replacement has it and nothing has to reapply anything at runtime.
Launch template versions and instance refresh. Once a new AMI exists, the group’s launch template points at it and an instance refresh cycles members onto it in batches, respecting a minimum healthy percentage and checkpointing between batches. This is the group-member rollout with health checks doing the verification.
Systems Manager Automation. Orchestration where the change is a sequence rather than one command, with branching, approvals and its own rate control. Reach for it when installing the agent involves several steps with decisions between them.
AWS Config with remediation. The detection layer. A rule evaluating whether an instance has the agent, with a remediation action attached, catches anything the rollout missed and anything that regresses. It is the backstop rather than the mechanism.
Distributor. Packages software for distribution through Systems Manager, versioned, so “install the agent” references a package rather than a script that curls something from the internet.
Evaluation
Side by side
| Mechanism | Applies once or holds | Reaches new instances | Batching | Verifies | Respects a freeze |
|---|---|---|---|---|---|
| Run Command | Once | ✗ | ✓ rate control | Exit status | Manually |
| State Manager | Holds, reapplies | ✓ by tag | ✓ rate control | Association compliance | Via a window |
| Image Builder + refresh | Baked into the image | ✓ by construction | ✓ per batch | Health checks | Via scheduling |
| Automation | Once, or on a trigger | Via EventBridge | ✓ rate control | Whatever you write | Via a window |
| Config + remediation | Continuous | ✓ | ✗ | Rule evaluation | ✗ |
| Maintenance window | n/a | n/a | n/a | n/a | ✓ |
The table separates the fleet cleanly. Group members want the image path, because a change in the AMI is a change every replacement inherits without a runtime mechanism. Long-lived instances want State Manager, because they are not replaced and something has to hold the state. Nothing on this list wants Run Command, which is worth noticing given it is the first tool most people reach for.
The solution
Bake it into the image for group members, hold it with State Manager for the rest, batch both, and use Config as the backstop.
Start with the image, because it covers 200 of the 900 instances and covers them permanently. Build a pipeline in Image Builder that installs and configures the agent on top of the current base image, runs a test component verifying the agent reports to the collector, and distributes the result to the four accounts. The test component matters: an image that builds is not an image that works, and Image Builder will fail the pipeline rather than distribute a broken AMI.
Roll the groups onto it with launch template versions and instance refresh, one group at a time, with a minimum healthy percentage that keeps the service up and checkpoints between batches. The group’s own health check is the verification, which is the cheapest verification available and the reason group members are the easy half.
The remaining 700 long-lived instances get a State Manager association targeting a tag, running a Distributor package rather than a script pulling from the internet. Because the association reapplies on a schedule, an instance that drifts is corrected and an instance that appears later carrying the tag is configured without anybody acting. That is what turns a one-off rollout into a standing state.
Configure the association’s rate control before enabling it: a concurrency of 5%, an error threshold of ten instances, and a maintenance window covering the approved period so nothing runs during the freeze. The error threshold is what makes the difference between a bad batch and a bad day.
Write the verification into the document rather than relying on exit status. The last step should confirm the agent process is running and that the collector has received a heartbeat, and fail the association if not. Association compliance then means what it says, and the compliance report becomes evidence rather than an assertion that a script ran.
Sequence the rollout by blast radius. Non-production first, then the internal-facing production instances, then the payments processing tier last and inside its window. Each stage runs for a few days before the next, which is what would have caught a problem that only appears under load.
Then add the Config rule with remediation as the backstop. It catches the instance that missed its tag, the one that was stopped during every association run, and the regression somebody introduces in six months. The rollout is the mechanism; the rule is how you know the mechanism is still working.
Why not Run Command across the fleet and be done in an afternoon. It is the fastest route to a compliance report that is true on the day and false a week later, because nothing reapplies it to the replacements the Auto Scaling groups are already making.
Why not State Manager for the group members too. It works, and it means every replacement instance spends its first minutes unconfigured while the association catches up, and the fleet’s configuration depends on a runtime mechanism where an image would have removed the question.
Worked example
The Image Builder pipeline fails on its first run, in the test component, because the agent needs a configuration value the build did not have. Catching it there rather than after distribution is the argument for having written the test component at all.
The group rollouts take a fortnight across eleven groups. One group’s refresh stalls at 40% because the new AMI’s agent binds a port the application already used, which is exactly the class of problem staging would not have shown. The refresh stops at the checkpoint, the group is left running a mix of versions with the service up, and the fix ships two days later.
The State Manager association runs first against 40 non-production instances at 5% concurrency. Six fail: three are running an OS version the package does not support, and three have no outbound path to the collector because they sit in a subnet with a restrictive NACL. The error threshold halts the run at six, which is the mechanism doing its job.
The OS version gap becomes a separate stream of work covering 31 production instances, which nobody had known about, and which a fleet-wide Run Command would have discovered by failing on all 31 at once with no threshold to stop it.
The payments tier goes last, inside its window, at a concurrency of two instances. It takes three nights and is uneventful.
The Config rule reports 100% compliance eleven days after the rollout finishes, then drops to 99.6% a month later when a team launches four instances from an old launch template. The remediation corrects them within the hour, and the launch template is fixed the same day. That is the state the whole exercise was aiming at: not a number that was once correct, but one that self-corrects.
What’s worth remembering
- Applying a change once and holding a state are different requirements; Run Command satisfies a compliance standard on the day it runs and fails it as soon as an Auto Scaling group replaces a member.
- Group members want the change baked into the AMI so replacements inherit it, and long-lived instances want a State Manager association that reapplies; using one mechanism for both is wrong in one direction or the other.
- Rate control with an error threshold is what turns an unpredicted failure into a small one, and a rollout without it is a bet that the testing covered production’s variety.
- Verify the outcome rather than the exit status. A step confirming the agent is reporting to the collector is what makes association compliance mean something.
- A maintenance window makes a change freeze a property of the mechanism rather than something a human has to remember at 2am.
- Finish with a Config rule and remediation as the backstop; the rollout is how the fleet gets there, and the rule is how you know it is still there in six months.