The situation
An operations team of six runs about forty documented procedures. Five of them account for most of the pages: clearing a full disk on an application instance, restarting a wedged worker process, rotating a credential when a scanner flags it, expanding an RDS volume approaching capacity, and quarantining an instance that GuardDuty has flagged.
Each procedure is a wiki page with console screenshots. Three of the five have been copied into a team-specific version with edits nobody merged back. The disk-space procedure has been executed 61 times in the last quarter, always the same way, always by hand. Mean time to acknowledge a page at night is eleven minutes, and mean time to resolve the disk-space page is another nineteen.
The team has capacity to automate perhaps two of these properly this quarter. Their existing scripts are a mix of Bash in a Git repository nobody deploys from and one Lambda function written by somebody who has left.
What actually matters
The first thing that matters is that a procedure and its trigger are separate problems, and the trigger is usually where the value is. Automating the disk-space fix so a human can run it in one click removes the nineteen minutes; wiring it to fire from the alarm removes the eleven minutes of acknowledgement as well and removes the page. The same script attached to different triggers produces very different outcomes.
The second is that not every step should be automated. A procedure that quarantines an instance takes a production resource out of service, and the correct design frequently keeps a human in the loop for the decision while automating everything around it: the gathering of evidence, the preparation of the action, the execution once approved, and the record. Automating a diagnosis is safe; automating a destructive action needs an argument.
The third is that automation is only trustworthy if what it can touch is bounded. A runbook executed by a person is bounded by that person’s permissions and their judgement. The same runbook executed by a service is bounded only by its role, and a role scoped to “whatever the procedure might need” is how a disk-cleanup automation ends up able to terminate instances.
The fourth is that these procedures are running because something upstream is wrong. A disk that fills 61 times in a quarter is a log rotation defect, and it is being handled as an operations task. Automating the cleanup makes the symptom cheap, which is worth doing, and it also removes the pressure that would eventually have produced the real fix. Both should be tracked.
Underneath it, four divergent copies of a procedure is itself the failure. Whatever replaces them has to be a single definition, versioned, that every team invokes rather than copies.
What we’ll filter on
- What triggers it: a human noticing, an alarm, a schedule, or a finding?
- What does the action do if it runs when it should not have?
- Does a human need to decide, or only to be told?
- What is the smallest set of permissions that completes it?
- Is there a record of what ran, against what, and who approved it?
- Does automating it hide a problem that should be fixed upstream?
The landscape
Systems Manager Automation. Runbooks as documents: multi-step, parameterised, with branching, error handling, and approval steps executed by the service rather than by a person. AWS publishes a large library of AWS- prefixed runbooks covering common operations, so a great deal of this is configuration rather than authoring. Executions run under a specified IAM role, are recorded, and support rate control and error thresholds when acting across a fleet. This is the default answer for operating AWS resources on a known sequence.
Systems Manager Run Command. A single command against a set of managed instances, targeted by tag or resource group. Right when the work is one action on hosts rather than an orchestration across services. It is a step within an Automation runbook as often as it is the whole answer.
EventBridge rules. The trigger layer. Rules match events from CloudWatch alarm state changes, GuardDuty findings, AWS Config compliance changes, Health events, or a schedule, and invoke a target. This is what converts a procedure from something a human starts into something that starts itself.
AWS Config rules with remediation. For anything shaped as “this resource is not in the state it should be”. A Config rule evaluates continuously and an attached remediation action, itself a Systems Manager Automation document, corrects the drift. Config decides whether to act; Automation performs the action.
Lambda. Right for logic, transformation, and calling APIs that Automation does not model well. Wrong as a substitute for Automation when the work is operating AWS resources on a fixed sequence, because you would be rebuilding approvals, rate control, and execution history in code.
Step Functions. For orchestration that Automation cannot express: long waits, complex branching, parallel fan-out, human approval through a task token with an arbitrary front end. It is the escalation from Automation rather than a competitor, and it costs more to build.
Incident Manager. The response side rather than the action side: engagement plans, on-call rotations, escalation, and a timeline of what happened. It attaches Automation runbooks to an incident so the response executes and is recorded against the incident record.
Auto Scaling and self-healing configuration. The cheapest automation is the one that needs no runbook. An instance that fails a health check and is replaced by its Auto Scaling group never generates a page for a wedged process. Where a procedure amounts to “make it go away and get a new one”, the fix is the group’s health check rather than a runbook.
Evaluation
Side by side
| Procedure | Trigger | Destructive | Human decision | Mechanism |
|---|---|---|---|---|
| Clear a full disk | CloudWatch alarm | Low | ✗ | EventBridge → Automation |
| Restart a wedged worker | Health check failure | Low | ✗ | ASG health check replaces it |
| Rotate a flagged credential | Scanner finding | Medium | ✗ if managed | Secrets Manager rotation |
| Expand an RDS volume | CloudWatch alarm | Low, costs money | ✗ | RDS storage autoscaling |
| Quarantine a flagged instance | GuardDuty finding | High | ✓ | EventBridge → Automation with approval |
The table makes the ranking obvious in a way the wiki did not. Two of the five procedures should not be automated at all, because the platform already has a mechanism that removes the need for them: an Auto Scaling health check for the wedged worker, and RDS storage autoscaling for the volume. One is a managed feature nobody enabled. Only two are genuinely runbooks, and only one of those needs a human in the middle.
The solution
Delete two procedures by configuring the platform, enable one managed feature, and automate the remaining two with the trigger wired in from the start.
The wedged worker goes first and produces the largest reduction in pages for the least work. The instances sit in an Auto Scaling group behind a load balancer, and the group’s health check is set to EC2 rather than ELB, which means it notices a terminated instance and not a wedged process. Switching the health check type and adding an application health endpoint means a wedged worker is replaced automatically. The runbook is deleted, not automated.
RDS storage autoscaling is a checkbox with a maximum threshold. Enabling it removes the volume-expansion procedure entirely, and the maximum is the control that stops a runaway from becoming a runaway bill. This is the one where the team’s reaction is that they should have known, and the useful response is that nobody audits for features they are not already using.
Credential rotation moves to Secrets Manager with a rotation schedule and a rotation function, so the scanner stops finding old credentials because there aren’t any. Where a credential cannot be managed that way, the procedure stays, and it is worth being explicit about which ones those are rather than leaving it implicit.
That leaves two. The disk-space procedure becomes a Systems Manager Automation runbook triggered by an EventBridge rule on the CloudWatch alarm’s state change. The runbook gathers the evidence first (what is consuming the space), performs the cleanup, verifies the result, and publishes to SNS so the team is told rather than woken. Its role is scoped to exactly the actions it takes on exactly the instances carrying the right tag. Because the alarm fires the runbook, the eleven minutes of acknowledgement disappear along with the nineteen minutes of work.
The quarantine procedure keeps a human, and automates everything either side of them. An EventBridge rule on GuardDuty findings above a severity threshold starts an Automation runbook that snapshots the volumes, captures the instance metadata, and then stops at an approval step. A responder approves or rejects from a notification; on approval the runbook applies the isolation security group, detaches the instance from its Auto Scaling group, and tags it for forensics. The decision stays human because the action removes a production resource, and everything around the decision is faster and recorded.
Then remove the divergence. The runbooks become documents in the shared services account, shared to the workload accounts, so there is one definition and four invocations rather than four copies. Version them, and treat a change to one as a change everyone gets.
Alongside all of it, keep the upstream fixes on the board. Sixty-one disk-space events in a quarter is a log rotation defect, and the automation makes it cheap rather than absent. Track the invocation count as a metric; a number that keeps climbing after the automation lands is the signal that the real fix is still outstanding.
Why not write Lambda functions for all five. It is the team’s instinct because they have one already. It means rebuilding approval steps, rate control, execution history and the AWS-published runbook library in code they then maintain.
Why not automate the quarantine end to end. It is the procedure where being wrong is most expensive, and a GuardDuty finding is a signal rather than a verdict. Automating the evidence-gathering removes most of the elapsed time without removing the judgement.
Worked example
The health check change lands in an afternoon and pages for wedged workers drop to zero over the following month. The team had assumed the group was already doing this, which is why nobody had questioned a procedure they ran weekly.
RDS storage autoscaling takes ten minutes and a conversation about the maximum, which is set at twice the current volume with a CloudWatch alarm at 80% of that maximum so growth is still visible.
The disk-space automation takes about a week, most of it spent on the evidence-gathering step rather than the cleanup. The first production firing works. The fourth does not: an instance has a full disk for a reason the cleanup does not address, and the runbook’s verification step correctly reports failure and pages a human rather than reporting success. That failure path is the part worth having built.
The quarantine runbook takes two weeks, with the approval step wired to the on-call rotation. Its first real use comes six weeks later at 02:40. Evidence is captured within ninety seconds of the finding, the responder approves at 02:51 after eleven minutes of looking, and isolation completes at 02:52. Under the old procedure the equivalent sequence had taken 40 minutes, most of it spent finding the wiki page and working out which volume to snapshot.
The invocation metric on the disk-space runbook sits at 58 in the following quarter, barely down from 61. The automation is working and the log rotation defect is still there, which is exactly what the metric was added to show. It gets fixed in the quarter after, and the count falls to four.
What’s worth remembering
- The procedure and its trigger are separate problems; automating the steps removes the work, and wiring it to the alarm through EventBridge removes the acknowledgement time and the page as well.
- Before automating, check whether the platform already removes the need: an Auto Scaling health check on the application rather than the instance, or RDS storage autoscaling, deletes a runbook rather than automating it.
- Systems Manager Automation is the default for operating AWS resources on a known sequence, because approvals, rate control, error thresholds and execution history come with it; reach for Lambda when the work is logic rather than operations.
- Automate around a destructive decision rather than through it: gather evidence and prepare the action automatically, keep the approval human, and execute and record once approved.
- Scope the execution role to exactly what the runbook does on exactly the tagged resources, because a human running a procedure is bounded by judgement and a service is bounded only by IAM.
- Track how often each automation fires; a count that does not fall is telling you the automation made a symptom cheap and the underlying defect is still there.