Lab: Auto-Remediate an AWS Config Rule

This is the first lab in the DevOps Pro hands-on track. The reference posts argue the decisions; this stands one up. Detection is the easy half of governance: a rule that flags a misconfiguration is worth little if someone still has to notice the finding and act on it. The half that scores on a Professional paper is remediation that runs on its own, so the estate corrects itself. Here we build the rule, the drift, and the fix, and wire them together by hand, because the wiring is what there is to learn. The full lab is in lab-dop-01-config-remediation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

AWS Config detects drift and records the result. It changes nothing. Somebody reads the list, opens the resource, and fixes it, and until they do the drift stands.

A remediation closes that gap. You attach one to a rule: when the rule reports a resource non-compliant, Config runs an SSM Automation document against it, with no human in the loop. The drift in this lab is a security group with an inbound rule allowing tcp port 22 from 0.0.0.0/0. The rule is the managed INCOMING_SSH_DISABLED (“restricted-ssh”), which returns NON_COMPLIANT when incoming SSH is allowed from 0.0.0.0/0 or ::/0, and which triggers on both configuration changes and periodic evaluations. The fix is the AWS-owned document AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules. Read what that document actually does before you wire it: it removes every ingress rule on the group that allows traffic from all source addresses, not only the SSH one. The fix is wider than the finding. Your job is to wire the two together and set it to run automatically.

The obvious version of this lab, an unencrypted S3 bucket, no longer works. Amazon S3 has applied SSE-S3 as the base level of encryption on every new bucket since 5 January 2023, at no additional cost, so S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED evaluates a bare bucket as compliant and the remediation never has anything to do. Port 22 open to 0.0.0.0/0 has no such default sitting behind it. Config evaluates the group, flags it, and the remediation has real work.

What you’re given

CloudFormation turns AWS Config on for you: the recorder (scoped to security groups to keep it fast and cheap), the delivery channel and its S3 bucket, and the role Config assumes to read your resources. On top of that it adds the managed rule, a small demo VPC, and a security group inside it with port 22 open to 0.0.0.0/0 for the rule to flag. What is missing is the remediation. The gap in src/template.yaml is a TODO block where two resources go: the role the automation assumes, and the RemediationConfiguration that ties the document to the rule. Complete, the remediation looks like this:

DemoSecurityGroupRemediation:
  Type: AWS::Config::RemediationConfiguration
  Properties:
    ConfigRuleName: !Ref RestrictedSshRule
    TargetType: SSM_DOCUMENT
    TargetId: AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules
    TargetVersion: '1'
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 60
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt RemediationRole.Arn
      SecurityGroupId:
        ResourceValue:
          Value: RESOURCE_ID

Deploying the stack as shipped succeeds and the rule evaluates the group as non-compliant, but a flagged group stays flagged: there is no remediation, so nothing closes it.

Your task

Fill in the TODO block with those two resources. The first is the role the document runs under. A remediation acts under a role you supply, not under your credentials, so the role has to be trusted by ssm.amazonaws.com. AWS documents four actions for this runbook’s AutomationAssumeRole: ec2:DescribeSecurityGroups and ec2:RevokeSecurityGroupIngress for the fix itself, plus ssm:StartAutomationExecution and ssm:GetAutomationExecution. Grant all four. EC2’s Describe actions take * as the resource, the way AWS’s own read-only example policy grants them. Give the role a RoleName so the named-IAM capability the deploy script passes covers it.

The second is the RemediationConfiguration, and two details do the work. Automatic: true makes Config run the document when the rule reports the group non-compliant, rather than waiting for someone to choose Remediate in the console. Unattended is not the same as instant: AWS’s auto-remediation works from a compliance data snapshot captured on a periodic basis, so the fix lands a while after the opening. MaximumAutomaticAttempts and RetryAttemptSeconds are both optional, and the template above sets them to their own defaults, 5 and 60 seconds. They bound failures rather than successes. Once that many failed attempts land inside that window, Config writes a remediation exception for the resource and stops retrying it. Attempts accept 1 to 25, the window up to 2,678,000 seconds.

The other detail is RESOURCE_ID, the only value a ResourceValue takes. Passing SecurityGroupId that way gives the document the flagged group’s own id, so one remediation covers whichever group the rule flags rather than an id you hard-code.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # forces an evaluation and checks the group
./scripts/teardown.sh        # empties the Config bucket and deletes everything

test.sh never launches an instance, which keeps it cheap. It forces a re-evaluation with start-config-rules-evaluation, waits for Config to report a compliance result for the demo group, then reads the group’s inbound rules with describe-security-groups. That last read is the real assertion. StartConfigRulesEvaluation re-runs the rule against the last known configuration state rather than re-reading the resource, so the compliance result trails the live group. It also rejects more than one call a minute. Config and remediation are both eventually consistent, so the “still waiting” lines are normal. With the unedited template the script finds the open SSH rule still in place; with the remediation wired it prints:

  ok: sg-... no longer allows SSH from 0.0.0.0/0
PASS: the Config rule flagged the group and the remediation ran, so the
open inbound SSH rule has been revoked.

If it fails

• The open SSH rule is never revoked. The RemediationConfiguration is missing, or it is not Automatic: true. Detection without remediation leaves the group flagged and nothing acts on it.
• The remediation is present but the group is still open. Check the automation role. It must be trusted by ssm.amazonaws.com and hold all four actions, and its ARN has to be the AutomationAssumeRole parameter. The document assumes that role to make the change, so a missing permission stalls the fix.
• It tried and then stopped. Five failures inside sixty seconds and Config writes a remediation exception for that group, which blocks further automatic attempts until the exception is deleted. Run describe-remediation-execution-status for the failing step and its error. Nothing else announces the exception, which is why an estate running this for real alarms on it.
• The stack fails to deploy with InsufficientCapabilities. The template creates named IAM roles, so it needs CAPABILITY_NAMED_IAM. The deploy script already passes it; if you deploy by hand, add it.
• The stack fails on the recorder or the delivery channel. An account gets one customer managed configuration recorder and one delivery channel per Region. If Config is already on here, drop those resources and the Config role and bucket, and keep the rule, the VPC, the group and the remediation.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• AWS Config detects drift; a RemediationConfiguration is what turns detection into a fix. With Automatic: true Config runs the document itself when the rule reports non-compliant. That means unattended, not immediate, because auto-remediation works from a periodic compliance snapshot.
• Remediations run SSM Automation documents. AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules is AWS-owned, so you do not write the fix logic. Read its scope anyway: it revokes every ingress rule open to all source addresses, which is wider than the SSH opening the rule flagged.
• RESOURCE_ID is how the flagged resource reaches the document, and it is the only value a ResourceValue takes. Config substitutes the non-compliant group’s id for SecurityGroupId, so one remediation covers whichever group the rule flags.
• A remediation acts under a role, not under you. The document assumes the AutomationAssumeRole you supply, and AWS names four actions that role needs here, or the fix never lands.
• Retries are bounded, and nothing pages when they run out. After MaximumAutomaticAttempts failures inside RetryAttemptSeconds, Config writes a remediation exception and leaves the resource non-compliant.
• One remediation configuration covers one rule, in one account, in one Region. Across an organisation the rule and the remediation deploy together in a conformance pack, which is the only place those two resource types travel as a unit, and remediation actions are not available in every Region.

Next

The rest of the DevOps Pro lab arc is in the track’s README.