Lab: Auto-Remediate an AWS Config Rule

This is the first lab in the DevOps Pro hands-on track. The reference posts argue the decisions; this stands one up. Detection is the easy half of governance: a rule that flags a misconfiguration is worth little if someone still has to notice the finding and act on it. The half that scores on a Professional paper is remediation that runs on its own, so the estate corrects itself. Here we build the rule, the drift, and the fix, and wire them together by hand, because the wiring is what there is to learn. The full lab is in lab-dop-01-config-remediation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A misconfiguration should be corrected by the platform, not filed in a queue for someone to action next week. AWS Config detects drift, but detection on its own just produces a list of things that are wrong. Somebody still has to read the list, open the resource, and fix it, and until they do the drift stands.

The interesting part is what happens after the rule fires. Config can attach a remediation to a rule: when the rule reports a resource non-compliant, Config runs an SSM Automation document against it, with no human in the loop. In this lab the drift is a security group left open to the world on SSH, with an inbound rule allowing tcp port 22 from 0.0.0.0/0. The rule is the managed INCOMING_SSH_DISABLED (“restricted-ssh”), and the fix is the AWS-owned document AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules, which revokes the ingress rules open to the world. Your job is to wire the two together and set it to run automatically.

This is a genuinely non-compliant starting point, which the old encryption scenario was not. Since January 2023 every new S3 bucket gets default SSE-S3 encryption for free, so the S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED rule reads a bare bucket as compliant and the remediation never has anything to do. An open security group has no such safety net: the moment Config evaluates it, port 22 open to 0.0.0.0/0 is flagged, and the remediation has real work to do.

What you’re given

CloudFormation turns AWS Config on for you: the recorder (scoped to security groups to keep it fast and cheap), the delivery channel and its S3 bucket, and the role Config assumes to read your resources. On top of that it adds the managed rule, a small demo VPC, and a security group inside it with port 22 open to 0.0.0.0/0 for the rule to flag. What is missing is the remediation. The gap in src/template.yaml is a TODO block where two resources go: the role the automation assumes, and the RemediationConfiguration that ties the document to the rule. Complete, the remediation looks like this:

DemoSecurityGroupRemediation:
  Type: AWS::Config::RemediationConfiguration
  Properties:
    ConfigRuleName: !Ref RestrictedSshRule
    TargetType: SSM_DOCUMENT
    TargetId: AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules
    TargetVersion: '1'
    Automatic: true
    MaximumAutomaticAttempts: 5
    RetryAttemptSeconds: 60
    Parameters:
      AutomationAssumeRole:
        StaticValue:
          Values:
            - !GetAtt RemediationRole.Arn
      SecurityGroupId:
        ResourceValue:
          Value: RESOURCE_ID

Deploying the stack as shipped succeeds and the rule evaluates the group as non-compliant, but a flagged group stays flagged: there is no remediation, so nothing closes it.

Your task

Fill in the TODO block with those two resources. The first is the role the document runs under. A remediation acts under a role you supply, not under your credentials, so the role has to be trusted by ssm.amazonaws.com and hold the permissions the fix needs: ec2:RevokeSecurityGroupIngress to remove the open rule, and ec2:DescribeSecurityGroups to read the group first. DescribeSecurityGroups does not support resource-level permissions, so the policy uses *. Give the role a RoleName so the named-IAM capability the deploy script passes covers it.

The second is the RemediationConfiguration itself, and two details do the work. Automatic: true is what makes it self-healing: Config triggers the document as soon as the rule reports the group non-compliant, rather than waiting for someone to click remediate. When you set it automatic you must also give MaximumAutomaticAttempts and RetryAttemptSeconds, so Config knows how hard to retry. The other detail is RESOURCE_ID: passing SecurityGroupId as a ResourceValue of RESOURCE_ID hands the flagged group’s own id to the document, so the one remediation fixes whichever group the rule flags rather than an id you hard-code.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # forces an evaluation and checks the group
./scripts/teardown.sh        # empties the Config bucket and deletes everything

test.sh never launches an instance, which keeps it cheap. It nudges the rule to re-evaluate with start-config-rules-evaluation, watches Config report a compliance result for the demo group, then reads the group’s inbound rules with describe-security-groups. Config and remediation are both eventually consistent, so the “still waiting” lines are normal. With the unedited template it finds the open SSH rule still in place; with the remediation wired it prints:

  ok: sg-... no longer allows SSH from 0.0.0.0/0
PASS: the Config rule flagged the group and the remediation ran, so the
open inbound SSH rule has been revoked.

If it fails

• The open SSH rule is never revoked. The RemediationConfiguration is missing, or it is not Automatic: true. Detection without remediation leaves the group flagged and nothing acts on it.
• The remediation is present but the group is still open. Check the automation role. It must be trusted by ssm.amazonaws.com and allow ec2:RevokeSecurityGroupIngress (and ec2:DescribeSecurityGroups), and its ARN has to be the AutomationAssumeRole parameter. The document assumes that role to make the change, so a missing permission stalls the fix.
• The stack fails to deploy with a capabilities error. The template creates named IAM roles, so it needs CAPABILITY_NAMED_IAM. The deploy script already passes it; if you deploy by hand, add it.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• AWS Config detects drift; a RemediationConfiguration is what turns detection into a fix. Set Automatic: true and Config runs the remediation itself when the rule reports non-compliant, with no human in the loop.
• Remediations run SSM Automation documents. AWSConfigRemediation-RemoveUnrestrictedSourceIngressRules is AWS-owned, so you do not write the fix logic; you point Config at the document and pass the parameters it expects.
• RESOURCE_ID is how the flagged resource flows into the document. Config substitutes the non-compliant group’s id for SecurityGroupId, so one remediation covers whichever group the rule flags.
• A remediation acts under a role, not under you. The document assumes the AutomationAssumeRole you supply, and that role has to hold the permission to make the change, or the fix never lands.

Next

The rest of the DevOps Pro lab arc is in the track’s README.