Lab: Event-Driven Auto-Remediation With EventBridge

This is a lab in the DevOps Pro hands-on track. Auto-remediation is one of the domain’s recurring shapes: something notices a problem, an event carries the detail, and a small piece of code fixes it without a human in the loop. The reference posts argue when that is a good idea; this stands one up. You build the smallest honest slice, a rule that routes one kind of finding to one Lambda that tags the offending instance. The full lab is in lab-dop-02-eventbridge-remediation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A security service in your account watches for EC2 instances that are missing a required tag. When it finds one, it emits a custom event on the default EventBridge bus: source acme.security, detail-type Instance Missing Required Tag, with the offending instance id inside the detail. You want that finding to trigger a Lambda that quarantines the instance by tagging it Status=quarantined, so the rest of your tooling can isolate it.

The lazy design is a broad rule that catches everything from acme.security and sorts it out in the function. That fires the Lambda on every finding the service ever emits, costs an invocation each time, and buries the routing decision in code. The design that scores on a Professional paper is a rule whose pattern matches the exact finding, so EventBridge drops everything else before the function is invoked at all. The routing is declarative, visible in the template, and cheaper.

What you’re given

CloudFormation builds the Lambda, an execution role that can tag and stop EC2 instances and write logs, the EventBridge rule on the default bus, and the permission that lets EventBridge invoke the function. The rule’s EventPattern is a template parameter, fed from a file you control:

RemediationRule:
  Type: AWS::Events::Rule
  Properties:
    EventBusName: default
    EventPattern: !Ref EventPattern
    State: ENABLED
    Targets:
      - Id: RemediationLambda
        Arn: !GetAtt Function.Arn

There are two gaps. The first is src/event-pattern.json, which ships as {"source": ["acme.security"]}. That matches every event from the source, so the rule fires on findings you did not mean to act on. The second is src/handler.py, a skeleton that receives the event but does not yet read the instance id, log the remediation, or tag anything.

Your first task: the pattern

Narrow the rule so it matches only the missing-tag finding. An EventBridge pattern matches a field when the event’s value for that field appears in the list you give, and you can require several fields at once. Match on the source and the detail-type together:

{
  "source": ["acme.security"],
  "detail-type": ["Instance Missing Required Tag"]
}

With the source alone, a Instance Terminated finding from the same service would still reach the Lambda. Adding the detail-type pins the rule to the one event you care about, and everything else is dropped at the bus.

Your second task: the handler

Fill in handler(). The rule delivers the whole EventBridge envelope, so the instance id is at event["detail"]["instanceId"]. Read it, log the remediation before you act so it is recorded even if the tag call fails, then call ec2.create_tags to add Status=quarantined:

instance_id = event["detail"]["instanceId"]
print(f"remediation: tagging instance {instance_id} Status=quarantined")
ec2.create_tags(
    Resources=[instance_id],
    Tags=[{"Key": "Status", "Value": "quarantined"}],
)

Wrap the tag call in try/except ClientError. The test sends a fake instance id so it can check the routing without a real instance running, and create_tags will raise InvalidInstanceID.NotFound on an id that does not exist. Catching it keeps the invocation clean; the log line is what proves the handler ran.

Run it

./scripts/deploy.sh          # deploys your pattern, uploads your handler
./scripts/test.sh            # puts two findings and reads the function's logs
./scripts/teardown.sh        # deletes the stack and the log group

test.sh puts two findings on the bus. One is the missing-tag finding, which should reach the Lambda; the other is an Instance Terminated finding from the same source, which the right pattern should filter out. It then reads the function’s CloudWatch logs and checks what the handler did:

Handler remediated i-0lab02remediate01: true   (expected true)
Handler saw ignored i-0lab02ignored0002:  false  (expected false)

PASS: the rule routed the missing-tag finding to the Lambda, the handler
remediated it, and the terminated-instance finding was filtered out.

If it fails

• Nothing remediated. The handler is probably still the skeleton, or the pattern does not match the source. Check src/event-pattern.json matches acme.security and that handler() logs and tags the instance.
• It also acted on the ignored finding. Your pattern is too broad. It matches the source but not the detail-type, so the Instance Terminated finding reaches the Lambda too. Add the detail-type list and redeploy.
• The handler ran but the tag did not stick. That is expected against the fake id the test sends; create_tags raises NotFound. Against a real instance the tag would apply. The test asserts routing and execution, not the tag itself.

Reveal the solution

Deploy both reference answers without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• An EventBridge rule is a router, and the event pattern is where the routing decision lives. Narrowing the pattern is cheaper and clearer than filtering in the Lambda, and it shrinks the blast radius of a buggy handler because the function never sees the events it should not touch.
• A pattern matches when the event’s field value appears in the list you give, and you can require several fields at once. Matching source alone is broad; adding detail-type pins it to one kind of finding.
• EventBridge can only invoke a Lambda when the function’s resource policy allows it, which is what the AWS::Lambda::Permission with principal events.amazonaws.com grants, scoped by SourceArn to this one rule.
• Auto-remediation is a routed event plus a least-privilege handler. The role here can tag and stop instances and nothing else, and in production you would scope even those to a tag or an account boundary rather than every instance.

Next

The rest of the DevOps Pro lab track is in the track’s README.