Lab: Event-Driven Auto-Remediation With EventBridge

This is a lab in the DevOps Pro hands-on track. Auto-remediation is one of the domain’s recurring shapes: something notices a problem, an event carries the detail, and a small piece of code fixes it without a human in the loop. The reference posts argue when that is a good idea; this stands one up. You build the smallest honest slice, a rule that routes one kind of finding to one Lambda that tags the offending instance. The full lab is in lab-dop-02-eventbridge-remediation.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A security service in your account watches for EC2 instances that are missing a required tag. When it finds one, it emits a custom event on the default EventBridge bus: source acme.security, detail-type Instance Missing Required Tag, with the offending instance id inside the detail. The default bus takes events from AWS services, from other authorised accounts, and from PutEvents calls, which is how a custom source like this one lands there. You want that finding to trigger a Lambda that quarantines the instance by tagging it Status=quarantined, so the rest of your tooling can isolate it.

The lazy design is a broad rule that catches everything from acme.security and sorts it out in the function. That fires the Lambda on every finding the service ever emits, bills a Lambda request for each one, and puts the routing decision in code instead of in the template. Narrow the pattern and EventBridge drops the rest before the function is invoked at all. The routing is then declarative and visible in the template, and a buggy handler never sees an event it had no business touching.

What you’re given

CloudFormation builds the Lambda, an execution role that can tag and stop EC2 instances and write its own logs, the EventBridge rule on the default bus, and the permission that lets EventBridge invoke the function. The rule’s EventPattern is a template parameter, fed from a file you control:

RemediationRule:
  Type: AWS::Events::Rule
  Properties:
    EventBusName: default
    EventPattern: !Ref EventPattern
    State: ENABLED
    Targets:
      - Id: RemediationLambda
        Arn: !GetAtt Function.Arn

There are two gaps. The first is src/event-pattern.json, which ships as {"source": ["acme.security"]}. That matches every event from the source, so the rule fires on findings you did not mean to act on. The second is src/handler.py, a skeleton that receives the event but does not yet read the instance id, log the remediation, or tag anything.

Your first task: the pattern

Narrow the rule so it matches only the missing-tag finding. An event pattern has the same structure as the events it matches. It matches a field when the event’s value for that field appears in the list you give, and you can require several fields at once. There is no partial match: a pattern matches an event or it does not. Match on the source and the detail-type together:

{
  "source": ["acme.security"],
  "detail-type": ["Instance Missing Required Tag"]
}

With the source alone, an Instance Terminated finding from the same service would still reach the Lambda. Adding the detail-type pins the rule to the one event you care about, and everything else is dropped at the bus. Matching is exact, so the detail-type string has to be the one the producer sends, character for character.

One more field starts to matter as soon as the bus is not only yours. A rule with no account field matches a forwarded finding from any account whose events the bus accepts. For an account configured to accept events from every AWS account, AWS strongly recommends adding account to every rule, because a pattern carrying that field does not match events from accounts it does not list. Forwarding is also one hop. A receiver account that re-sends what it received to a third account does not reach it, three buses inside one account behave the same way, and a hub-and-spoke design therefore has exactly one hub.

Your second task: the handler

Fill in handler(). The rule delivers the whole EventBridge envelope, so the instance id is at event["detail"]["instanceId"]. Read it, log the remediation before you act so it is recorded even if the tag call fails, then call ec2.create_tags to add Status=quarantined:

instance_id = event["detail"]["instanceId"]
print(f"remediation: tagging instance {instance_id} Status=quarantined")
ec2.create_tags(
    Resources=[instance_id],
    Tags=[{"Key": "Status", "Value": "quarantined"}],
)

Wrap the tag call in try/except ClientError. The test sends synthetic instance ids so it can prove the routing without a running instance, and create_tags returns an error for an id that is not a live instance. Which error depends on the id: InvalidInstanceID.Malformed where the id is not in the documented i-xxxxxxxx or i-xxxxxxxxxxxxxxxxx form, InvalidInstanceID.NotFound where a well-formed id does not exist. Catch the ClientError class, not one code. The log line is what proves the handler ran.

Logging first has a second use. EventBridge retries a delivery for 24 hours and up to 185 times by default. Lambda retries a failed asynchronous invocation up to two times. One finding can therefore reach this handler more than once. create_tags adds or overwrites the tags you name, up to 50 on a resource, so a second run on the same instance writes the same Status=quarantined over the top and nothing drifts. A handler that incremented a counter or opened a ticket would need a guard of its own.

Run it

./scripts/deploy.sh          # deploys your pattern, uploads your handler
./scripts/test.sh            # puts two findings and reads the function's logs
./scripts/teardown.sh        # deletes the stack and the log group

test.sh puts two findings on the bus. One is the missing-tag finding, which should reach the Lambda; the other is an Instance Terminated finding from the same source, which the right pattern should filter out. It then reads the function’s CloudWatch logs and checks what the handler did:

Handler remediated i-0lab02remediate01: true   (expected true)
Handler saw ignored i-0lab02ignored0002:  false  (expected false)

PASS: the rule routed the missing-tag finding to the Lambda, the handler
remediated it, and the terminated-instance finding was filtered out.

If it fails

• Nothing remediated. The handler is probably still the skeleton, or the pattern does not match the source. Check src/event-pattern.json matches acme.security and that handler() logs and tags the instance.
• It also acted on the ignored finding. Your pattern is too broad. It matches the source but not the detail-type, so the Instance Terminated finding reaches the Lambda too. Add the detail-type list and redeploy.
• The handler ran but the tag did not apply. Expected against the synthetic id the test sends. create_tags returns an InvalidInstanceID error and the handler catches it. Against a real instance the tag goes on. The test asserts routing and execution, not the tag.

What the lab does not exercise

Two failure paths run through different machinery, and the green test only proves one of them.

A delivery failure is EventBridge’s to handle. By default it retries for 24 hours and up to 185 times, with exponential backoff and jitter. When the attempts run out the event is dropped, unless the target has a dead-letter queue. Some failures skip the retries altogether: missing permissions to a target, a target that no longer exists, or a target that cannot be found because of an invalid address or a DNS lookup failure go straight to the DLQ with no retry attempts. Delete the AWS::Lambda::Permission from this stack and that is the path the finding takes. Nothing here has a DLQ, so there is nothing to read afterwards.

A handler failure is Lambda’s. A Lambda target is invoked asynchronously: Lambda puts the event in a queue, returns a success response, and a separate process sends it to the function, so an exception inside handler() is Lambda’s to retry. AWS documents two retries for a failed asynchronous invocation, after which the event needs Lambda’s own dead-letter queue to survive. Catching ClientError in the handler cuts all of that out. That is right for an id that will never exist, and wrong for a tag call that failed on a throttle.

The rule has a ceiling too. Five targets per rule, and that number is fixed, so a finding that has to reach a function, a queue and a central bus is already at the limit. Rules per event bus default to 300 per account per Region, or 100 in af-south-1 and eu-south-1, and that quota is adjustable.

Reveal the solution

Deploy both reference answers without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• An EventBridge rule is a router, and the event pattern is where the routing decision lives. Narrowing the pattern costs less and reads better than filtering in the Lambda. It also shrinks the blast radius of a buggy handler, which never sees the events it should not touch.
• A pattern has the same structure as the event, it matches when the event’s field value appears in the list you give, and you can require several fields at once. Matching source alone is broad; adding detail-type pins it to one kind of finding, on an exact string.
• EventBridge can only invoke a Lambda when the function’s resource policy allows it, which is what the AWS::Lambda::Permission with principal events.amazonaws.com grants, scoped by SourceArn to this one rule. Take it away and the finding is not retried: a missing target permission goes to the dead-letter queue immediately, or nowhere if there isn’t one.
• Delivery retries and handler retries are separate. EventBridge retries delivery for 24 hours and up to 185 times; Lambda, invoked asynchronously, retries a function error twice and then needs its own dead-letter queue. A remediation can therefore run more than once, which is survivable here only because create_tags overwrites by key.
• Auto-remediation is a routed event plus a least-privilege handler. The role here can tag and stop instances and write its own logs. In production, narrow the tagging with the aws:RequestTag and aws:TagKeys conditions AWS documents for ec2:CreateTags, so the role writes Status and nothing else.
• One rule covers one bus, in one account, in one Region, and forwarding between accounts is a single hop charged to the sender. Across an organisation the findings come to one bus and the rules that act on them sit beside it.

Next

The rest of the DevOps Pro lab track is in the track’s README.