Lab: A Composite Alarm That Only Pages When It Matters

This is the third lab in the DevOps Pro hands-on track. The reference posts argue the decisions; this stands one up. An alarm that fires when nothing is wrong is worse than no alarm, because after a few false pages the team learns to swipe the notification away without looking, and then the real one gets swiped away too. The half that scores on a Professional paper is an alarm that only pages when the signals agree, so a page still means something. Here we build the log group, the metrics, and the alarms, and you wire the part that carries the decision, since that is where the thinking lives. The full lab is in lab-dop-03-composite-alarm.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A single metric filter counts ERROR lines and trips its alarm at 3am. A downstream call retried, logged a handful of errors, and recovered on its own. Nobody needed to wake up, but the pager went off, and after a few nights of that the alarm has taught the team to distrust it. That is the real cost of a noisy signal: it does not just waste one night, it erodes the credibility of every alarm you have.

The answer is to page only when several signals agree the service is genuinely down. Errors happen. Timeouts happen. Errors and timeouts breaching at the same moment is a much stronger claim that something is actually broken. A CloudWatch composite alarm says exactly that. It watches other alarms rather than a metric, and fires on a boolean rule over their states, so you can require both children to be in ALARM together and hand the SNS action to the composite instead of to either noisy child.

What you’re given

CloudFormation builds a log group with short retention, two metric filters that count ERROR and TIMEOUT lines into custom metrics (ErrorCount and TimeoutCount in the AcmeLab namespace), two child alarms, ErrorCountHigh and TimeoutHigh, an SNS topic, and the composite alarm that should page it. What is missing is two things: the filter patterns that decide which lines get counted, and the composite’s rule that decides when to page. As shipped, the filters carry CHANGE_ME placeholders that match nothing real, and the composite watches a single child, so it would page on errors alone. Complete, the composite’s rule looks like this:

ServiceDownComposite:
  Type: AWS::CloudWatch::CompositeAlarm
  Properties:
    AlarmName: AcmeServiceDown
    AlarmRule: 'ALARM("ErrorCountHigh") AND ALARM("TimeoutHigh")'
    AlarmActions:
      - !Ref AlertTopic

Deploying the stack as shipped succeeds, but it behaves wrong. The placeholder patterns mean the metrics never move, so nothing alarms; and the single-child rule is the noisy alarm the scenario is trying to kill.

Your task

Close both gaps in src/template.yaml. First, write the two filter patterns. ErrorMetricFilter should match any log event containing ERROR, and TimeoutMetricFilter any containing TIMEOUT. For an unstructured line the term pattern is just the quoted word, so FilterPattern: '"ERROR"' counts every line with ERROR in it. Leave the metric transformations alone; the DefaultValue: 0 on each is doing quiet work, keeping the metric reporting zero through calm periods so the child alarms settle to OK instead of stalling in INSUFFICIENT_DATA.

Second, rewrite the composite AlarmRule. The shipped rule watches ErrorCountHigh on its own, which is no better than the child alarm. Combine both children with AND so the composite fires only when errors and timeouts are breaching at the same time. That correlation is the whole idea: either signal by itself is suppressed, and only the agreed-upon combination pages a human. The child alarms still trip and still show red on a dashboard; they just do not own the SNS action any more. The composite does.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # drives the log group and watches the composite
./scripts/teardown.sh        # deletes everything

test.sh never launches compute, which keeps it cheap. It writes log lines with put-log-events and polls describe-alarms. First it writes ERROR lines only and confirms the composite stays out of ALARM while timeouts are absent, which is the suppression you are building; then it writes ERROR and TIMEOUT lines together and confirms the composite pages. Metric filters and alarms take a minute or three to react, so the “still waiting” lines are normal. With the wiring in place it prints:

  t+90s  error-child=ALARM  timeout-child=ALARM  composite=ALARM
PASS: the composite stayed calm on errors alone and paged only once both
signals breached. That is a page worth waking up for.

If it fails

• The composite paged during the errors-only phase. Its AlarmRule is watching one child. Combine both with AND, so a single signal cannot page.
• The composite never paged, even with both signals breaching. Either the filter patterns still match nothing (the CHANGE_ME placeholders never move the metrics), or the rule does not name both children. Check both.
• A child alarm sits in INSUFFICIENT_DATA. The metric filter is producing no data. Confirm the FilterPattern matches the lines the test writes, and that the metric transformation keeps its DefaultValue: 0.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A composite alarm watches other alarms, not metrics. Its AlarmRule is a boolean expression over child alarm states, and it can carry its own actions.
• Giving the SNS action to the composite rather than to the noisy children is how you suppress a single flappy signal. The child still trips and still shows red; only the agreed-upon combination pages a human.
• AND requires the children to be in ALARM at the same time, and that correlation is a far stronger statement than either metric alone, which is why it earns the right to wake someone.
• A metric filter with DefaultValue: 0 keeps its metric reporting through quiet periods, so the child alarms resolve to OK instead of stalling in INSUFFICIENT_DATA waiting for a first data point.

Next

The rest of the hands-on lab tracks are listed in the labs README: event-driven auto-remediation, guarded backups, weighted traffic shifting, and more.