Lab: A Composite Alarm That Only Pages When It Matters

This is the third lab in the DevOps Pro hands-on track. The reference posts argue the decisions; this stands one up. An alarm that fires when nothing is wrong is worse than no alarm, because after a few false pages the team learns to swipe the notification away without looking, and then the real one gets swiped away too. The work is to build an alarm that pages only when several signals agree, so a page still means something. Here we build the log group, the metrics, and the alarms, and you wire the part that carries the decision. The full lab is in lab-dop-03-composite-alarm.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A single metric filter counts ERROR lines and trips its alarm at 3am. A downstream call retried, logged a handful of errors, and recovered on its own. Nobody needed to wake up, but the pager went off, and after a few nights of that the team has stopped reading the page before clearing it. The damage spreads past the one alarm: the next page, from a different alarm, gets the same shrug.

Page only when several signals agree the service is down. Errors happen. Timeouts happen. Errors and timeouts breaching at the same moment is a much stronger claim that something is broken, and a composite alarm is how CloudWatch states it. A composite monitors the states of other alarms rather than a metric. Its AlarmRule is a boolean expression over those states, built from ALARM(), OK() and INSUFFICIENT_DATA() joined with AND, OR, NOT, or AT_LEAST(M, STATE, (alarms)) for an M-of-N rule. Hand the SNS action to the composite, leave the children without one, and a single noisy signal reaches nobody.

What you’re given

CloudFormation builds a log group with one-day retention, two metric filters that count ERROR and TIMEOUT lines into custom metrics (ErrorCount and TimeoutCount in the AcmeLab namespace), two child alarms, ErrorCountHigh and TimeoutHigh, an SNS topic, and the composite alarm that should page. The log group stays in the Standard log class, because metric filters are not supported on Infrequent Access. Each child sums its metric over a 60-second period, evaluates one period, and trips at three or more. Both set TreatMissingData: notBreaching, so a period with no data point reads as within the threshold instead of sending the child to INSUFFICIENT_DATA.

Two things are missing: the filter patterns that decide which lines get counted, and the composite’s rule that decides when to page. As shipped, the filters carry CHANGE_ME placeholders that match nothing real, and the composite watches a single child, so it would page on errors alone. Complete, the composite’s rule looks like this:

ServiceDownComposite:
  Type: AWS::CloudWatch::CompositeAlarm
  Properties:
    AlarmName: AcmeServiceDown
    AlarmRule: 'ALARM("ErrorCountHigh") AND ALARM("TimeoutHigh")'
    AlarmActions:
      - !Ref AlertTopic

Deploying the stack as shipped succeeds, but it behaves wrong. The placeholder patterns mean the metrics never move, so nothing alarms, and the single-child rule is the noisy alarm the scenario is trying to kill.

Your task

Close both gaps in src/template.yaml. First, write the two filter patterns. ErrorMetricFilter should match any log event containing ERROR, and TimeoutMetricFilter any containing TIMEOUT. For an unstructured line a single-term pattern is the bare word, FilterPattern: ERROR. Double quotes are for exact phrases and for terms carrying non-alphanumeric characters, so the quoted '"ERROR"' the solution uses matches the same lines. Matching is case sensitive either way, so Error and error go uncounted.

Leave the metric transformations alone, and read DefaultValue: 0 precisely. It makes the filter report zero for a period in which logs were ingested and nothing matched, which stops the metric going spotty. It does nothing for a service that has gone quiet: when no log events arrive at all in a one-minute period, no value is reported, and TreatMissingData is what decides the child’s state through real silence.

Second, rewrite the composite AlarmRule. The shipped rule watches ErrorCountHigh alone, which is no better than the child alarm. Combine both children with AND so the composite reaches ALARM only while errors and timeouts are breaching together. The children go on evaluating and go on showing red on a dashboard; they just no longer hold the SNS action.

Then be honest about what AND gives up. An outage that produces errors and no timeouts never pages anybody, and neither does a fault whose two signals never overlap, because the rule reads current states rather than a window. Where that is too strict, AT_LEAST(2, ALARM, (...)) over a wider set of children is the middle ground. For a known deployment window, add AND NOT ALARM(DeploymentInProgress) to the rule, or point the composite at an ActionsSuppressor alarm, which also takes a wait period and an extension period. Pick the failure you would rather own: a page for a blip, or silence through something real.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # drives the log group and watches the composite
./scripts/teardown.sh        # deletes everything

test.sh never launches compute, which keeps it cheap. It writes log lines with put-log-events and polls describe-alarms. First it writes six ERROR lines and no timeouts, and confirms the error child trips while the composite stays out of ALARM, which is the suppression you are building; then it writes six of each together and confirms the composite pages. CloudWatch aggregates and reports metric filter values every minute, and each alarm then needs an evaluation period of its own, so the “still waiting” lines are normal. With the wiring in place it prints:

  t+90s  error-child=ALARM  timeout-child=ALARM  composite=ALARM
PASS: the composite stayed calm on errors alone and paged only once both
signals breached. That is a page worth waking up for.

Tear the stack down when you are finished. Two custom metrics, two metric alarms and the composite each carry a monthly charge, small but not nothing, and the reaper only catches a stack you forget for a day.

If it fails

• The composite paged during the errors-only phase. Its AlarmRule is watching one child. Combine both with AND, so a single signal cannot page.
• The composite never paged, even with both signals breaching. Either the filter patterns still match nothing (the CHANGE_ME placeholders never move the metrics), or the rule does not name both children. Check both.
• A child alarm sits in INSUFFICIENT_DATA. No data point has reached the metric at all. Confirm the FilterPattern matches the lines the test writes, case included, and that the filter is attached to the log group the test writes to.

Reveal the solution

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A composite alarm monitors other alarms, not metrics. Its AlarmRule is a boolean expression over child alarm states, and it can carry its own actions.
• A composite can notify an SNS topic, invoke a Lambda function, or create a Systems Manager OpsItem. It cannot perform an EC2 or Auto Scaling action, so an alarm that also has to scale a group stays a metric alarm.
• AND requires both children in ALARM at the same moment, which is a stronger statement than either metric alone and a coarser net: a single-signal outage passes through it in silence. AT_LEAST(M, STATE, (alarms)) is the M-of-N middle ground.
• After creation a composite never returns to INSUFFICIENT_DATA, but a child stuck there makes ALARM() on that child false, so the rule can read healthy while a metric has stopped reporting.
• DefaultValue: 0 fills a period where logs arrived and nothing matched. It publishes nothing when no logs arrive at all, which is why TreatMissingData decides the child’s state through a genuine lull.
• Every child sits in the same account and Region as the composite, unless the composite lives in a CloudWatch cross-account observability monitoring account. Across an estate, that is where an aggregate health alarm belongs.

Next

The rest of the DevOps Pro lab arc is in the track’s README.