DOP Lab 03 - A composite alarm that only pages when it matters

Scaffold: 3/5. The log group, both metric filters, both child alarms, the SNS topic, and the composite alarm are all built. What is missing is the two filter patterns that turn log lines into metrics, and the composite’s rule that decides when to page. You write those; everything else is wired.

The scenario

One signal is flappy. A single metric filter counting ERROR lines trips its alarm at 3am because a downstream retry logged a handful of errors and then recovered on its own. Nobody needed to wake up, but the pager went off anyway, and after a few nights of that the team starts ignoring it. That is the failure mode: an alarm that fires when nothing is actually wrong trains people to distrust every alarm.

The fix is to page only when several signals agree the service is genuinely down. Errors alone happen. Timeouts alone happen. Errors and timeouts breaching at the same time is a much stronger statement that something is broken. A CloudWatch composite alarm expresses exactly that: it watches other alarms and fires on a boolean rule over their states, so you can say “page me only when both of these are in ALARM,” and give the SNS action to the composite rather than to either noisy child.

The requirement

A log group receives application lines. Two metric filters count ERROR and TIMEOUT lines into custom metrics (ErrorCount and TimeoutCount in the AcmeLab namespace). Two child alarms, ErrorCountHigh and TimeoutHigh, each watch one metric. The composite alarm AcmeServiceDown must page the SNS topic only when both child alarms are in ALARM at once, so a burst of errors with no timeouts, or a run of timeouts with no errors, never pages.

What’s provided

src/template.yaml — the log group, both metric filters, both child alarms, the SNS topic, and the composite alarm, plus the stack Outputs. Two clearly marked TODO blocks hold the gap: the metric filters ship with placeholder FilterPattern values that match nothing real, and the composite AlarmRule watches a single child.
scripts/ — deploy, test, and teardown.
solution/template.yaml — the complete, correct template.

Deploying src/template.yaml as shipped succeeds, but it behaves wrong. The placeholder patterns mean ErrorCount and TimeoutCount never move, so nothing alarms; and even once you fix the patterns, the composite’s rule still watches only ErrorCountHigh, so a burst of errors with no timeouts would page. That is the noisy alarm the scenario is trying to kill.

Your task

Open src/template.yaml and close both gaps:

Write the two filter patterns. ErrorMetricFilter should match any log event containing ERROR; TimeoutMetricFilter should match any containing TIMEOUT. For an unstructured line the term pattern is the quoted word, e.g. FilterPattern: '"ERROR"'.
Write the composite AlarmRule. Combine the two children so the composite fires only when both are in ALARM at the same time: ALARM("ErrorCountHigh") AND ALARM("TimeoutHigh").

Run it

# Defaults: stack dop-lab-03, region ap-southeast-2.
./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # drives the log group and watches the composite
./scripts/teardown.sh        # deletes everything

test.sh never launches compute, so it stays cheap. It writes log lines with put-log-events and polls describe-alarms. First it writes ERROR lines only and confirms the composite stays out of ALARM while timeouts are absent; then it writes ERROR and TIMEOUT lines together and confirms the composite pages. Metric filters and alarms take a minute or three to react, so the test polls patiently and prints each reading.

What success looks like

./scripts/test.sh ends with:

  t+40s  error-child=ALARM  composite=OK
  ...
  t+90s  error-child=ALARM  timeout-child=ALARM  composite=ALARM
PASS: the composite stayed calm on errors alone and paged only once both
signals breached. That is a page worth waking up for.

If it fails

The composite paged during the errors-only phase. Its AlarmRule is watching one child. Combine both with AND so a single signal cannot page.
The composite never paged, even with both signals breaching. Either the filter patterns still match nothing (the shipped CHANGE_ME placeholders never move the metrics), or the rule does not include both children. Check both.
A child alarm sits in INSUFFICIENT_DATA. The metric filter is not producing data. Confirm the FilterPattern matches the lines the test writes, and that the metric transformation keeps its DefaultValue: 0.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

A composite alarm watches other alarms, not metrics. Its AlarmRule is a boolean expression over child alarm states, and it can have its own actions.
Giving the SNS action to the composite, not to the noisy children, is how you suppress a single flappy signal: the child still trips and shows red on a dashboard, but only the agreed-upon combination pages a human.
AND requires the children to be in ALARM at the same time. That correlation is a far stronger signal than either metric alone, which is why it earns the right to wake someone.
A metric filter with DefaultValue: 0 keeps its metric reporting through quiet periods, so the child alarms resolve to OK instead of stalling in INSUFFICIENT_DATA waiting for the first data point.

The rest of the hands-on lab tracks are listed in labs/README.md.