DOP Lab 04 - Canary-deploy a Lambda with automatic rollback

Scaffold: 3/5. The function, its published version and live alias, the Errors alarm, the CodeDeploy application, and a deployment group with the canary config and a service role are all built. You wire the two things that make a bad release safe: the alarm configuration and the auto-rollback configuration on the deployment group.

The scenario

A bad Lambda release used to take down 100% of traffic the instant it shipped. Update the function, and every request hits the new code at once; if the new code throws, everyone gets the error until someone notices and rolls it back by hand.

A canary deployment shifts a slice first. CodeDeploy moves 10% of traffic to the new version, waits five minutes, then shifts the rest (CodeDeployDefault.LambdaCanary10Percent5Minutes). During that five-minute window a CloudWatch alarm watches the new version’s error rate, and if it fires, CodeDeploy rolls the alias back to the old version before the other 90% is ever exposed. The slice is the blast radius, and the alarm is the tripwire.

The catch is that the canary schedule and the tripwire are separate settings. A deployment group can run the canary and still never roll back, because nothing is watching. In this lab the canary config and service role are already in place; what is missing is the alarm and the auto-rollback that turn a slow shift into a safe one.

What’s provided

• src/template.yaml — the Lambda function (a trivial handler that returns 200), an explicit AWS::Lambda::Version and a live alias in front of it, a CloudWatch alarm on the function+alias Errors metric, an AWS::CodeDeploy::Application for Lambda, and an AWS::CodeDeploy::DeploymentGroup with the canary DeploymentConfigName and a service role. There is a clearly-marked TODO block on the deployment group where the alarm and rollback wiring goes.
• scripts/ — deploy, test, and teardown.
• solution/template.yaml — the complete, correct template.

Deploying src/template.yaml as shipped succeeds, and a canary deployment even runs to completion, but a canary that errored would keep shifting to 100%: nothing is watching the alarm, and nothing rolls back.

Your task

Open src/template.yaml and fill in the TODO block on CanaryDeploymentGroup with two properties:

• AlarmConfiguration — Enabled: true, with Alarms listing the Errors alarm (- Name: !Ref ErrorAlarm). This makes the deployment watch the alarm while the canary slice is live.
• AutoRollbackConfiguration — Enabled: true, with Events including DEPLOYMENT_FAILURE and DEPLOYMENT_STOP_ON_ALARM. This tells CodeDeploy to shift the alias back to the old version when the deployment fails or the alarm fires.

Together they close the loop: a canary that starts erroring trips the alarm, CodeDeploy stops the deployment on the alarm, and auto-rollback returns the alias to the version that was working.

If you want the mechanics first, read the walk-through post linked at the bottom.

Run it

# Defaults: stack dop-lab-04, region ap-southeast-2.
./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # runs a real canary and checks the group config
./scripts/teardown.sh        # deletes everything

test.sh takes about five minutes. It publishes a new (healthy) function version, hands CodeDeploy an AppSpec that shifts the live alias to it, and polls until the canary deployment succeeds — a genuine LambdaCanary10Percent5Minutes shift, 10% first then the rest after the five-minute bake. It then confirms the alias points at the new version, and inspects the deployment group to assert the alarm and auto-rollback are wired.

What success looks like

./scripts/test.sh prints:

  ok: canary completed and the live alias now points at version 2
  ok: deployment group watches the Errors alarm (dop-lab-04-canary-errors)
  ok: deployment group auto-rolls back on DEPLOYMENT_STOP_ON_ALARM
PASS: the canary shifted a healthy release to completion, and the deployment
group is wired to roll back a bad canary automatically (alarm + auto-rollback).

The test never forces a failing canary — a forced rollback is flaky and slow to assert. It proves the mechanics on a healthy change and inspects the group’s config to prove the rollback safety is in place.

If it fails

• deployment group has no alarm configuration. The AlarmConfiguration block is missing from the deployment group. Add it with Enabled: true and the Errors alarm listed under Alarms.
• deployment group has no enabled auto-rollback. The AutoRollbackConfiguration block is missing or not enabled. Add it with Enabled: true and DEPLOYMENT_STOP_ON_ALARM in Events.
• The canary deployment itself failed. Check the deployment in the CodeDeploy console; a healthy handler that returns 200 should shift cleanly, so a failure here usually means the service role or alias wiring was changed.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A canary deployment for Lambda shifts alias traffic in stages (LambdaCanary10Percent5Minutes: 10% first, then the rest after five minutes), so a bad version only reaches a slice before it can be caught.
• The canary schedule and the rollback tripwire are separate settings. A deployment group can run the canary and never roll back, because AlarmConfiguration and AutoRollbackConfiguration are what make it watch and undo.
• The alarm is scoped to the function and alias with the Resource dimension, so it measures the errors of the version taking canary traffic, not the whole function.
• CodeDeploy shifts an alias, never the raw function. Traffic flows through the live alias, so a deployment only has to move the alias from the current version to the target version, and a rollback moves it back.

Next

The rest of the DOP lab track is listed in labs/README-dop.md.