Lab: Canary-Deploy a Lambda With Automatic Rollback

This is a lab in the DevOps Pro hands-on track. Safe deployment is one of the domain’s recurring shapes: ship the new version to a slice, watch it, and undo it automatically if it misbehaves. The reference posts argue when a canary beats a straight cutover; this stands one up. You build the smallest honest slice, a Lambda behind an alias that CodeDeploy shifts a little at a time, with an alarm that pulls it back. The full lab is in lab-dop-04-lambda-canary.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A bad Lambda release used to take down 100% of traffic the instant it shipped. Update the function, every request hits the new code at once, and if that code throws, everyone gets the error until a human notices and rolls it back by hand.

A canary deployment shifts a slice first. CodeDeploy moves 10% of traffic to the new version, waits five minutes, then shifts the rest; the config that does this is CodeDeployDefault.LambdaCanary10Percent5Minutes. During the five-minute bake, a CloudWatch alarm watches the new version’s error rate. If it fires, CodeDeploy stops the deployment and rolls the alias back to the old version before the other 90% is ever exposed. The slice is the blast radius, and the alarm is the tripwire.

The design that scores on a Professional paper is the pairing, not either half alone. A deployment group can run the canary schedule and still never roll back, because the schedule and the tripwire are separate settings. If nothing is watching, a canary that errors just shifts slowly to 100% instead of quickly. The safety comes from wiring the alarm and the auto-rollback onto the group.

What you’re given

CloudFormation builds the function (a trivial handler that returns 200), an explicit published version and a live alias in front of it, a CloudWatch alarm on the function-and-alias Errors metric, a CodeDeploy application for Lambda, and a deployment group that already carries the canary DeploymentConfigName and a service role. Traffic reaches the function only through the alias, so a deployment just has to move the alias from one version to the next.

The gap is on the deployment group. As shipped it runs the canary but watches nothing:

CanaryDeploymentGroup:
  Type: AWS::CodeDeploy::DeploymentGroup
  Properties:
    ApplicationName: !Ref CanaryApplication
    ServiceRoleArn: !GetAtt CodeDeployServiceRole.Arn
    DeploymentConfigName: CodeDeployDefault.LambdaCanary10Percent5Minutes
    # TODO: the alarm and the auto-rollback go here.

Deploying it as shipped succeeds, and a canary even runs to completion, but a canary that errored would keep shifting to 100%: nothing is watching the alarm, and nothing rolls back.

Your task

Fill in the TODO with two properties on the deployment group. The first points the deployment at the Errors alarm; the second tells CodeDeploy to undo the deployment when it fails or the alarm fires:

AlarmConfiguration:
  Enabled: true
  Alarms:
    - Name: !Ref ErrorAlarm

AutoRollbackConfiguration:
  Enabled: true
  Events:
    - DEPLOYMENT_FAILURE
    - DEPLOYMENT_STOP_ON_ALARM

The two close the loop. AlarmConfiguration makes the deployment watch the alarm while the canary slice is live; AutoRollbackConfiguration shifts the alias back to the old version on DEPLOYMENT_STOP_ON_ALARM. Without the first, the alarm can go red and CodeDeploy will not notice. Without the second, CodeDeploy notices and stops, but leaves the alias where it landed instead of returning it. The alarm itself is scoped to the function and the alias with a Resource dimension, so it measures the errors of the version taking canary traffic rather than the whole function.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # runs a real canary and checks the group config
./scripts/teardown.sh        # deletes the stack and the log group

test.sh takes about five minutes, deliberately. It publishes a new, healthy function version, hands CodeDeploy an AppSpec that shifts the live alias to it, and polls until the deployment finishes. This is a genuine LambdaCanary10Percent5Minutes shift, so 10% of traffic moves first, then the rest after the five-minute bake. Once it succeeds, the test confirms the alias now points at the new version, then reads the deployment group back and asserts the alarm and auto-rollback are wired:

  ok: canary completed and the live alias now points at version 2
  ok: deployment group watches the Errors alarm (dop-lab-04-canary-errors)
  ok: deployment group auto-rolls back on DEPLOYMENT_STOP_ON_ALARM
PASS: the canary shifted a healthy release to completion, and the deployment
group is wired to roll back a bad canary automatically (alarm + auto-rollback).

The test deploys a healthy change on purpose. A forced failing canary is flaky and slow to assert, so the test proves the mechanics on a version that returns 200 and inspects the group’s config to prove the rollback safety is in place.

If it fails

• deployment group has no alarm configuration. The AlarmConfiguration block is missing. Add it with Enabled: true and the Errors alarm listed under Alarms, then redeploy.
• deployment group has no enabled auto-rollback. The AutoRollbackConfiguration block is missing or not enabled. Add it with Enabled: true and DEPLOYMENT_STOP_ON_ALARM in Events.
• The canary deployment itself failed. A handler that returns 200 should shift cleanly, so a failure here usually means the alias or service role wiring was changed. Check the deployment in the CodeDeploy console for the reason.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A canary for Lambda shifts alias traffic in stages. With LambdaCanary10Percent5Minutes, 10% goes first and the rest follows after five minutes, so a bad version only reaches a slice before it can be caught.
• The canary schedule and the rollback tripwire are separate settings. DeploymentConfigName decides how traffic moves; AlarmConfiguration and AutoRollbackConfiguration decide what is watched and what happens when it goes wrong. A group can have the first and still never roll back.
• Scope the alarm to the function and the alias with the Resource dimension, so it measures the version taking canary traffic instead of averaging the errors across every version behind the function.
• CodeDeploy shifts an alias, never the raw function. Traffic flows through live, a deployment moves the alias from the current version to the target, and a rollback is just moving it back.

Next

The rest of the DevOps Pro lab track is in the track’s README.