Lab: Canary-Deploy a Lambda With Automatic Rollback

This is a lab in the DevOps Pro hands-on track. Safe deployment is one of the domain’s recurring shapes: ship the new version to a slice, measure it, and undo it automatically if the errors rise. The reference posts argue when a canary beats a straight cutover; this stands one up. You build the smallest honest slice, a Lambda behind an alias that CodeDeploy shifts a little at a time, with an alarm that pulls it back. The full lab is in lab-dop-04-lambda-canary.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every lab tags its stack for the reaper on deploy.

The scenario

A bad Lambda release used to take down 100% of traffic the instant it shipped. Point the alias at the new version and every request hits the new code at once. If that code throws, everyone gets the error until a human notices and moves the alias back by hand.

A canary deployment shifts a slice first. CodeDeployDefault.LambdaCanary10Percent5Minutes shifts 10 percent of traffic in the first increment and deploys the remaining 90 percent five minutes later. Underneath it is the alias’s weighted routing configuration: live points at the old version, with the new one carrying 10% of invocations. A CloudWatch alarm on the alias’s Errors count runs through that window. If an associated alarm is activated, the deployment stops and its status becomes Stopped. The auto-rollback configuration then redeploys the last successful revision, which returns the whole alias to the old version before the other 90% is exposed.

A deployment group can run the canary schedule and still never roll back. The schedule and the alarm wiring are separate settings. With no alarm associated, nothing on the group detects a bad version, and an erroring canary shifts slowly to 100% instead of quickly. Both halves have to be set, and the alarm has to resolve inside the bake window. Lambda sends metric data to CloudWatch in one-minute intervals, so a one-minute period over one evaluation period is the tightest alarm worth building here. Five minutes of bake leaves it room to transition. An alarm on a fifteen-minute period would go red long after the rest of the traffic had moved.

What you’re given

CloudFormation builds the function (a trivial handler that returns 200), an explicit published version and a live alias in front of it, a CloudWatch alarm on the function-and-alias Errors metric, a CodeDeploy application for Lambda, and a deployment group that already carries the canary DeploymentConfigName and a service role. That role has the AWS-managed AWSCodeDeployRoleForLambda policy, which grants cloudwatch:DescribeAlarms alongside lambda:UpdateAlias, so reading an alarm is already permitted. Traffic reaches the function only through the alias, so a deployment has to move the alias from one version to the next and nothing else.

The gap is on the deployment group. As shipped it runs the canary but watches nothing:

CanaryDeploymentGroup:
  Type: AWS::CodeDeploy::DeploymentGroup
  Properties:
    ApplicationName: !Ref CanaryApplication
    ServiceRoleArn: !GetAtt CodeDeployServiceRole.Arn
    DeploymentConfigName: CodeDeployDefault.LambdaCanary10Percent5Minutes
    # TODO: the alarm and the auto-rollback go here.

Deploying it as shipped succeeds, and a canary even runs to completion, but a canary that errored would keep shifting to 100%: no alarm is associated, so nothing stops the deployment and nothing rolls it back.

Your task

Fill in the TODO with two properties on the deployment group. The first points the deployment at the Errors alarm; the second tells CodeDeploy to undo the deployment when it fails or the alarm fires:

AlarmConfiguration:
  Enabled: true
  Alarms:
    - Name: !Ref ErrorAlarm

AutoRollbackConfiguration:
  Enabled: true
  Events:
    - DEPLOYMENT_FAILURE
    - DEPLOYMENT_STOP_ON_ALARM

The two close the loop. AlarmConfiguration associates the alarm with the deployment group, so CodeDeploy polls its state for the length of the deployment; AutoRollbackConfiguration redeploys the last successful revision on DEPLOYMENT_STOP_ON_ALARM, which returns the alias to the old version. Without the first, the alarm can go red and CodeDeploy never polls it. Without the second, the alarm stops the deployment and the alias keeps the weights it had reached, leaving 10% of invocations on the bad version. ignorePollAlarmFailure is false unless you set it, so a deployment that cannot retrieve alarm state from CloudWatch stops rather than carrying on blind.

The alarm is scoped with a Resource dimension of dop-lab-04-canary:live, which names the alias, not either version behind it. Every invocation through live lands in that metric, whichever version served it, so through the shift the alarm sums the errors of both. That is still the right trigger: errors from the 10% slice raise the alias total and trip it. What an alias-scoped alarm cannot do is attribute them, so a function already erroring on the old version trips it on every deployment. ExecutedVersion is the dimension AWS publishes for the attribution question, comparing error rates between two versions that are both targets of a weighted alias.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # runs a real canary and checks the group config
./scripts/teardown.sh        # deletes the stack and the log group

test.sh takes about five minutes, deliberately. It publishes a new, healthy function version, hands CodeDeploy an AppSpec that shifts the live alias to it, and polls until the deployment finishes. This is a genuine LambdaCanary10Percent5Minutes shift, so 10% of traffic moves first, then the rest after the five-minute bake. Once it succeeds, the test confirms the alias now points at the new version, then reads the deployment group back and asserts the alarm and auto-rollback are wired:

  ok: canary completed and the live alias now points at version 2
  ok: deployment group watches the Errors alarm (dop-lab-04-canary-errors)
  ok: deployment group auto-rolls back on DEPLOYMENT_STOP_ON_ALARM
PASS: the canary shifted a healthy release to completion, and the deployment
group is wired to roll back a bad canary automatically (alarm + auto-rollback).

The test deploys a healthy change on purpose. A forced failing canary is flaky and slow to assert, so the test proves the mechanics on a version that returns 200 and inspects the group’s config to prove the rollback safety is in place.

If it fails

• deployment group has no alarm configuration. The AlarmConfiguration block is missing. Add it with Enabled: true and the Errors alarm listed under Alarms, then redeploy.
• deployment group has no enabled auto-rollback. The AutoRollbackConfiguration block is missing or not enabled. Add it with Enabled: true and DEPLOYMENT_STOP_ON_ALARM in Events.
• The canary deployment itself failed. A handler that returns 200 should shift cleanly, so a failure here usually means the alias or service role wiring was changed. Check the deployment in the CodeDeploy console for the reason.
• A deployment stops before any traffic shifts. The Errors alarm is probably still in ALARM from an earlier attempt, and an alarm the last failed deployment left red blocks the next one. Wait for it to return to OK, or create the deployment with overrideAlarmConfiguration and polling turned off, which is what AWS documents that parameter for.

Reveal the solution

Deploy the complete reference template without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

What you just learned

• A canary for Lambda shifts alias traffic in two increments. LambdaCanary10Percent5Minutes shifts 10 percent first and deploys the remaining 90 percent five minutes later, so a bad version reaches only a slice before it can be caught.
• The canary schedule and the rollback wiring are separate settings. DeploymentConfigName decides how traffic moves; AlarmConfiguration and AutoRollbackConfiguration decide what is polled and what happens when it goes to ALARM. A group can have the first and still never roll back.
• A Resource dimension of function:alias scopes the metric to the alias, not to either version behind it, so it sums the errors of both through the shift. That still trips on a bad slice. Attribution needs ExecutedVersion, the dimension for comparing two versions that are both targets of a weighted alias.
• The alarm has to resolve faster than the bake window. Lambda emits metric data at one-minute intervals, so a one-minute period over one evaluation period fits inside five minutes, and a long period does not.
• Lambda splits weighted-alias traffic with a probabilistic model, and at low traffic levels the actual split varies widely from the configured one. A 10% canary on a quiet function may serve almost nothing in five minutes, and an alarm with no data is not a safety net.
• CodeDeploy implements a rollback by redeploying the last successful revision as a new deployment with its own deployment ID, which shifts the alias back. A deployment group runs one deployment at a time, so nothing else reaches the group until the rollback finishes.
• The alarm and the rollback events live on the deployment group, which is a per-group setting. Across many accounts and Regions they have to come from the template or stack set that creates the group, or one team’s group ships unguarded.

Next

The rest of the DevOps Pro lab track is in the track’s README.