Lab: Blue/Green Traffic Shifting With a Weighted Forward

This is the fourth lab in the SA Pro hands-on track. The reference posts argue the decisions; this stands one up. Traffic shifting in steps is the mechanism behind safe deployments and behind the strangler-fig routing facade alike: two routable fleets, one dial. Here you build the dial. The full lab is in lab-sa-pro-04-blue-green.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every SA Pro lab tags its stack for the reaper on deploy.

The scenario

A service is live behind an Application Load Balancer. The team has a new version, green, deployed and tested but dark: it has never seen a production request, and nobody wants its first bug found at 100% of traffic. The blue fleet answers everything.

The canary answer is to give green a small slice of real traffic and watch it. The ALB’s weighted forward is the dial: a listener action that splits requests across two target groups by weight. Shift 90/10, watch the errors and latency, then 50/50, then 0/100, and blue is drained. If green misbehaves at any step, rollback is the same weight change in reverse. No redeploy, no rebuild, seconds not minutes. That reversibility is what earns the pattern its place on a Professional paper: the design question is never just “how do I ship it” but “how do I un-ship it while the pager is quiet”.

The two versions in this lab are Lambda functions that announce their colour, because the lesson is the routing, not the fleet. The same listener mechanic shifts traffic between two auto-scaling groups, two ECS services, or an on-prem target and a cloud one mid-migration.

What you’re given

CloudFormation builds a small VPC (an ALB needs two subnets in two AZs), the internet-facing load balancer, the blue and green functions, invoke permissions for both, and blue’s target group. The listener forwards 100% of traffic to blue: the “before” of the cutover. The gap in src/template.yaml is a TODO block where green’s target group and the weighted forward go. The forward is the part worth working out: the listener’s default action swaps its TargetGroupArn shorthand for a ForwardConfig that lists both target groups with weights, plus a stickiness setting.

Deploying the stack as shipped succeeds, and every single request comes back blue.

Your task

Two edits. First, give green a target group mirroring blue’s: TargetType: lambda, the green function’s ARN as its one target, and a DependsOn the invoke permission, because the target group registers the function the moment it is created. Second, rewrite the listener’s default action as a weighted forward, 90 blue to 10 green. The weights are relative rather than percentages, so 90 and 10 out of a sum of 100 keeps them readable at a glance.

The stickiness line matters more than it looks. With stickiness on, a returning client is pinned to whichever side answered it first, and the observed split drifts away from the weights: your “10% canary” quietly becomes whatever your returning-visitor mix makes it. For a canary you want every request to roll the dice.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # sends 100 requests and counts the colours
./scripts/teardown.sh        # deletes everything

test.sh waits for the fresh load balancer to start answering (a minute or two), then sends 100 requests and counts who replies. With the unedited template it reports 100 blue and explains why; with the weighted forward in place it prints:

blue:  91   (expected roughly 90)
green: 9    (expected roughly 10)

PASS: the canary is live. Green is taking a minority share of real traffic,
and shifting further (or rolling back) is now a weight change, not a redeploy.

Ten percent of a hundred requests is a coin-flip count, so anywhere from a couple of greens to the low twenties is a healthy canary; zero means the forward is not weighted.

If it fails

• Every request is blue. The listener is still using the shorthand TargetGroupArn action, or the green target group is missing. Both halves of the TODO are needed.
• The split is wildly off. Check the two weights, and check stickiness is disabled; with it enabled the distribution stops tracking the weights.
• The load balancer never answers. Give it another minute (fresh ALBs take a little while to provision and propagate DNS), then check the stack events.

Reveal the solution

Work out the forward yourself first; it’s a small block and the whole lesson. When you want the reference answer, deploy it without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

Or unfold it here:

DefaultActions:
  - Type: forward
    ForwardConfig:
      TargetGroups:
        - TargetGroupArn: !Ref BlueTargetGroup
          Weight: 90
        - TargetGroupArn: !Ref GreenTargetGroup
          Weight: 10
      TargetGroupStickinessConfig:
        Enabled: false

What you just learned

• Blue/green needs two routable fleets and one dial. The weighted forward is the dial: traffic moves by changing two integers, and rollback is the same change in reverse.
• The canary step exists so the new version’s first bug is found at 10% of traffic. Watch green’s errors and latency at each step before shifting more.
• Stickiness and canaries don’t mix: pinned clients skew the observed split away from the weights.
• Target groups decouple what the fleet is from how traffic reaches it. The same forward shifts between Lambda functions, auto-scaling groups, ECS services, or IP targets, which is why the pattern appears in migration cutovers as often as deployments.

Next

The rest of the SA Pro lab arc is in the track’s README: tag-driven cost visibility, and a DMS change-data-capture migration.