Lab: Blue/Green Traffic Shifting With a Weighted Forward

This is the fourth lab in the SA Pro hands-on track. The reference posts argue the decisions; this stands one up. Traffic shifting in steps is the mechanism behind safe deployments and behind the strangler-fig routing facade alike: two routable fleets, one dial. Here you build the dial. The full lab is in lab-sa-pro-04-blue-green.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. Every SA Pro lab tags its stack for the reaper on deploy.

Everything runs in ap-southeast-2, and unlike most of these labs this one is not free while it stands. An Application Load Balancer is charged per load-balancer-hour, USD$0.0252 on the Sydney list price with a partial hour billed as a full one, plus USD$0.008 per capacity unit-hour. Capacity units are measured on whichever of four traffic dimensions is highest, averaged over the hour, and a hundred requests come to a rounding error against all four, so the hours are the bill: roughly USD$0.60 if a forgotten stack runs the reaper’s full 24 hours. Nothing else in the stack is worth counting. Run teardown.sh.

The scenario

A service is live behind an Application Load Balancer. The team has a new version, green, deployed and tested but dark: it has never seen a production request, and nobody wants its first bug found at 100% of traffic. The blue fleet answers everything.

The canary answer is to give green a small slice of real traffic and watch it. The ALB’s weighted forward is the dial: a listener action that splits requests across target groups by weight, up to five of them, each weight an integer from 0 to 999. Shift 90/10, watch the errors and latency, then 50/50, then 0/100, and blue is drained. If green misbehaves at any step, rollback is the same weight change in reverse: one modify-listener call on the default action, with no redeploy and no image rebuild. Designing the shift is designing the way back, and here the way back is an API call rather than a pipeline run.

The two versions in this lab are Lambda functions that announce their colour, because the lesson is the routing, not the fleet. The same listener mechanic shifts traffic between two auto-scaling groups, two ECS services, or an on-prem target and a cloud one mid-migration.

What you’re given

CloudFormation builds a small VPC (an ALB needs two subnets in two AZs), the internet-facing load balancer, the blue and green functions, invoke permissions for both, and blue’s target group. The listener forwards 100% of traffic to blue: the “before” of the cutover. The gap in src/template.yaml is a TODO block where green’s target group and the weighted forward go. The forward is the part worth working out: the listener’s default action swaps its TargetGroupArn shorthand for a ForwardConfig that lists both target groups with weights, plus a stickiness setting.

Deploying the stack as shipped succeeds, and every single request comes back blue.

Your task

Two edits. First, give green a target group mirroring blue’s: TargetType: lambda, the green function’s ARN as its one target (a lambda target group holds exactly one function), and a DependsOn the invoke permission, because the target group registers the function the moment it is created. Second, rewrite the listener’s default action as a weighted forward, 90 blue to 10 green. The weights are ratios, not percentages. Two of them summing to 100 read as percentages at a glance, and they are still plain integers anywhere in the 0 to 999 range.

Target group stickiness is off by default, so the Enabled: false line writes the default down rather than switching anything off. Turning it on changes the routing rule itself. The load balancer sets an encrypted AWSALBTG cookie naming the target group it picked, and a later request carrying that cookie goes to that group whatever the weights now say. That is what you want when a target group has sticky sessions of its own, and the trade is your measurement: the observed split stops tracking the weights and starts tracking your returning-visitor mix. For a canary, every request should be a fresh draw.

Run it

./scripts/deploy.sh          # deploys src/template.yaml
./scripts/test.sh            # sends 100 requests and counts the colours
./scripts/teardown.sh        # deletes everything

test.sh polls the fresh load balancer for up to two minutes, then sends 100 requests and counts which colour answers each one. With the unedited template it reports 100 blue and explains why; with the weighted forward in place it prints:

blue:  91   (expected roughly 90)
green: 9   (expected roughly 10)

PASS: the canary is live. Green is taking a minority share of real traffic,
and shifting further (or rolling back) is now a weight change, not a redeploy.

Ten in a hundred is an average, not a guarantee. The green count has a standard deviation of three, so single digits through the mid-teens is an ordinary run. test.sh passes anything from 2 to 30 greens with blue ahead. Zero means the forward is not weighted.

If it fails

• Every request is blue. The listener is still using the shorthand TargetGroupArn action, or the green target group is missing. Both halves of the TODO are needed.
• The stack update fails on the listener. ForwardConfig and TargetGroupArn may coexist only when the ForwardConfig names that same single target group. Leaving the old shorthand line in place beside a two-group forward is rejected outright rather than half-applied, so delete it.
• The split is wildly off. Check the two weights first. If you switched stickiness on while experimenting, the distribution stops tracking them.
• The load balancer never answers. Give it another minute (fresh ALBs take a little while to provision and propagate DNS), then check the stack events.

Reveal the solution

Work out the forward yourself first; it is a small block, and writing it is what the lab is for. When you want the reference answer, deploy it without editing anything:

SRC=solution ./scripts/deploy.sh && ./scripts/test.sh

Or unfold it here:

DefaultActions:
  - Type: forward
    ForwardConfig:
      TargetGroups:
        - TargetGroupArn: !Ref BlueTargetGroup
          Weight: 90
        - TargetGroupArn: !Ref GreenTargetGroup
          Weight: 10
      TargetGroupStickinessConfig:
        Enabled: false

What you just learned

• Blue/green needs two routable fleets and one dial. The weighted forward is the dial: traffic moves by changing two integers in the 0 to 999 range, and rollback is the same change in reverse. A forward action holds at most five target groups, and that quota is not adjustable.
• The canary step exists so the new version’s first bug is found at 10% of traffic. Watch green’s errors and latency at each step before shifting more.
• Target group stickiness is off by default. Switch it on and the AWSALBTG cookie pins a returning client to the group it first reached, so the observed split stops tracking the weights.
• A weighted forward does not fail away from a target group that is empty or holds only unhealthy targets. In production the alarm on green’s 5XX and unhealthy-host counts has to exist before the first request shifts, and the weight change has to be a one-command runbook step.
• Target groups decouple what the fleet is from how traffic reaches it. The same forward shifts between Lambda functions, auto-scaling groups, ECS services, or IP targets, which is why the pattern appears in migration cutovers as often as deployments.

Next

The rest of the SA Pro lab arc is in the track’s README: immutable replicated backups with S3 Object Lock, and tag-driven cost visibility.