The situation
A booking platform commits to 99.95% availability in its customer contracts, which allows roughly 22 minutes of error a month. The service runs on ECS behind an Application Load Balancer, deploys about eight times a week, and has spent the last three months burning most of the budget on deployments rather than on infrastructure failures.
The deployment strategy is a rolling update with a 50% minimum healthy percentage. When a release is bad, half the fleet serves errors until somebody notices, decides, and triggers a rollback, which historically takes eleven minutes. Two bad releases a month is 22 minutes, which is the entire budget.
The team’s proposal is to deploy less often. The business objects, because the release cadence is what lets them respond to competitors. The requirement is to keep eight deploys a week and stop them consuming the budget.
What actually matters
The first thing that matters is that a deployment strategy is a bet about detection time. Rolling exposes a fraction of traffic and relies on somebody noticing; canary exposes a small fraction and can decide automatically; blue/green exposes none until the switch and then all of it. What separates them in budget terms is how many user-facing errors occur between the bad code starting and the traffic stopping.
The second is that the eleven minutes is the actual defect. Whatever strategy is chosen, a human in the detection loop makes the exposure a function of attention. Automating the decision is worth more than changing the exposure fraction, because it turns eleven minutes into ninety seconds regardless of strategy.
The third is that the alarm the rollback keys on has to be fast and specific. An alarm on a five-minute average error rate cannot trigger a rollback in ninety seconds, and an alarm on overall service health will not distinguish the new version from the old. The metric has to be attributable to the deployment.
The fourth is that not every change carries the same risk, and applying the slowest strategy to all of them makes the pipeline slow enough that people route around it. A configuration change and a database migration are different bets, and a strategy per risk class is more useful than one strategy per service.
Underneath it, an error budget is a decision tool rather than a report. Its use here is to make the choice between strategies arithmetic instead of a preference.
What we’ll filter on
- How much traffic sees a bad release, and for how long?
- Is the decision to roll back automatic, and what does it key on?
- How long does the rollback itself take?
- What does it cost in infrastructure during the deployment?
- Can the database schema tolerate two versions running at once?
- Does the strategy scale to eight deploys a week without a human per deploy?
The landscape
Rolling update. Replaces instances or tasks in batches, controlled by minimum healthy percentage and maximum percent. Cheap, because no additional capacity is needed beyond the overlap, and it exposes a fraction of traffic proportional to how far the roll has progressed. Rollback means rolling forward to the previous version, which takes as long as the deployment did.
Blue/green. Stands up a complete second environment, tests it, then shifts traffic. Exposure before the shift is zero, and after the shift it is everything. Rollback is a traffic shift back, which is fast. The cost is double capacity during the deployment, and the constraint is that both versions must be able to share the database.
Canary. Shifts a small percentage of traffic to the new version, holds, evaluates, and then either proceeds or rolls back. This is blue/green with a graduated shift, and it is the strategy that limits exposure and provides an evaluation window in the same mechanism. CodeDeploy supports canary and linear configurations for ECS and Lambda with an automatic rollback on alarm.
CodeDeploy deployment configurations. The named strategies: AllAtOnce, HalfAtATime, OneAtATime for rolling, and the canary and linear variants for the traffic-shifting ones. The specific value here is the alarm-based automatic rollback, which removes the human from the detection loop.
CloudWatch alarms wired to rollback. The trigger. What matters is that the alarm is fast, specific to the new version, and unlikely to be caused by something else. A per-target-group 5xx rate over one minute is fast and attributable; a service-wide latency average over five minutes is neither.
Feature flags. Decouple deployment from release: ship the code dark, enable it for a fraction of users, and disable it without a deployment. AppConfig provides this with its own gradual rollout and alarm-based rollback, and the rollback path is a configuration change rather than a redeployment, which is seconds rather than minutes.
Lambda weighted aliases. For Lambda specifically, an alias can route a percentage of invocations to a new version, which is the canary mechanism without a load balancer.
Database migration strategy. The constraint underneath all traffic-shifting strategies. Two versions running simultaneously means the schema must be compatible with both, which is the expand-and-contract pattern: add the new column, deploy code that writes both, backfill, deploy code that reads the new, then remove the old. Without it, blue/green and canary are unavailable regardless of the deployment tooling.
Evaluation
Side by side
| Strategy | Traffic exposed to a bad release | Rollback time | Extra capacity | Auto-rollback |
|---|---|---|---|---|
| Rolling, 50% healthy | Up to 50%, until detected | Full redeploy | None | Via alarm, slow |
| Blue/green | 0% then 100% at the shift | Seconds, shift back | 2× during deploy | ✓ on alarm |
| Canary 10% for 5 min | 10% for the canary window | Seconds, shift back | 2× during deploy | ✓ on alarm |
| Linear 10% every minute | Ramps from 10% | Seconds | 2× during deploy | ✓ on alarm |
| Feature flag | Whatever fraction is flagged | Seconds, no deploy | None | ✓ via AppConfig |
Putting the current situation in the table’s terms explains the three months: up to 50% of traffic, for eleven minutes, twice a month. A canary at 10% for five minutes with automatic rollback exposes a twentieth of the users for a quarter of the time, which is roughly a fortieth of the error minutes.
The solution
Move to canary with alarm-based automatic rollback, fix the alarm so it can fire in a minute, and put the riskiest changes behind feature flags so the deployment and the release are separate events.
Start with the alarm, because every strategy on the page depends on it and the current one cannot support any of them. Create a per-target-group 5xx rate alarm with a one-minute period and a low evaluation threshold, and a latency alarm on the same basis. These are attributable to the new version because the canary target group is separate, which is what makes an automatic decision safe.
Then switch to CodeDeploy blue/green for ECS with a canary configuration: 10% of traffic for five minutes, then the remainder. Register both alarms with the deployment group so a breach during the canary window triggers an automatic rollback, which is a traffic shift back to the original target group and takes seconds. The eleven minutes of human detection disappears from the arithmetic entirely.
Accept the capacity cost. A blue/green deployment runs double the tasks for the duration, and on eight deploys a week that is a real number. It is smaller than the cost of the error budget being spent, and framing it that way is what gets it approved.
Handle the database with expand-and-contract as a standing practice rather than a per-migration decision. Two versions run simultaneously during every canary, so a migration that is not backward-compatible breaks the strategy rather than merely being risky. Making it a review checklist item is cheaper than discovering it during a deploy.
Put the riskiest changes behind AppConfig feature flags, so the deployment ships inert code and the release is a configuration change with its own gradual rollout and its own alarm-based rollback. This is the strategy that gets exposure and rollback below what any deployment mechanism can offer, because the rollback is a flag flip rather than a traffic shift, and it separates “the code is deployed” from “the behaviour is on”.
Then classify changes so the pipeline is not uniformly slow. Configuration and content changes take a faster path; anything touching the request handling takes the full canary; anything touching the schema takes the canary and a migration review. Eight deploys a week only works if most of them are cheap.
Finally, report the budget. A CloudWatch dashboard showing error minutes consumed against the monthly allowance, split by cause, turns the strategy discussion into a measurement. If deployments stop being the largest consumer, the next conversation is about something else, which is the outcome.
Why not deploy less often, as the team proposed. It reduces the number of bad releases and increases the size of each one, because a week of changes ships together and the failure is harder to attribute. It also removes the capability the business is paying for.
Why not blue/green without the canary step. The shift is all-or-nothing, so a bad release reaches 100% of traffic for however long the alarm takes to fire. The canary window is what limits exposure while the evaluation happens.
Worked example
The alarm work takes two days and is the least interesting and most important part. The existing alarm was a five-minute average across the whole service; the replacement is a one-minute 5xx rate per target group with a threshold set from three months of baseline data.
The first canary deployment rolls back automatically eleven days later. A change to a serialisation library broke one endpoint, the canary target group’s 5xx rate crossed the threshold 40 seconds in, and CodeDeploy shifted traffic back. Total user-facing errors: roughly 10% of traffic for 90 seconds, against 50% for eleven minutes under the old strategy. That single event is about 4% of the previous budget consumption.
The expand-and-contract discipline catches a problem in review the following month: a migration dropping a column that the outgoing version still reads. Under a rolling deployment it would have half-worked; under a canary it would have broken the old version while the new one was fine, which is a confusing failure to debug at speed.
Feature flags go on the two changes classified as highest risk that quarter, including a pricing calculation change. It ships dark, is enabled for 1% of users, sits there for a day, and ramps over a week. Nothing goes wrong, which is unsatisfying and is the intended experience.
Three months on, deployments account for about 2.5 minutes of the monthly budget rather than 22. The cadence is nine deploys a week rather than eight, because the pipeline got faster and nobody was avoiding it.
What’s worth remembering
- A deployment strategy is a bet about detection time, and the budget cost is the traffic exposed multiplied by how long it stays exposed.
- The human in the detection loop is usually the largest term. Automating the rollback decision is worth more than changing the exposure fraction, because it shrinks minutes to seconds regardless of strategy.
- The alarm has to be fast and attributable to the new version: a per-target-group error rate over one minute can drive an automatic rollback, and a service-wide five-minute average cannot.
- Traffic-shifting strategies require both versions to share a database, so expand-and-contract migrations are a prerequisite rather than a refinement.
- Feature flags separate deployment from release, and the rollback is a configuration change rather than a traffic shift, which is the fastest option available.
- Classify changes by risk so the pipeline is not uniformly slow, because a strategy that makes every deploy expensive reduces the deploy rate whether or not that was the intent.