Making a Running Workload Survive an AZ Failure

The situation

An order management platform has been in production for three years and its diagram has said “Multi-AZ” the whole time. A routine review before a peak trading period found otherwise.

The Auto Scaling group spans three subnets, which sounds right, but all three sit in ap-southeast-2a, because they were cut from a template with the zone hardcoded. The Application Load Balancer is enabled in two zones, since it cannot be created with fewer, and only one of those zones holds a target. There is a single NAT gateway in 2a, and every private subnet routes to it. RDS is a Multi-AZ DB instance deployment, which is the one thing that was already correct. A self-managed Redis instance holding session state runs on one EC2 instance with an attached EBS volume. Two batch workers run on named instances in that same zone, and their Elastic IP addresses are written into a partner’s firewall rules.

The business will accept a change window of two hours on a Sunday, once. Everything else has to happen without an outage. Peak trading starts in five weeks.

What actually matters

“Multi-AZ” is a property of every component in the request path, not of the account and not of the diagram. A load balancer enabled in two zones, in front of instances in one, loses everything when that one zone goes. So the first job is an inventory, not a design. It has to reach the things nobody draws: route tables, mount targets, the subnet list on a Lambda function’s VPC configuration.

Then sort the components into the ones that fail over and the ones that stop. An RDS Multi-AZ DB instance promotes its standby on its own. A single NAT gateway does not. When its zone goes, every private subnet pointing at it loses outbound connectivity, including the subnets in perfectly healthy zones. One shared single-zone dependency turns a zonal event into a total outage for this workload, which is the worst available shape. Surviving the loss of a whole Region is a separate design, and a far more expensive one.

State decides difficulty. Stateless instances behind a load balancer become multi-AZ as soon as the group is given more zones, and nothing goes down. Anything holding state that other components depend on needs that state moved or replicated first. A Redis instance with live sessions, an EBS volume, an address a third party has allow-listed. Data carries a zonal dimension people forget. An EBS volume lives in one zone and cannot be attached across zones. An instance store disappears with its instance. A Regional EFS file system or an S3 bucket stops being part of the problem at all.

A design that should survive a zone failure and one proven to are separate claims, and five weeks before a peak only the second is worth much. Proving it means causing the failure deliberately, more than once, and watching the components the inventory did not list.

What we’ll filter on

1. Is this component single-zone today, and does the request path depend on it?
2. Does it fail over automatically, degrade, or stop?
3. Does it hold state, and is that state zonal?
4. Can the change be made online, or does it need the window?
5. Does anything outside our control depend on a specific address?
6. After the change, how do we prove the failover works?

The landscape

Subnets and the Auto Scaling group. A subnet sits entirely within one zone and cannot span zones, so a group spans the zones of the subnets it holds. Add subnets in the two unused zones and attach them to the group. Auto Scaling handles the redistribution from there. Changing the zones associated with a group is one of the actions that triggers an Availability Zone rebalance, and that rebalance launches replacements before terminating anything. It can push the group temporarily past its maximum capacity to do that, by ten percent or one instance, whichever is larger. An instance maintenance policy narrows that band where the extra instances would be a problem. An instance refresh is a different tool, meant for rolling out a new launch template, and using it here replaces the whole fleet to get a redistribution that rebalancing already performs.

Load balancer zones. An Application Load Balancer must be given at least two subnets, each in a different zone, so a genuinely single-zone ALB is not a configuration that can exist. What can exist is an enabled zone with no registered target. Cross-zone load balancing is always on at the load balancer level for an ALB, and can only be turned off per target group. So the empty zone’s node forwards its whole share of client traffic across into the zone holding the instances. On a Network Load Balancer and a Gateway Load Balancer, cross-zone is off by default. Enabling the third zone is an online change; registering targets in it is what makes the change mean anything.

NAT gateways, zonal and regional. A zonal NAT gateway lives in one zone. AWS’s guidance is to create one per zone and point each private subnet’s route table at the gateway beside it. A regional NAT gateway is the newer shape: one gateway ID used from every zone, no public subnet needed to host it, expanding and contracting across zones as workloads appear and disappear. AWS recommends it for everything except private NAT, which it does not support. It is charged hourly per zone it occupies, so three zones cost about what three zonal gateways cost, and expansion into a newly occupied zone can take up to an hour. Converting from zonal resets existing connections, which puts the conversion in a window. Neither shape changes the per-GB data processing charge; what changes is the cross-zone transfer, which is why AWS’s own cost guidance is to keep resources in the same zone as their gateway. The larger lever on the processing charge is a gateway or interface endpoint for whichever services take most of the traffic.

Aurora and RDS. A Multi-AZ DB instance deployment keeps a synchronous standby in a second zone and fails over by changing the DNS record, typically in 60 to 120 seconds. A Multi-AZ DB cluster runs a writer and two readable replicas across three zones and replicates semisynchronously. It fails over typically in under 35 seconds, though it supports a shorter list of instance classes. An Aurora cluster volume holds copies across three zones regardless of how many instances the cluster runs, and a reader is promoted when the writer fails. In all three the endpoint’s DNS record changes, so a client that caches the old address stays broken after the database has recovered. A JVM with its DNS TTL left at the default is the usual offender.

ElastiCache replication groups. A Valkey or Redis OSS replication group with Multi-AZ enabled keeps at least one replica in a different zone. When the primary fails, it promotes the least-lagged replica and propagates that node’s DNS name to the primary endpoint. Writes resume within seconds. Replication is asynchronous, so a promotion loses whatever had not yet reached the replica. For a session store that means a slice of customers is logged out even on a clean failover, which is smaller than losing all of them but not nothing. Valkey 9.0 and later can run with durability enabled, persisting committed writes to a Multi-AZ transactional log so that synchronous writes survive a failover intact. Durability is chosen when the group is created and cannot be added to an existing group afterwards. The TestFailover API promotes a replica on demand, for up to fifteen shards in a rolling day, which is how the configuration gets checked without waiting for a real zone failure.

EFS and S3 in place of EBS. An EBS volume is zonal and cannot be attached to an instance in another zone. Where the data is file-shaped and shared, a Regional EFS file system has one mount target per zone and removes the anchor. A One Zone file system has exactly one mount target and is a zonal anchor by design, which makes the storage class the thing to read rather than the service name. S3 carries the same split: Standard and the Glacier classes store redundantly across at least three zones and are designed to survive losing one, while One Zone-IA stores within a single zone. Choosing the cheaper class here re-creates the problem being fixed. Where the volume genuinely has to be a block device attached to one instance, the story is a snapshot and a restore into another zone, which is recovery rather than availability.

Elastic IP addresses and fixed endpoints. An Elastic IP address is scoped to a Region and remaps between network interfaces, so the address outlives the instance behind it. It does not outlive a partner’s change-control process. A Network Load Balancer takes one Elastic IP address per enabled zone, which puts stable addresses in front of targets that are free to move. AWS charges hourly for every public IPv4 address, attached or idle. The default quota is five Elastic IP addresses per Region, adjustable on request. Three for the load balancer plus the two the batch workers already hold reaches that default exactly. The NLB attribute dns_record.client_routing_policy sets how much zonal affinity clients get, which decides whether a client that resolved one zone’s address keeps using it.

Zonal shift and zonal autoshift. Amazon Application Recovery Controller shifts traffic away from a single zone in one action. It supports Auto Scaling groups, EKS clusters, ALBs and NLBs, and a shift expires between one minute and 72 hours after it starts. It is a mitigation rather than a design, and it only works if the remaining zones already hold enough capacity to take the load. Zonal autoshift hands the trigger to AWS, so a shift starts on AWS telemetry instead of on somebody noticing. It requires weekly practice runs, each moving one resource’s traffic away for about half an hour. Those practice runs are the standing proof, and they report a failure when the capacity was never pre-scaled.

Fault Injection Service. The AZ Availability: Power Interruption scenario assembles the symptoms of a zone losing power rather than one fault at a time. Instances are stopped. Launches in that zone return InsufficientInstanceCapacity. Subnet connectivity is cut for two minutes, an RDS cluster is failed over, ElastiCache nodes are interrupted, and one EBS volume is left unresponsive. It runs 30 minutes of interruption followed by 30 minutes of recovery symptoms, finds most of its targets by tag, and can start zonal autoshift five minutes in. Two details decide whether a run is worth anything. The scenario ships with no stop conditions, so the CloudWatch alarms that would page anyway have to be added to the template. And an action whose targets do not resolve is skipped rather than failed, so one resource missing its tag is never touched while the experiment still finishes and reports success.

Evaluation

Side by side

Component	Single-zone today	Behaviour on zone loss	Fix	Needs the window
ASG subnets	✓ all three in 2a	All compute gone	Add subnets, let AZ rebalance run	✗
ALB zones	✗ two enabled, one empty	Empty node has nowhere left to forward	Enable the third zone, register targets	✗
NAT gateway	✓ single, in 2a	Every private subnet loses egress	Regional gateway, or one per zone	✗
RDS Multi-AZ	✗ standby in a second zone	Fails over in 60 to 120 seconds	None	✗
Self-managed Redis	✓ instance plus EBS volume	Sessions gone, checkout errors	Valkey replication group, Multi-AZ	✓
Batch workers on EIPs	✓ named instances in 2a	Batches stop	NLB with one EIP per zone	✓ partner-dependent

One row needs no work, three are online changes, and two need the window. That split decides the plan: the two hours on Sunday go on Redis and the batch workers, and everything else lands before it. The NAT gateway row is the one worth staring at. It converts a single zone’s failure into a total one, and a diagram review would never have flagged it.

The solution

Make every online change first, spend the window on state, then prove it with an experiment and keep proving it.

Start with subnets, because everything else needs somewhere to go. Create subnets in the two unused zones with a CIDR plan that leaves room to grow, and write down why each range is what it is. The template with the hardcoded zone is the reason this happened, so the replacement has to be readable by whoever inherits it. AWS maps physical zones to zone names randomly per account, so ap-southeast-2a here is not the same building as ap-southeast-2a in the account next door. Pin the plan to AZ IDs such as apse2-az1, which are the same everywhere, because this template is reused across the platform’s accounts.

Enable the third zone on the load balancer, then add all three subnets to the Auto Scaling group. Changing the group’s zones starts an Availability Zone rebalance on its own, launching before terminating, so the site stays up and no instance refresh is needed. Raise the group’s minimum so that two zones can carry the traffic, because a group sized for three zones and running at capacity has nothing spare when one goes.

Then the NAT gateway, the highest-value change here and the one nobody would have ranked first from the diagram. A regional gateway removes the per-zone route table work and the need to expand it again the next time a zone is added, which is the failure mode that produced this state. A gateway per zone stays the right answer where private NAT is in play. Either way, existing connections through the old gateway break when the route changes. Move one subnet at a time, outside business hours, even though this is not the formal window. Cross-zone transfer charges fall as a side effect, because traffic stops hopping zones to reach a shared gateway.

The window goes on Redis. Stand up a Valkey replication group with Multi-AZ and automatic failover, with a replica in each of the other two zones. Point the application at the primary endpoint and accept that sessions do not migrate. Two hours on a Sunday morning is chosen because logging everyone out then is tolerable. Where it would not be, the alternative is dual-writing sessions to both stores for a week and moving reads across, which is more code and no window. Either way, size the replication group for two zones rather than three, since a promotion leaves the group short until the failed zone returns.

The batch workers are the slowest thread and start first even though they finish last, because they wait on a partner editing firewall rules. Put a Network Load Balancer with one Elastic IP address per zone in front of them, hand the partner the new addresses, and run both paths until the partner confirms. Count the Elastic IP addresses before any of this. Three for the load balancer plus the two the workers hold is exactly the default five per Region, so the parallel run has no spare address, and anything else in the account holding one means asking for an increase first. The workers then become ordinary Auto Scaling group members, since nothing outside references them individually.

Finally, prove it, then arrange to keep proving it. Build the AZ Availability: Power Interruption scenario as a template. Tag the workload’s instances, Auto Scaling group, subnets, volumes and replication group so nothing gets skipped, and give the launch-pause action the ARN of the provisioning role, which is the one target the scenario does not find by tag. Attach the alarms that would page anyway as stop conditions, because the scenario arrives without any. The scenario’s RDS action targets a DB cluster, and this database is a Multi-AZ DB instance, so add aws:rds:reboot-db-instances with forceFailover set rather than assuming the scenario covered it. Run it in staging, then in production during a quiet hour. Once the run comes back clean, enable zonal autoshift on the load balancer and the Auto Scaling group. The weekly practice runs then keep testing the same claim after everyone has moved on, and a real impairment starts shifting traffic on AWS telemetry rather than on a pager.

Why not add zones to the Auto Scaling group and stop. That change makes the diagram true and leaves the NAT gateway converting any zone failure into a full outage. Compute spread across three zones that all reach the internet through one gateway is not multi-AZ.

Why not move sessions into RDS or Aurora to avoid a second technology. Sessions in a relational database work and add latency to the hottest path in the application. A managed cache is the same shape as what already exists, minus the zonal anchor, and it fails over in seconds rather than a minute. The wider decision about what belongs in a cache at all does not change here.

Worked example

The inventory takes a day and turns up one thing nobody expected. A Lambda function that reconciles payments is attached to a single subnet, put there years ago so it could reach a private endpoint. It appears on no diagram and would have stopped with the zone. Adding the two new subnets to its VPC configuration takes a minute and is the cheapest fix on the list.

Subnets, the third load balancer zone and the group’s new zones land in one afternoon with no user impact. Auto Scaling rebalances over about 25 minutes, briefly running one instance above the group maximum while it launches before terminating, which nobody notices except on the graph.

The NAT gateway change is done one subnet at a time over an evening. A long-running outbound connection from a batch process breaks and retries cleanly, which is spotted only because somebody was watching the logs.

The Redis cutover takes 40 minutes of the two-hour window. Everyone is logged out, which was announced, and the remaining 80 minutes go on watching error rates rather than fixing anything.

The partner takes three weeks to update its firewall rules, which is why starting that thread first mattered. The NLB runs alongside the old Elastic IP addresses throughout, and the old addresses are released after a fortnight of zero traffic, landing at week five with the peak.

The production run impairs ap-southeast-2a. Compute sheds and recovers, the load balancer stops sending traffic to the affected zone, RDS fails over in 74 seconds, and the replication group promotes a replica in about four seconds. One component misbehaves. An internal service opens its database connections with a 30-second timeout and no retry, so every request during the 74-second failover returns an error. A retry would have turned that into one slow response. That finding matters more than the rest of the exercise, and it surfaced only because somebody caused the failure on purpose.

What’s worth remembering

1. Look for the empty ALB zone. An ALB needs two zones, so the single-zone defect is an enabled zone with no registered target.
2. One single-zone dependency sinks everything. A single NAT gateway is the usual one; private subnets in healthy zones lose egress with it.
3. Prefer the regional NAT gateway. One gateway ID serves every zone and expands as workloads move; keep zonal gateways for private NAT.
4. Changing zones triggers a rebalance. Auto Scaling launches before terminating, so an instance refresh is the wrong tool.
5. Read the storage class. Regional EFS and S3 Standard span multiple zones; One Zone EFS and One Zone-IA are zonal anchors.
6. Designed to survive is not proven. The FIS power-interruption scenario ships with no stop conditions and skips untagged targets, so add alarms and tags.