The situation
An order management platform has been in production for three years and its diagram has said “Multi-AZ” the whole time. A routine review before a peak trading period found otherwise.
The Auto Scaling group spans three subnets, which is correct, but all three are in ap-southeast-2a because the subnets were created from a template where the AZ was hardcoded. The Application Load Balancer has one subnet mapping. There is a single NAT gateway, in that same zone, and every private subnet routes to it. RDS is Multi-AZ, which is the one thing that was right. A self-managed Redis instance holding session state runs on a single EC2 instance with an attached EBS volume. Two batch workers run on named instances with Elastic IPs baked into a partner’s firewall rules.
The business will accept a change window of two hours on a Sunday, once. Anything else has to happen without an outage. Peak trading starts in five weeks.
What actually matters
The first thing that matters is that “multi-AZ” is a property of every component in the request path, not of the account or the diagram. A load balancer in two zones in front of instances in one zone is a single-zone system with extra steps. The work starts as an inventory rather than a design.
The second is the difference between a component that fails over and one that simply stops. RDS Multi-AZ promotes a standby automatically. A single NAT gateway does not: when its zone goes, every private subnet routing to it loses outbound connectivity, including the ones in healthy zones. Failure of a shared single-zone dependency converts a zonal outage into a regional one for that workload, which is the worst shape available.
The third is that state decides difficulty. Stateless instances behind a load balancer become multi-AZ by adding subnets and letting the group replace members, which needs no outage at all. Anything holding state that other things depend on, a Redis instance with sessions, an EBS volume, a fixed IP address a partner has allow-listed, needs its state moved or replicated, and that is where the change window gets spent.
The fourth is that data has a zonal dimension people forget. An EBS volume exists in one Availability Zone and cannot be attached to an instance in another. An instance store disappears with the instance. EFS and S3 are regional and stop being part of this problem, which is often the cheapest fix available for a workload whose only zonal anchor is a volume.
Underneath it, a design that survives a zone failure is not the same as a design proven to. Once the changes are in, the remaining question is whether the failover behaves as expected, and that is answered by causing a failure deliberately rather than by waiting.
What we’ll filter on
- Is this component single-zone today, and does the request path depend on it?
- Does it fail over automatically, degrade, or stop?
- Does it hold state, and is that state zonal?
- Can the change be made without an outage, or does it need the window?
- Does anything outside our control depend on a specific address?
- After the change, how do we know it works?
The landscape
Subnets and the Auto Scaling group. Subnets are zonal by definition. A group spans the zones of the subnets it is given, so adding subnets in two more zones and updating the group is the entire fix for stateless compute. New instances launch across zones as the group scales or replaces members, and an instance refresh forces the redistribution immediately rather than waiting. No outage, provided the group’s minimum leaves capacity while members cycle.
Load balancer subnet mappings. An ALB must be given a subnet in each zone it serves, and it only routes to targets in zones it has been enabled for. Adding a subnet mapping is an online operation. Cross-zone load balancing determines whether a node distributes to targets in other zones; it is on by default for ALB and off by default for NLB, and that default is a common trap when the zones hold uneven target counts.
NAT gateways. A NAT gateway is zonal. The resilient pattern is one per zone, with each private subnet’s route table pointing at the gateway in its own zone. This costs more (an hourly charge per gateway) and removes both the single point of failure and the cross-zone data transfer charge that a shared gateway quietly generates. Creating gateways and repointing route tables happens online, with existing connections through the old gateway breaking at the moment the route changes.
Aurora and RDS. RDS Multi-AZ maintains a standby in another zone and fails over by moving the endpoint, taking a minute or so. Multi-AZ DB cluster deployments add two readable standbys and cut failover further. Aurora replicates storage across three zones by design and promotes a reader on writer failure. This is the component most likely to already be right, because it is a checkbox rather than a design.
ElastiCache replication groups. A managed Redis or Valkey replication group with Multi-AZ enabled keeps replicas in other zones and promotes automatically. Replacing a self-managed single instance with a managed replication group removes the zonal anchor and the operational burden together. Migration means either accepting the loss of session state at cutover or running both and draining, which is the decision that determines whether this needs the window.
EFS and S3 in place of EBS. An EBS volume is zonal and cannot be attached across zones. Where the data is file-shaped and shared, EFS is regional with mount targets per zone and removes the anchor entirely. Where it is object-shaped, S3 does the same. Where the volume genuinely must be block storage attached to one instance, the failure story is a snapshot and a restore into another zone, which is recovery rather than availability.
Elastic IPs and fixed addresses. An Elastic IP is regional and can move between instances, but a partner allow-listing a specific address makes any change a negotiation with a third party. A Network Load Balancer with an Elastic IP per zone gives stable addresses in front of instances that can move, which converts a pinned instance into a pinned load balancer with healthy targets behind it.
Fault Injection Service. Once the design is multi-AZ, an experiment that impairs a zone tests whether it behaves that way. Templates carry stop conditions wired to CloudWatch alarms, and scoping is by tag, so the blast radius is bounded to the workload under test.
Evaluation
Side by side
| Component | Single-zone today | Behaviour on zone loss | Fix | Needs the window |
|---|---|---|---|---|
| ASG subnets | ✓ all three in one AZ | Total loss of compute | Add subnets, instance refresh | ✗ |
| ALB subnet mapping | ✓ one mapping | Cannot route anywhere else | Add mappings | ✗ |
| NAT gateway | ✓ single | All private egress fails, every zone | One per zone, per-zone routes | ✗ |
| RDS | ✗ already Multi-AZ | Automatic failover | None | ✗ |
| Self-managed Redis | ✓ instance plus EBS | Sessions lost, app errors | ElastiCache replication group | ✓ cutover |
| Batch workers on EIPs | ✓ named instances | Batches stop | NLB with zonal EIPs, or partner update | ✓ partner-dependent |
The table sorts itself: four rows need no outage and two do, which means the two-hour window is spent on Redis and the batch workers, and everything else can land before it. The NAT gateway is the row worth noticing, because it is the one where a single-zone component turns a zonal failure into a total one.
The solution
Do everything that needs no outage first, spend the window on state, and prove it with an experiment afterwards.
Start with subnets, because everything else depends on having somewhere to put things. Create private and public subnets in the two unused zones with a CIDR plan that leaves room, and take the opportunity to write down why each range is what it is, since the hardcoded-AZ template is the reason this happened at all.
Add the ALB subnet mappings and add the new subnets to the Auto Scaling group. Run an instance refresh so the group redistributes immediately rather than waiting for organic replacement; with a minimum healthy percentage set the site stays up throughout. At this point the request path is genuinely multi-AZ for anything stateless.
Then fix the NAT gateways, which is the highest-value change on the page and one nobody would have prioritised from the diagram. Create a gateway in each new zone, then repoint each private subnet’s route table at the gateway in its own zone. Existing connections through the old gateway break as the route changes, so do it per subnet rather than all at once, and do it outside business hours even though it is not the formal window. The side effect is a reduction in cross-zone data transfer charges, because traffic stops hopping zones to reach a shared gateway.
The window goes on Redis. Stand up an ElastiCache replication group with Multi-AZ enabled and automatic failover, point the application at the new endpoint, and accept that sessions do not migrate. Two hours on a Sunday morning is chosen precisely because logging everyone out then is tolerable. If it were not, the alternative is dual-writing sessions to both stores for a week and cutting over reads, which is more code and no window.
The batch workers are the slowest thread and should start first even though they finish last, because they depend on a partner changing firewall rules. Put a Network Load Balancer with an Elastic IP in each zone in front of them, give the partner the new addresses, and run both paths until the partner confirms the cutover. The workers themselves then become replaceable instances in an Auto Scaling group, since nothing external references them individually any more.
Finally, prove it. Build a Fault Injection Service experiment scoped by tag to this workload that impairs one zone, with stop conditions on the alarms that would page anyway. Run it in staging, then in production during a quiet window, and watch what actually happens rather than what the diagram says will. The output is evidence for the peak trading review, and the run itself frequently finds the thing the inventory missed.
Why not add zones to the Auto Scaling group and stop. It is the change that makes the diagram true and leaves the NAT gateway converting any zone failure into a full outage. Compute spread across three zones that all route egress through one is not multi-AZ.
Why not move Redis to RDS or Aurora to avoid a second technology. Sessions in a relational database work and cost latency on the hottest path in the application. The managed cache is the same shape as what exists, minus the zonal anchor.
Worked example
The inventory takes a day and finds one thing nobody expected. Beyond the known items, a Lambda function that reconciles payments is attached to a single subnet, put there years ago so it could reach a private endpoint. It is invisible in the diagram and would have stopped with the zone.
Subnets, ALB mappings and the instance refresh land in one afternoon with no user impact. The refresh takes 25 minutes at 50% minimum healthy.
The NAT gateway change is done one subnet at a time over an evening. One long-running outbound connection from a batch process breaks and retries cleanly, which is noticed only because somebody was watching the logs. The following month’s bill shows a fall in cross-AZ data transfer that roughly offsets the two additional gateways.
The Redis cutover takes 40 minutes of the two-hour window. Everyone is logged out, which was communicated, and the remaining 80 minutes are spent watching error rates rather than fixing anything.
The partner takes three weeks to update firewall rules, which is why starting it first mattered. The NLB runs alongside the old Elastic IPs for the whole period, and the old addresses are released only after a fortnight of zero traffic.
The Fault Injection run in production impairs ap-southeast-2a. Compute sheds and recovers, the ALB stops routing to the affected targets, RDS fails over in 51 seconds, and ElastiCache promotes a replica. One thing does not behave: an internal service has a hardcoded 30-second connection timeout with no retry, so it returns errors for the whole RDS failover instead of the two seconds it should have. That finding is worth more than the rest of the exercise, and it is only visible because somebody caused the failure on purpose.
What’s worth remembering
- Multi-AZ is a property of every component in the request path, not of the account; a load balancer across three zones in front of instances in one is a single-zone system.
- A shared single-zone dependency, and a single NAT gateway is the usual one, converts a zonal failure into a total outage, because private subnets in healthy zones lose egress too.
- Sort the work by state: stateless compute becomes multi-AZ by adding subnets and refreshing, with no outage, while anything holding zonal state is what the change window is for.
- EBS volumes are zonal and cannot cross Availability Zones; where the data is file- or object-shaped, moving to EFS or S3 removes the anchor rather than working around it.
- An address a third party has allow-listed is a schedule dependency, not a technical one, so start it first even though it finishes last.
- A design that should survive a zone failure and one proven to are different claims; a Fault Injection experiment with stop conditions turns the first into the second, and usually finds a timeout nobody knew about.