How to Design Multi-Region Failover Under a Minute

The situation

We run a payments API in eu-west-1. A handful of Fargate services sit behind an ALB, an Aurora PostgreSQL cluster holds the ledger, a DynamoDB table holds idempotency keys, and an S3 bucket holds receipts. The stack lives in a payments account inside an organisation that also runs a shared-services account for DNS and a security account for the audit trail, and the deployment pipeline reaches both from a tooling account.

Revenue per minute of downtime is high enough that the business has set a recovery-time objective of 60 seconds and a recovery-point objective of zero. No acknowledged transaction is lost, even if Dublin goes dark.

The plan is to run the same stack in eu-west-2 and move traffic across. We can do that. What needs deciding is which replication shape each store gets, because each one sets its own ceiling on how fast writes come back and how much unreplicated work goes over the side. The request path and the data path get designed separately, then reconciled against the same 60 seconds.

What actually matters

A write that commits in eu-west-1 and replicates afterwards returns at local latency and leaves a window in which London does not yet know about it. A write that waits for a second Region to acknowledge closes that window and adds a cross-Region round trip to every request. AWS publishes no latency figures between one Region and another, so the size of that penalty is something to measure for the pair we pick. The choice does not have to be the same for all three stores.

Payments makes eventual consistency awkward in one specific place. If an idempotency key written in Dublin has not reached London when Dublin fails, the retry that lands in London finds nothing and the customer is charged twice. The ledger is different. The card network holds its own record of every authorisation, so a ledger a second behind is reconcilable against a source we do not own. Receipts are different again: one that arrives late is an inconvenience, not an incident. Three stores, three RPO requirements, and only the first of them genuinely needs zero.

Route 53 moves requests by changing what resolvers return, which puts two delays in the failover path. The first is declaring the Region unhealthy, and that is tunable: health checks run at 10 or 30 seconds, the failure threshold runs from one to ten consecutive checks, and the defaults are 30 seconds and three. The second is waiting for every cached answer to expire, and that one has no lever. A client that ignores the record’s TTL keeps calling the dead Region for as long as it likes. On a 60-second budget, an uncontrollable delay is the part that decides whether the budget holds.

The cutover finishes when the slowest dependency finishes. A stateless service behind a load balancer is ready as soon as requests arrive. A key-value store that accepts writes in every Region needs no failover step at all. A single-writer relational engine needs a promotion, and AWS puts that at a few minutes rather than seconds. Whatever the request path manages, the relational writer sets the floor on write-RTO, which is also an argument for keeping both Regions genuinely hot: a Region that only takes traffic during an outage is a Region whose configuration drift surfaces during the outage.

What we’ll filter on

1. RPO at the data layer, how much acknowledged work disappears when a Region does?
2. Write availability during the outage, do writes succeed in the surviving Region or wait for a promotion?
3. Whether a client’s DNS cache sits in the failover path.
4. Conflict model, what happens when both Regions write the same key?
5. Constraints the shape imposes: Regions pulled into scope, features given up, account boundaries.
6. Steady-state cost shape, what gets billed twice and what does not?

The landscape

Moving the requests

Route 53 latency-based routing with health checks. Two records for the same name, one per Region, each associated with a health check against that Region’s ALB. Resolvers get the nearer Region; a failing health check drops a Region out of the answer set. The detection half is tunable down to a 10-second interval with a threshold of three, so about 30 seconds. The propagation half depends on resolver and client behaviour that we do not control. Health checks are billed monthly per health check, with surcharges for the fast interval, for HTTPS and for string matching.

AWS Global Accelerator. Two static anycast IPv4 addresses (plus IPv6 for dual-stack) advertised from AWS edge locations, with an endpoint group per Region pointing at the ALBs. Clients hold addresses that never change, so failover becomes an internal routing decision instead of a DNS propagation. AWS states that Global Accelerator detects an unhealthy endpoint and takes it out of service in less than a minute, and that change propagation takes a matter of seconds because it does not depend on client IP address caching. One detail to know: the health-check settings on an endpoint group (10 or 30 seconds, threshold defaulting to three, TCP by default) apply to EC2 and Elastic IP endpoints. For ALB and NLB endpoints, Global Accelerator reads the load balancer’s own health, so the target group’s settings are the ones that matter. TrafficDialPercentage per endpoint group, 0 to 100, is the manual dial.

Amazon Application Recovery Controller (ARC) routing controls. The service formerly branded Route 53 Application Recovery Controller. A routing control is an on/off switch with its own regional endpoints, surfaced to Route 53 as a RECOVERY_CONTROL health check, so flipping it moves traffic without waiting for an endpoint to look sick. That health check sits on a DNS record, so a routing control shifts traffic the way any Route 53 failover does, and inherits the same resolver caching. Safety rules constrain what the switches can be set to together, which stops an operator turning both Regions off during an incident. ARC readiness checks, which audit quotas and capacity in the standby Region, closed to new customers on 30 April 2026; AWS points at Region switch instead, which expresses a recovery as a plan of execution blocks, one for Aurora Global Database, and evaluates whether the plan would still run.

Moving the state

DynamoDB global tables, eventual consistency (MREC). The default mode. Any replica takes reads and writes, changes replicate asynchronously, and AWS puts the lag at typically a second or less. Conflicts resolve last-writer-wins on an internal per-item timestamp. RPO is the replication delay, which AWS describes as usually a few seconds depending on the Regions. Writes never stop in the surviving Region, so write-RTO is zero. Replicas can sit in any Region where DynamoDB runs, and in other accounts: multi-account global tables give each account its own replica, its own KMS boundary and its own CloudTrail trail, on version 2019.11.21 only.

DynamoDB global tables, strong consistency (MRSC). A write is replicated synchronously to at least one other Region before it returns, and AWS states an RPO of zero. The constraints are real. An MRSC table is deployed in exactly three Regions, either as three replicas or as two replicas plus a witness that holds the data but serves no reads or writes. The three come from a published list of fifteen Regions, six of them in Europe, in any combination, including Regions on different continents. It must be created from an empty table, replicas cannot be added later, and it is same-account only. TTL, transactions and local secondary indexes are all unsupported, and a concurrent write to the same item in another Region returns ReplicatedWriteConflictException for the caller to retry.

Aurora Global Database. One primary Region taking writes, up to ten read-only secondaries, replication through the storage layer rather than the engine, with latency AWS describes as typically under a second. A secondary cluster carries up to 16 readers against the usual 15. Two distinct operations move the primary. A switchover, previously called managed planned failover, synchronises first and has an RPO of zero, and it requires every cluster healthy. A failover with --allow-data-loss is the unplanned path: it does not wait for synchronisation, so the loss equals the replication lag at the moment of failure, and AWS says the chosen secondary typically assumes the primary role within a few minutes. The global writer endpoint follows the primary automatically across both operations. On Aurora PostgreSQL, rds.global_db_rpo puts a hard bound on that loss by blocking commits on the primary when every secondary is further behind than the value set, from 20 seconds upward.

Aurora cross-Region read replicas. The pre-Global-Database shape: a separate cluster fed by binlog replication, up to five per source, promoted by hand, with no lag figure AWS publishes. It is an Aurora MySQL feature: for Aurora PostgreSQL, AWS’s answer to a reader in another Region is Global Database, so on this ledger it is not an option at all.

S3 replication. Asynchronous object replication between versioned buckets, configured per rule and per direction. Writes succeed in either bucket at any time, and there is no shared key space to conflict over. Plain replication gives no timing commitment. S3 Replication Time Control adds a service level agreement of 99.9% of objects inside 15 minutes, replication metrics for pending operations and bytes, and OperationMissedThreshold events when an object misses the window, so the receipts gap becomes measurable rather than assumed. RTC’s SLA lapses above a default 1 Gbps replication transfer quota, which is adjustable through Service Quotas.

Application-layer dual write. The service writes to both Regions and returns only when both acknowledge. Strongly consistent, and every write carries the cross-Region round trip. The failure modes land on us: partial writes when one Region is slow, retry storms when one is dead, and conflict resolution we now have to write. Reach for it only when nothing native fits.

Single-Region primary with cross-Region backups. One Region serves, the other holds AWS Backup copies, cross-Region snapshots and replicated objects. RPO in minutes to hours, RTO in tens of minutes once somebody works the playbook. Cheapest by a wide margin, and roughly two orders of magnitude outside a 60-second target. Included to anchor the scale.

Evaluation

Side by side

Option	RPO	Write availability during outage	Conflict model	Constraints it imposes	Cost shape
Global tables, MREC	Seconds or less	Writes continue	Last-writer-wins	Any Regions, cross-account allowed	2× replicated write units, no transfer charge
Global tables, MRSC	Zero	Writes continue	Conflict exception, caller retries	Exactly 3 supported Regions, same account, no TTL or transactions	2× replicated write units, witness free
Aurora Global Database	Zero on switchover, lag on failover	Blocked until promotion	Single writer	Matched engine versions both ends	Second cluster, per-Region storage, replicated write I/O
Aurora cross-Region RR	Replication lag, unpublished	Blocked until manual promotion	Single writer	Aurora MySQL only, manual promotion	Second cluster
S3 replication with RTC	99.9% within 15 min	Writes continue	None, separate key spaces	Versioning both ends, rule per direction	2× storage in GiB, transfer out, replication PUTs, RTC fee
Single-Region + backups	Minutes to hours	Blocked	Not applicable	Restore procedure	Storage only

The request path, compared

Option	Detection	Is client DNS caching in the path?	Manual control	Fails safe?
Route 53 health checks	~30s at 10s interval, threshold 3	Yes, and unbounded	Record edits	No guardrails
Global Accelerator	Under a minute, from the ALB’s own health	No, addresses are static	Traffic dial per Region	Endpoint weights
ARC routing control	Operator decides	Yes, it flips a DNS health check	Routing control state	Safety rules

Read by data layer rather than by option, the assignment falls out:

• Idempotency keys. Zero RPO is the requirement, the item count is small, and the writes are short. MRSC.
• The ledger. Single-writer relational, reconcilable against the card network, so a bounded non-zero RPO is survivable. Aurora Global Database.
• Receipts. Already objects, already asynchronous, wanted mainly for audit. S3 replication with RTC, a rule in each direction.
• The services themselves. Stateless behind an ALB, deployed to both Regions by the same pipeline. Global Accelerator in front, its traffic dial as the manual switch.

The topology

The solution

Global Accelerator in front of both ALBs, with Route 53 holding an alias record for api.payments.example.com that points at the accelerator. Clients resolve the name once and keep two addresses that never change, so a Region going away is a routing decision inside AWS rather than a wait for caches to expire. Each endpoint group carries one ALB, and Global Accelerator takes the load balancer’s own health as the signal, which means the ALB target group’s interval and unhealthy threshold set the detection time rather than anything configured on the accelerator. AWS puts removal of an unhealthy endpoint at under a minute and propagation of the change at seconds.

The deliberate switch is the accelerator’s own traffic dial, 0 to 100 per endpoint group. Health-based failover handles the Region that stops answering; the dial handles the Region that answers and is wrong, which is the harder incident, and it needs no DNS change. A routing control would bring back the resolver caching we chose Global Accelerator to avoid. The orchestration goes into an ARC Region switch plan, which spans both accounts, keeps a data plane in every Region, and carries an Aurora Global Database block for the step below.

DynamoDB in MRSC mode for idempotency keys. Replicas in eu-west-1 and eu-west-2 with a witness in eu-west-3, all three on AWS’s list for the mode. A CreatePaymentAttempt(key) write returns only once it has been replicated synchronously beyond the Region that took it, so a retry landing anywhere finds the record and short-circuits with the prior result. That meets the zero-RPO requirement in the one store where it is non-negotiable, and it gives up three things. Paris enters the compliance footprint even though it serves no traffic. TTL is unavailable, so a scheduled sweeper expires keys instead, which is a job to run and monitor. Concurrent writes to one item across Regions return ReplicatedWriteConflictException, so the client library retries on it.

MRSC also has to be same-account, which cuts across the usual instinct to let each Region’s data sit in its own account. Multi-account global tables are MREC only. Here that trade goes the way of consistency: the idempotency table stays in the payments account with replicas the payments account owns, and the security account reads it through a cross-account role rather than through a replica of its own.

Aurora Global Database for the ledger. Writer in eu-west-1, secondary cluster in eu-west-2 sized to take the full write load rather than a token instance. The application connects through the global writer endpoint, which follows the primary automatically through both switchover and failover, so there is no application-side endpoint change and nothing for us to script. DNS cache TTL on that endpoint goes down to about five seconds, per AWS’s guidance, and the RDS event that fires when Aurora observes the DNS change becomes the signal that it is safe to resume writes. rds.global_db_rpo is set to its minimum of 20 seconds, which blocks commits on the primary whenever every secondary is further behind than that, turning an unbounded data-loss risk into a bounded one and stalling writes whenever replication falls behind. It goes on the primary’s parameter group only: AWS recommends leaving it at the default in the secondary’s, because with just two Regions an RPO enforced there would pause transactions after a failover until the old primary is rebuilt.

This is where the strict reading of zero RPO gives way. A switchover is lossless but needs both clusters healthy, so it is the planned-maintenance path and the fail-back path, not the outage path. An unplanned failover is failover-global-cluster with --allow-data-loss, run against the secondary’s Region with the target cluster’s ARN, and it takes whatever the replication lag was. The ledger can absorb that because the card network holds the authoritative record of each authorisation and reconciliation closes the gap. Nothing else in the design can absorb it, which is why nothing else is on Aurora.

S3 replication for receipts, with a rule in each direction between versioned buckets and RTC turned on for both. During an outage the surviving Region takes the writes, and when the other comes back each bucket converges toward the union. Two rules, not one: a single Dublin-to-London rule leaves anything written in London during the incident stranded there. Delete-marker replication is switched on explicitly, since by default S3 adds the marker in the source bucket alone, and those markers fall outside the SLA. The replication metrics plus OperationMissedThreshold events make the receipts gap a number on a dashboard rather than an assumption. Lifecycle rules transition to Glacier Instant Retrieval at 30 days in both buckets, where the 90-day minimum storage duration then applies.

The whole thing deploys from one pipeline in the tooling account, assuming a role in the payments account in each Region, with the same Terraform modules and per-Region inputs. Both Regions get every merge. Something has to watch the standby’s quotas and capacity, because the Fargate task limit and the Aurora instance quota are per-Region, per-account values, and a Region that cannot scale to 100% of production is not a failover target however healthy it looks. ARC readiness checks are closed to new customers, so that falls to Region switch plan evaluation and alarms on Service Quotas usage metrics.

Worked example

eu-west-1 starts failing at 14:07:00. The Dublin ALB’s target group marks its targets unhealthy at a 10-second interval with a threshold of two, so the load balancer stops reporting healthy by about 14:07:20. Global Accelerator reads that health and removes the Dublin endpoint group; AWS puts that inside a minute, with propagation in seconds. Clients are holding the same two anycast addresses throughout, so nothing waits for a TTL. In-flight connections reset, the client SDK retries, and the retry lands in London.

London goes from roughly half the traffic to all of it, so about 2× baseline, and Fargate autoscaling adds tasks against a service quota the standby’s usage alarms have been watching all quarter.

Idempotency writes never stop. The MRSC table has lost the Dublin replica, and the London replica plus the Paris witness are the other two of the three, which is what the witness is deployed for. Every key acknowledged before 14:07:00 is already in London, so a retry of a payment started in Dublin finds its record and returns the original result instead of charging again. No promotion runs and no operator touches it.

Ledger writes fail. An operator dials the Dublin endpoint group to zero to make the intent explicit, then runs aws rds failover-global-cluster --region eu-west-2 --global-cluster-identifier payments-global --target-db-cluster-identifier arn:aws:rds:eu-west-2:123456789012:cluster:payments-eu-west-2 --allow-data-loss. AWS puts promotion at typically a few minutes. Call it 14:10. The global writer endpoint re-points on its own, the RDS event confirms Aurora has seen the DNS change, and writes resume without a deploy. Anything committed in Dublin inside the final replication lag is gone, bounded by rds.global_db_rpo at 20 seconds, and reconciliation against the card network recovers it.

So reads and the payment-attempt records that decide whether a customer is charged twice are back inside the 60 seconds. The request path is close rather than certain: 20 seconds of target-group detection plus a removal AWS bounds only at under a minute can land past 60. It does not hold for the ledger writer, which comes back in minutes. Closing that last gap means giving up a single-writer relational ledger, and the business has looked at both and chosen the reconcilable few minutes.

What’s worth remembering

1. The slowest layer sets failover speed. With a single-writer relational store that layer is the promotion, which AWS puts at a few minutes, not seconds.
2. Static anycast addresses bypass DNS caching. A DNS-based cutover leaves a delay you cannot control inside a 60-second budget.
3. Only MRSC gives DynamoDB RPO zero. Exactly three of fifteen supported Regions, same account; TTL, transactions and LSIs unsupported.
4. Switchover is lossless; failover loses data. Unplanned Aurora failover needs --allow-data-loss, and rds.global_db_rpo bounds that loss.
5. Manual switches catch wrong answers. Dial an accelerator endpoint group to zero when a Region answers incorrectly, which health checks alone miss.
6. Replicate S3 in both directions. Replication is per rule and per direction, so one Dublin-to-London rule strands London writes made during the incident.