Proving a DR Plan Works Before You Need It

The situation

A payments processor in Melbourne has a documented DR plan with a four-hour recovery time objective and a fifteen-minute recovery point objective. The plan describes a warm standby in ap-southeast-4: an RDS cross-Region read replica, AMIs copied nightly, an Auto Scaling group at minimum capacity, and Route 53 failover records behind health checks. Payment state lives in a Multi-AZ RDS instance in ap-southeast-2. Uploaded settlement documents live in S3.

Production runs in its own account. Security tooling, the log archive and shared networking each sit in accounts of their own, all under one organisation, and the standby is a second Region inside the same production account.

The plan has never been executed. It was written two years ago, reviewed annually as a document, and signed off each time. Nobody has promoted the replica, nobody has scaled the standby group, and nobody has confirmed that the application in the standby Region can reach the things it needs.

An auditor has asked for evidence that the four hours is achievable. The business will not accept an unplanned production outage to find out, and the platform team suspects, without being able to say so in the review, that the figure is optimistic.

What actually matters

An untested plan has an unknown recovery time rather than the one written down. The four hours came from adding up estimates, and estimates omit the steps nobody thought of: the manual approval, the DNS TTL, the certificate that only exists in the primary Region, the secret that was never replicated. Every one of those is invisible until somebody tries.

Testing has levels, and the argument about not being able to test DR in production usually collapses once they are separated. Reading the runbook aloud in a room finds missing steps. Restoring the backups in a spare account finds unrecoverable data. Failing over one component in production finds integration problems. Failing over the whole workload finds the real number. Each level takes more effort and finds different things, and jumping to the last is what makes the argument feel binary.

The mechanism the plan turns on decides how often it can be rehearsed at all. Promoting an RDS cross-Region read replica is a one-way operation: RDS stops replication and reboots the instance, and afterwards it is a standalone database that cannot serve as a replication target again. Getting back to the original shape means building a fresh replica in the other direction and waiting for it to catch up. An Aurora global database behaves differently. A switchover waits for the secondary to synchronise, promotes it with no data loss, and keeps the same clusters in the same Regions, and AWS names regional rotation for regulated systems as one of its uses. That design can be exercised on a schedule. A plan built on replica promotion can be exercised once.

Drift is the failure mode the exercises exist to catch. A standby built two years ago and never launched is not the standby the document describes. Configuration diverges, dependencies get added in the primary Region only, IAM roles change. The interval between tests is the interval in which drift accumulates unseen, so cadence is part of the design rather than an afterthought.

A test needs a defined abort before it starts. Without stop conditions, an exercise is an outage with paperwork attached. Knowing which signal ends the test, who can call it, and how long the rollback takes is what makes it safe to run against production at all.

The organisation boundary decides which tools fit. Restores want an account of their own, the audit record belongs somewhere the production team cannot edit, and the standby spans a Region boundary inside the production account. So the mechanisms worth reaching for are the ones that drive several accounts from one place: experiments that run from an orchestrator account against tagged resources elsewhere, restore tests that land in a dedicated account, one policy shared across the organisation so every application is measured against the same targets. Anything that only works inside a single account turns the exercise into a scheduling problem between teams.

The useful output is a measured recovery time and a list of what broke. The second attempt is what proves the capability. The first says where it is not there yet.

What we’ll filter on

1. What does this level find, and what can it not find?
2. Does it need production, a spare account, or neither?
3. Can it run unattended on a schedule, or does it need a mobilisation each time?
4. Does it reach across accounts and Regions from one place?
5. How is it aborted, and how fast is the rollback?
6. Does it leave a dated record an auditor accepts?

The landscape

A tabletop walkthrough. The team reads the runbook against a scenario in a room, saying out loud what they would do at each step. It takes a couple of hours, touches nothing, and reliably finds missing steps, stale console instructions, and steps that depend on a person who has left. It is also the only level that reaches failback without any setup, because reading the failback section aloud is what reveals there isn’t one. It says nothing about whether the automation works.

Restoring from backup in a spare account. Take the actual recovery points and rebuild the workload where nothing depends on it. This tests the artefacts rather than the process, and it answers whether the data is recoverable at all, which a surprising number of organisations have never confirmed. AWS Backup restore testing automates it. A restore testing plan runs on a cron schedule, picks either the latest or a random recovery point from the vaults you nominate, restores it, records how long the restore job took, optionally runs a validation step, then deletes what it created. Assignable resource types include RDS, Aurora, EC2, EBS, S3, DynamoDB, EFS and FSx, so both halves of this workload are covered. AWS Backup Audit Manager carries a control that reports whether restore time met a target, which turns the run into the dated record. AWS recommends a dedicated account for restore tests, and since a plan restores from vaults in its own account, the recovery points have to be copied across that boundary first. None of it tests failover, DNS, or the primary Region getting out of the way.

Component failover in production. Failing over one thing while the rest keeps running. An RDS Multi-AZ instance fails over on demand with reboot-db-instance --force-failover, and an Aurora cluster with failover-db-cluster. This measures a real number for one step and exposes how the application behaves during it, which is usually where the surprises are. It has a limit worth being honest about here: neither command exercises a cross-Region read replica promotion, so the plan’s central step stays an estimate. On an Aurora global database, switchover-global-cluster does exercise the cross-Region step, at RPO zero and with the topology intact afterwards.

Fault Injection Service. Managed experiments that inject faults against real resources, scoped by tag, with stop conditions wired to CloudWatch alarms so a run halts itself when a guardrail trips. Two scenarios in its library fit this workload. AZ Availability: Power Interruption combines zonal compute loss, blocked rescaling of compute in that zone, subnet connectivity loss and an RDS failover. Cross-Region: Connectivity disrupts cross-Region VPC traffic, access to public endpoints, S3 cross-Region replication and DynamoDB global table replication. Individual actions reach the same places: aws:s3:bucket-pause-replication, aws:dynamodb:global-table-pause-replication, aws:network:transit-gateway-disrupt-cross-region-connectivity. Templates schedule natively through EventBridge Scheduler, one-off or recurring, and a multi-account template runs from an orchestrator account against targets in other accounts in the same Region. It reports what happened, not what to change.

AWS Resilience Hub. An assessment against a resilience policy that carries RTO and RPO targets per disruption type, separately for application, infrastructure, Availability Zone and Region. It returns an estimated workload RTO and RPO alongside a compliance status that reads as breached where the deployed configuration cannot meet the targets, and it names the components responsible. It also recommends FIS actions and scenarios for that specific application and runs them from its own console, so it is not only analysis, and one policy can be attached to as many applications as you have described. Sharing one policy with an organisation’s member accounts belongs to the next generation of Resilience Hub, launched on 28 May 2026, and only the management account or the delegated administrator can turn it on. What neither version can see is anything outside the deployed configuration.

Elastic Disaster Recovery. Agent-based continuous replication of servers into a staging area subnet in the recovery Region, built on low-cost storage and minimal compute. Its drills launch real instances from replicated data without disturbing the source or ongoing replication, and failback drills work the same way. Recovery plans sequence dependent servers in ordered steps, with wait steps of one to 120 minutes between them, and run in either drill or recovery mode. A plan covers one account and Region, and it does not terminate the instances a drill launches. It replicates servers, so the RDS instance here sits outside it; the fit is the EC2 tier, or a workload that keeps its state on the instances.

A full regional failover exercise. Promote the replica, scale the standby, flip the failover records, run production in the second Region for a defined period, then fail back. It is the only exercise that produces the actual number, and the only one that measures failback, which on a read-replica design means rebuilding replication from scratch. It needs a window, an executive sponsor, and a rollback everyone believes in.

Route 53 failover records and health checks. Detection and propagation are both part of recovery time and neither is a runbook step. Standard health checks poll every 30 seconds, fast ones every 10 at additional charge, and the record only flips after the configured number of consecutive failures, so a threshold of three at the standard interval is around 90 seconds before DNS changes at all. Then the TTL: 300 seconds leaves five minutes of cached answers behind the decision, and resolvers that ignore TTLs leave more. Where failover is a decision rather than a detection, ARC routing controls are the on-off switch, with safety rules that stop both Regions being switched off at once and a data plane spread across endpoints in five Regions. ARC’s readiness check, which audited quota and capacity parity between Regions, is closed to new customers. AWS’s documented replacement is ARC Region switch, whose plan evaluation monitors the state of a Region switch plan for readiness.

Evaluation

Side by side

Level	Finds	Needs production	Covers failback	Unattended on a schedule	Reaches other accounts	Evidence
Tabletop walkthrough	Missing and stale steps	✗	On paper	✗	✗	Weak
AWS Backup restore test	Whether data restores, and how long	✗	✗	✓	✓ via copies	Strong for restores
Resilience Hub assessment	Configuration gaps against targets	✗	✗	✓	✓	Moderate
Elastic DR drill	Whether real instances launch in order	✗	✓ drills	✗	✓	Strong for servers
Component failover	Real timing for one step	✓ narrow	✗	✓	✓	Strong for that step
FIS experiment	Behaviour under real failure	✓ scoped	✗	✓	✓	Strong
Full regional exercise	The measured RTO, and failback	✓ fully	✓	✗	✓	Definitive

Only the last row produces the number the auditor asked for, and it is the one the business will not authorise first. The sequence through the table is the answer. Each level makes the next safe enough to attempt, and by the time the full exercise runs it is confirming a number rather than discovering one. Read the failback column alongside it: four of the seven levels never touch failback, and on this design failback is the longer half.

The solution

Climb the ladder, and treat the first run of each level as measurement rather than assessment. The plan is being timed, and the failures are the deliverable.

Start with a Resilience Hub assessment and a tabletop in the same week. Between them they take two days and find the obvious failures. Resilience Hub compares the deployed configuration against the four-hour and fifteen-minute targets, per disruption type, and names the components that cannot meet them. Attach the same policy to the other applications you have described while you are there, because the same gap is usually in more than one place. The tabletop finds the human gaps: the step that says ask Dave, the console screenshot from a redesign ago, the approval nobody has authority to give at 3am.

Then restore the backups in an account nobody is using. This tells you whether the data is recoverable, separately from whether the failover works, and nothing else on the list has worse consequences when it fails. Do the first one by hand, from the actual automated backups and the actual copied AMIs rather than a fresh snapshot taken for the occasion, because a manual run is what exposes the missing steps. Then wire it into an AWS Backup restore testing plan so it repeats without anybody remembering, and turn on the Audit Manager control that checks restore time against the target so each run leaves its own evidence.

Next, take the component failovers in production, one at a time, in a low-traffic window. Trigger the RDS Multi-AZ failover and time it. Watch what the application does during those seconds, because that is where the hardcoded timeout with no retry lives. Each one replaces an estimate with a measured number for one line of the runbook. Be clear about what is still unmeasured: the cross-Region promotion is not among them, and it cannot be, which is the strongest argument available for moving this database to an Aurora global database where the cross-Region step is a switchover that can be run again next quarter.

Wire the repeatable ones into Fault Injection Service templates at that point, scoped by tag to this workload, with stop conditions on the alarms that already page. Run them from an orchestrator account so one template covers production and the accounts around it. Include the Cross-Region: Connectivity scenario, and pause S3 replication deliberately with aws:s3:bucket-pause-replication, because that measures the fifteen-minute RPO instead of assuming it. Schedule the templates monthly through EventBridge Scheduler, which moves drift from something found during an incident to something found on the second Tuesday.

Only then run the full regional exercise, with a window, a sponsor, a defined abort, and a rollback the team has rehearsed. Measure everything: time to decide, time to promote, time to scale, time for DNS to take effect, time to the first successful transaction in the second Region. Then fail back and measure that too. On a read-replica design, failback means standing up a new replica pointing the other way and waiting for it to catch up before the cutover can even start, so expect hours rather than minutes for it.

Set the cadence in the same decision. Component failovers and FIS experiments monthly, restore tests weekly now that they are automated, the full exercise annually, with the annual one scheduled rather than intended.

Why not go straight to the full exercise. It is the only test that produces the number, and running it first means discovering the missing certificate and the unreplicated secret inside the one window the business granted, then spending that window fixing rather than measuring.

Why not accept the Resilience Hub assessment as the evidence. It reads the deployed configuration and the experiments it recommends, so it will not tell you that step 6 of the runbook names a team that no longer exists, that the approval path has no delegate after hours, or how long DNS actually took. It narrows what the exercise has to find. It does not replace it.

Worked example

Resilience Hub reports the fifteen-minute RPO as unachievable for one component. The S3 bucket holding settlement documents has no replication configured, so its recovery point is the last backup, measured in hours. Nobody had noticed, because the plan says S3 is durable, and durability is a different property from being readable in a second Region.

The tabletop finds four missing steps, including the one that matters: the TLS certificate in the standby Region was issued for the old primary hostname and never reissued after a domain change. The failover would have completed and served certificate errors.

The first restore works, which is a relief and worth having confirmed rather than assumed. It takes 70 minutes against an estimate of 30, because the AMI copy had to be shared into the test account first, a step the runbook does not mention. Once that is fixed the team turns it into a weekly restore testing plan with a 90-minute target on the Audit Manager control.

The RDS Multi-AZ failover in production takes 71 seconds, inside the 60 to 120 seconds AWS publishes as typical. The application takes 4 minutes to serve traffic normally again, because a connection pool holds dead connections until an idle timeout expires. Nearly three of those four minutes are the pool, not the database, and a configuration change removes them.

The full exercise runs in March, in a window scheduled for six hours that needs nine. Time from decision to first successful transaction in the second Region is 3 hours 41 minutes, inside the four-hour objective, with 22 of those minutes spent on a manual approval the runbook described as immediate. Failback takes 5 hours 10 minutes, almost all of it waiting for a new replica to catch up, and nobody had ever estimated it because failback was not in the document.

The auditor gets a measured number, a dated record of the exercise, and a remediation list. The remediation list is what the team gets out of it: S3 Replication Time Control on the settlement bucket, which replicates most objects in seconds and 99.9 percent within 15 minutes and raises an OperationMissedThreshold event when it does not, plus a Batch Replication job for the documents already in the bucket, since a new rule only covers objects written after it. The certificate, the connection pool timeout and the AMI sharing step follow, along with an approval path with a named delegate. A second exercise is brought forward to six months rather than waiting a year, to confirm the list landed, and comes in at 2 hours 15 minutes. After that the cadence settles to annual.

What’s worth remembering

1. Untested plans have unknown recovery time. The written four hours is a sum of estimates that omits the steps nobody thought of.
2. Test in levels. Tabletop, restore test, component failover, injected fault, full regional exercise; only the last measures the real RTO and failback.
3. Replica promotion is one-way. Promoting an RDS read replica can be rehearsed once; an Aurora global database switchover keeps the topology, RPO zero.
4. Schedule tests to catch drift. AWS Backup restore testing and FIS templates both run unattended; the interval between tests is the drift window.
5. Measure DNS detection and failback separately. Three failed 30-second health checks is about 90 seconds before DNS flips, then the TTL; failback is usually longer.
6. The remediation list is the deliverable. A first exercise is valued for what broke; a run finding nothing has probably tested less than it thinks.