Proving a DR Plan Works Before You Need It

The situation

A payments processor has a documented DR plan with a four-hour recovery time objective and a fifteen-minute recovery point objective. The plan describes a warm standby in a second region: RDS with a cross-region read replica, AMIs copied nightly, an Auto Scaling group at minimum capacity, and Route 53 health checks ready to shift traffic.

The plan has never been executed. It was written two years ago, reviewed annually as a document, and signed off each time. Nobody has promoted the replica, nobody has scaled the standby group, and nobody has confirmed that the application in the standby region can reach the things it needs.

An auditor has asked for evidence that the RTO is achievable. The business will not accept an unplanned production outage to find out, and the platform team suspects, without being able to say so in the review, that four hours is optimistic.

What actually matters

The first thing that matters is that an untested plan has an unknown recovery time rather than the one written down. The four hours came from adding up estimates, and estimates omit the steps nobody thought of: the manual approval, the DNS TTL, the certificate that only exists in the primary region, the secret that was never replicated. Every one of those is invisible until somebody tries.

The second is that testing has levels, and the argument about “we cannot test DR in production” usually collapses once the levels are separated. Reading the runbook aloud in a room finds missing steps. Executing it against a non-production copy finds broken automation. Failing over a single component in production finds integration problems. Failing over the whole workload finds the truth. Each level costs more and finds different things, and skipping straight to the last is what makes the argument feel binary.

The third is that drift is the failure mode that testing exists to catch. A standby built two years ago and never exercised is not the standby described in the document. Configuration diverges, dependencies get added in the primary region only, and IAM roles change. The interval between tests is the interval in which drift accumulates undetected, which makes cadence part of the design rather than an afterthought.

The fourth is that a test needs a defined abort. An exercise without stop conditions is an outage with paperwork. Knowing in advance what signal ends the test, who can call it, and how long the rollback takes is what makes it safe to run in production at all.

Underneath it, the useful output is not a pass. It is a measured recovery time and a list of the things that broke, because the second attempt is what proves the capability and the first attempt is what tells you where it isn’t.

What we’ll filter on

1. What does this level of testing actually find, and what can it not find?
2. Does it need production, a copy, or neither?
3. What does it cost in effort, spend, and risk?
4. How is it aborted, and how fast is the rollback?
5. Does it produce evidence an auditor accepts?
6. Can it run on a schedule, or does it need a mobilisation each time?

The landscape

A tabletop walkthrough. The team reads the runbook against a scenario in a room, saying out loud what they would do at each step. It costs a couple of hours, touches nothing, and reliably finds missing steps, stale console instructions, and steps that depend on a person who has left. It cannot find anything about whether the automation works.

Restoring into an isolated environment. Take the actual backups and the actual AMIs and build the workload in an account nobody is using. This tests the artefacts rather than the process, and it answers whether the data is recoverable at all, which is a question a surprising number of organisations have not confirmed. It does not test failover, DNS, or anything about the primary region getting out of the way.

AWS Elastic Disaster Recovery. Continuous block-level replication into a staging area in the recovery region, with the ability to launch drill instances that do not disturb the source or the actual recovery capability. The drill is the feature that matters here: it produces real instances from real replicated data on demand, so a recovery can be rehearsed repeatedly without touching production. It suits server-based workloads and is the closest thing to a managed answer for this whole problem.

Component-level failover in production. Failing over one thing while the rest keeps running. An RDS Multi-AZ failover can be triggered on demand via reboot-with-failover; an Aurora failover can be triggered directly. This measures a real number for one step and exposes the application’s behaviour during it, which is usually where the surprises are.

Fault Injection Service. Managed experiments that inject faults against real resources, scoped by tag, with stop conditions wired to CloudWatch alarms so the experiment halts itself when a guardrail trips. It covers instance termination, API throttling, network latency, and Availability Zone impairment. Experiments are templates, so they are repeatable and schedulable rather than mobilised each time. What it does not do is tell you what to change; it tells you what happened.

A full regional failover exercise. Promote the replica, scale the standby, shift DNS, run production in the second region for a defined period, then fail back. It is the only exercise that produces the actual number, and it is the only one that tests failback, which is the half of the plan nobody writes down. It needs a window, an executive sponsor, and a rollback everyone believes in.

Route 53 health checks and failover records. The routing mechanism itself, and one worth testing separately, because DNS TTL is a component of recovery time that no runbook step controls. A 300-second TTL puts five minutes of cached answers between the decision and the effect, and clients that ignore TTLs put more.

AWS Resilience Hub. Assesses a workload against defined RTO and RPO targets, identifies where the current configuration cannot meet them, and tracks the assessment over time. It is analysis rather than execution: it finds the gap between the target and the design without running anything.

Evaluation

Side by side

Level	Finds	Touches production	Tests failback	Repeatable	Evidence quality
Tabletop walkthrough	Missing and stale steps	✗	Partly	✓ cheap	Weak
Restore into isolation	Whether the data recovers	✗	✗	✓	Moderate
Resilience Hub assessment	Design gaps against targets	✗	✗	✓ continuous	Moderate
Elastic DR drill	Whether real instances launch	✗	✗	✓ on demand	Strong
Component failover	Real timing for one step	✓ narrow	✗	✓	Strong for that step
Fault Injection experiment	Behaviour under real failure	✓ scoped	✗	✓ scheduled	Strong
Full regional exercise	The actual RTO, and failback	✓ fully	✓	Annually at best	Definitive

Only the last row produces the number the auditor asked for, and it is the one the business will not authorise first. The sequence through the table is the answer: each level makes the next one safe enough to attempt, and by the time the full exercise runs it is confirming a number rather than discovering one.

The solution

Climb the ladder, and treat the first run of each level as discovery rather than assessment. The plan is not being tested for a pass; it is being measured, and the failures are the deliverable.

Start with a Resilience Hub assessment and a tabletop in the same week, because between them they cost two days and find the cheap failures. Resilience Hub compares the deployed configuration against the four-hour and fifteen-minute targets and names the components that cannot meet them. The tabletop finds the human gaps: the step that says “ask Dave”, the console screenshot from a redesign ago, the approval nobody has authority to give at 3am.

Then restore the backups into an isolated account. This answers whether the data is recoverable, separately from whether the failover works, and it is the question with the worst consequences if the answer is no. Do it from the actual automated backups and the actual copied AMIs, not from a fresh snapshot taken for the occasion.

Next, take the component failovers in production, one at a time, in a low-traffic window. Trigger an RDS failover and time it. Watch what the application does during those seconds, because that is where the hardcoded timeout with no retry lives. Each of these produces a real measured number for one line of the runbook, replacing an estimate, and the sum of them starts to show whether the four hours is defensible.

Wire the repeatable ones into Fault Injection Service templates at that point, scoped by tag to this workload, with stop conditions on the alarms that already page. Once they are templates they can run monthly on a schedule, which converts drift from something discovered during an incident into something discovered on the second Tuesday.

Only then run the full regional exercise, with a window, a sponsor, a defined abort, and a rollback the team has rehearsed. Measure everything: time to decide, time to promote, time to scale, time for DNS to take effect, time to first successful transaction in the second region. Then fail back, and measure that too, because failback is usually longer than failover and is almost never written down.

Set the cadence in the same decision. Component failovers and Fault Injection experiments monthly, a restore test quarterly, and the full exercise annually, with the annual one scheduled rather than intended.

Why not go straight to the full exercise. It is the only test that produces the number, and running it first means discovering the missing certificate and the unreplicated secret during the one window the business granted, then spending the window fixing rather than measuring.

Why not accept the Resilience Hub assessment as the evidence. It analyses the configuration and cannot tell you that the runbook’s step 6 references a team that no longer exists, or that DNS propagation adds twelve minutes. It narrows what the exercise has to find; it does not replace it.

Worked example

Resilience Hub reports that the RPO of fifteen minutes is not achievable for one component: an S3 bucket holding uploaded documents has no replication configured, so its recovery point is whatever the last backup was, measured in hours. Nobody had noticed, because the plan says “S3 is durable” and durability is not the same property as availability in a second region.

The tabletop finds four missing steps, including the one that matters: the TLS certificate in the standby region was issued for the primary hostname and never renewed after a domain change. The failover would have completed and served certificate errors.

The isolated restore works, which is a relief and worth having confirmed rather than assumed. It takes 70 minutes against an estimate of 30, because the AMI copy had to be shared into the test account first, a step the runbook does not mention.

The RDS failover in production takes 48 seconds. The application takes 4 minutes to fully recover, because a connection pool holds dead connections until an idle timeout expires. That gap is a configuration change, and it would have added four minutes to a real recovery at the worst possible moment.

The full exercise runs in March with a two-hour window. Actual time from decision to first successful transaction in the second region is 3 hours 41 minutes, inside the four-hour objective, with 22 of those minutes spent on a manual approval the runbook described as “immediate”. Failback takes 5 hours 10 minutes, which nobody had ever estimated because failback was not in the document.

The auditor gets a measured number, a dated record of the exercise, and a remediation list. The remediation list is the part the team values: S3 cross-region replication, the certificate, the connection pool timeout, the AMI sharing step, and an approval path with a named delegate. The second exercise, six months later, comes in at 2 hours 15 minutes.

What’s worth remembering

1. An untested DR plan has an unknown recovery time, not the one in the document; the written figure is a sum of estimates and omits the steps nobody thought of.
2. Testing has levels, and each makes the next safe enough to attempt: tabletop, isolated restore, component failover, injected fault, full regional exercise. Only the last produces the real number, and running it first wastes the one window you were given.
3. Drift is what testing catches, so cadence is part of the design: the interval between exercises is the interval in which the standby silently diverges from the primary.
4. Fault Injection Service makes the repeatable parts schedulable, with stop conditions on existing alarms bounding the blast radius, which turns an annual mobilisation into a monthly routine.
5. Measure failback, because it is usually longer than failover and is the half of the plan that never gets written down.
6. The deliverable from a first exercise is the remediation list, not a pass; a run that finds nothing has probably tested less than it thinks.