The situation
A B2B scheduling platform has just been through a Well-Architected review. The output is four lists.
Cost Explorer and Compute Optimizer between them identify about AUD$18,000 a month of over-provisioning across EC2 and RDS. Security Hub reports 63 open findings, 4 of them critical. The reliability section of the review flags a single-region deployment against a stated recovery objective the current architecture cannot meet. And the team has its own list, headed by a deployment pipeline that takes ninety minutes and fails about one run in five, which is why they deploy on Thursdays and dread it.
There is one quarter and three engineers. The exec sponsor wants to know what gets done and what does not, and expects the answer to be justified rather than asserted.
What actually matters
The first thing that matters is that these items are measured in incompatible units. Dollars a month, findings by severity, an unmet recovery objective, and engineer frustration do not compare directly, and any ranking that pretends they do is smuggling in a judgement. The useful move is to convert each to the same two questions: what does it cost us if we do nothing for a quarter, and what does it cost to fix.
The second is that some improvements make the others cheaper. A deployment pipeline that works is not a competing priority with the security findings; it is the mechanism by which the security fixes ship. Ranking it as one item among many misses that it is a multiplier on the rest of the quarter’s throughput.
The third is that risk without probability is not comparable to cost with certainty. AUD$18,000 a month of waste is a known, recurring, measurable loss. An unmet recovery objective is a loss of unknown size at unknown frequency. Both are real; treating the second as automatically more important because it is scarier, or automatically less important because it is hypothetical, are equally lazy. What makes it comparable is asking what the business has actually committed to and what happens when the commitment is missed.
The fourth is that effort estimates are the least reliable number in the exercise, and the ranking should survive them being wrong. An item that is worth doing only if it takes two weeks and not if it takes six is a fragile choice; an item worth doing either way is a safe one.
Underneath all of it, doing nothing has to be an option on the list, with its consequence stated. A quarter that commits to everything delivers a quarter of everything.
What we’ll filter on
- What does it cost, per month, to leave this alone?
- Is that cost certain and recurring, or probabilistic?
- What does the fix cost in engineer-weeks, and how confident is the estimate?
- Does fixing it make anything else cheaper or faster?
- Does it expire: is there a date after which this becomes much worse?
- Who accepts the consequence if it is deferred?
The landscape
AWS Well-Architected Tool. The framework and the workload review that produced three of these lists. It records answers per pillar, produces an improvement plan with high and medium risk items, and keeps the review as a versioned artefact so the next one is a delta rather than a restart. Its value here is that it has already run; its limit is that it ranks by pillar rather than across them.
Cost Explorer and Compute Optimizer. Between them, waste with a number attached. This is the only category on the list where the cost of inaction is directly measurable, which makes it the anchor everything else is compared against rather than simply the easiest item.
Security Hub findings, re-ranked. A severity score is a category ranking, not a probability. Re-ranking by reachability, whether anything internet-facing is affected, turns 63 findings into a much smaller urgent set and a larger backlog. Four critical findings on internal resources may be less pressing than a medium on the public path.
Resilience Hub. Assesses a workload against stated RTO and RPO targets and reports where the architecture cannot meet them. It converts “we should be multi-region” from an aspiration into a specific gap against a specific commitment, which is what makes it comparable to a dollar figure.
DORA-style delivery metrics. Deployment frequency, lead time, change failure rate and time to restore. The team’s ninety-minute pipeline with a 20% failure rate is a change failure rate of 20% and a deployment frequency of weekly, and stating it that way makes it a business metric rather than a complaint.
Trusted Advisor. A continuous checklist across cost, performance, security, fault tolerance, service limits and operational excellence. Useful here as the thing that keeps running after the quarter ends, catching the next drift without another review.
Service quotas. Worth a specific mention because they expire in a way most items do not. A workload approaching an account limit fails at a date determined by growth rather than by choice, and a quota increase request has a lead time. Anything in this category jumps the queue regardless of its size, because the option to do it later disappears.
Evaluation
Side by side
| Item | Cost of inaction | Certainty | Effort | Multiplier | Expires |
|---|---|---|---|---|---|
| Right-size EC2 and RDS | AUD$18k/month | Certain | 3 weeks | ✗ | ✗ |
| Fix the deployment pipeline | 20% failure rate, weekly releases | Certain | 4 weeks | ✓ everything ships through it | ✗ |
| Critical security findings (reachable) | Breach exposure | Probabilistic | 1 week | ✗ | ✗ |
| Remaining security findings | Audit exposure | Probabilistic | 5 weeks | ✗ | ✓ next audit |
| Second region | Unmet recovery objective | Probabilistic, high impact | 10 weeks+ | ✗ | ✗ |
| Service quota headroom | Hard failure at current growth | Certain, dated | 2 days | ✗ | ✓ by growth |
Two things fall out of the table that were not visible in four separate lists. The quota item is two days of work with a date attached, which makes it first by a margin that has nothing to do with its size. And the pipeline is the only row with a multiplier: every other item ships through it, so its four weeks buy back time on everything after it.
The second region is the item that will not fit. Ten weeks against a quarter with three engineers, alongside everything else, means committing to it is committing to it instead of the rest.
The solution
Quota headroom, then the pipeline, then the reachable security findings, then right-sizing. The second region gets a decision rather than a start.
Service quotas go first because they are the only item where the deadline is set by something other than us. Two days to review the limits the workload is approaching and raise the requests, which have their own lead time. Doing this in week one and finding out in week ten that the increase needs a support conversation is a better sequence than the reverse.
The pipeline goes next, against the instinct to do the cheap valuable thing first. Four weeks on delivery infrastructure feels like four weeks not spent on the list, and it is the item that changes the cost of every subsequent item. A 20% change failure rate means one deploy in five needs a rollback and a re-run, and ninety minutes of pipeline means each attempt costs a morning. Every security fix, every right-sizing change, and any second-region work all ship through it. Fixing it first means the rest of the quarter runs at a different speed.
Then the reachable security findings, which is one week rather than the six the full backlog would take, because the reachability triage separates the urgent from the merely open. Four critical findings turn out to be two that matter and two on resources nothing external can reach.
Right-sizing lands in the back half of the quarter, and lands faster than its three-week estimate because the pipeline now works. AUD$18,000 a month recurring is the largest certain number on the page and the argument for doing it earlier is strong; the argument for doing it after the pipeline is that the pipeline makes it a two-week job and de-risks the rollback if a resize is wrong.
The second region gets a decision, not a start. Ten weeks does not fit, and starting it means a half-built standby, which is worse than none because it invites the belief that there is one. The right output is to take the recovery objective back to the sponsor with the Resilience Hub assessment attached and ask which of two things is true: the objective is real, in which case it is next quarter’s single priority with proper funding, or the objective is aspirational, in which case it should be restated to something the architecture can meet. Either answer is progress. A quarter of unfunded partial work is not.
The remaining security findings get a conformance pack and remediation, so the backlog shrinks continuously rather than through effort, and Trusted Advisor stays on to catch the next drift.
Why not do the AUD$18,000 first. It is the most defensible item on the page in isolation and the ranking is not about items in isolation. Three weeks of right-sizing shipped through a pipeline that fails one run in five is not three weeks.
Why not start the second region and get partway. A standby that exists but has never been proven is the failure mode described in every DR post-mortem. Partial credit is not available here.
Worked example
The quota review finds three limits within 20% of current usage, one of which (Elastic IPs per region) would have blocked the second-region work if it had gone ahead. The increases are requested in week one and two of the three are granted within days; the third takes eleven days, which would have been an eleven-day stall at the worst possible moment.
The pipeline work takes five weeks rather than four. The estimate was wrong in the direction estimates usually are, and the ranking survives it, which is the property that made it a safe choice. Change failure rate falls to 4% and pipeline duration to 22 minutes, and the team moves from weekly Thursday deploys to deploying most days.
The security triage takes two days and the fixes take four. Two of the four critical findings are on an internal batch subnet with no route to the internet; they are recorded as backlog with reasons rather than fixed in the urgent window.
Right-sizing takes eleven days against a three-week estimate, and the difference is entirely the pipeline. Two resizes are wrong and get reverted the same day, which under the old pipeline would have been a same-week rollback and a reason not to try the next one.
The second-region conversation happens in week three, with the assessment attached. The sponsor’s answer is that the four-hour objective came from a customer contract covering 6% of revenue, and that a twelve-hour objective would satisfy the rest. The objective is restated for the majority of the business and the four-hour commitment is scoped to the customers who are paying for it, which makes next quarter’s work a third of the size. That conversation was worth more than ten weeks of building.
The quarter closes with AUD$18,000 a month recovered, a working pipeline, the reachable findings closed, and a recovery objective that means something. The four review lists still have items on them, and they are now ranked, owned and dated.
What’s worth remembering
- Convert every item to the same two questions before ranking: what does inaction cost per month, and what does the fix cost. Dollars, severity labels and unmet objectives do not compare until you do.
- Anything with an external deadline jumps the queue regardless of size; a service quota that fails at a date set by growth is two days of work that cannot be deferred like the rest.
- Find the multiplier. A delivery pipeline is not one item among many, because every other item ships through it, and fixing it changes the cost of everything after it.
- Re-rank security findings by reachability rather than by severity label; the urgent set is usually a small fraction of the open set, and the rest is a backlog for automated remediation.
- Certain recurring cost and probabilistic risk are both real; make them comparable by asking what the business has actually committed to, rather than by assuming risk always outranks waste or never does.
- An item that does not fit deserves a decision rather than a partial start, and taking an unmet objective back to the sponsor frequently resolves it more cheaply than building toward it would have.