The situation
A grocery delivery platform had a two-hour outage caused by a recommendations service. The recommendations service is not important: it populates a “you might also like” panel, and the site works without it.
What happened is that it got slow rather than failing. Its p99 went from 40 ms to 30 seconds. The checkout service calls it synchronously with a 60-second timeout inherited from a default nobody had set deliberately. Threads in the checkout service filled waiting on it, new requests queued, health checks began timing out, the load balancer removed targets, and the remaining targets received more load and filled faster. The site went down because a panel of product suggestions was slow.
The requirement now is that a non-essential dependency cannot do this again, across a service graph of about thirty services where nobody has written down which dependencies are essential.
What actually matters
The first thing that matters is that slow is worse than down. A dependency returning errors immediately frees the caller’s thread; a dependency that hangs holds it. Most resilience thinking is about availability, and the failure that propagates is latency.
The second is that the classification is missing and it is the actual work. “Which dependencies may take the request down with them” is a product decision, not a technical one, and until somebody answers it per call, no timeout value is defensible. A timeout is an assertion about how long a dependency is allowed to matter.
The third is that timeouts alone are insufficient. A one-second timeout on a dependency that is failing means every request waits one second and then fails, which is better than 60 and still means the caller’s capacity is consumed by a dependency that is not working. A circuit breaker stops calling it at all after a threshold, which returns the capacity.
The fourth is that retries make it worse if they are naive. A dependency that is struggling under load and receives three times the requests because everyone is retrying is a dependency that will not recover. Backoff, jitter and a retry budget are what stop a retry policy becoming a denial of service against your own system.
Underneath it, the health check design contributed. A health check that fails because a non-essential dependency is slow tells the load balancer to remove a target that could still serve most requests.
What we’ll filter on
- Is this dependency essential to the request, or optional?
- What does the caller do when it fails: fail, fall back, or omit?
- How long is it allowed to take before that decision?
- Does the caller stop calling it when it is clearly broken?
- Do retries make the dependency’s situation worse?
- Does this dependency’s health affect the caller’s health check?
The landscape
Timeouts. The first control and the one most often left at a default. Every client library has one, most defaults are far too generous, and the value should be derived from the dependency’s observed p99 plus headroom rather than chosen for comfort. A timeout is what bounds the damage; it does not prevent it.
Circuit breakers. After a threshold of failures, stop calling the dependency and fail immediately, retrying occasionally to see whether it has recovered. This returns the caller’s capacity rather than spending it on calls that will time out. Implemented in the application, in a service mesh, or in some SDKs directly.
Fallbacks. What the caller does instead. A cached previous response, a default, or omitting the feature. For the recommendations panel, omitting it is the correct fallback and the whole failure would have been a panel that did not render.
Bulkheads. Separate connection or thread pools per dependency, so one slow dependency cannot consume the resources needed to call the others. This is the control that stops a single dependency filling the service’s entire capacity, and it is the one most often missing.
Retries with exponential backoff and jitter. Retry on transient failure, waiting longer each time, with randomness so callers do not synchronise into waves. The AWS SDKs implement this and it is worth checking rather than assuming, because a retry policy with no jitter turns a brief blip into a thundering herd.
Retry budgets and adaptive retries. A limit on what fraction of requests may be retries, so a broadly failing dependency does not receive several times its normal load. The AWS SDKs’ adaptive retry mode does this.
SQS between the caller and the work. Where the call does not have to be synchronous, a queue absorbs the mismatch entirely: the producer’s latency is the queue’s put latency regardless of what the consumer is doing. This removes the failure mode rather than bounding it, and it changes the interaction’s semantics.
Health check design. A health check that only reflects the service’s own ability to serve should not fail because an optional dependency is unavailable. Separating a liveness check from a dependency-aware readiness check is what stops the load balancer removing targets that are still useful.
Service mesh policy. For a large graph, expressing timeouts, retries and circuit breaking as mesh configuration rather than in each application means the policy is consistent and changeable without a deployment.
Evaluation
Side by side
| Control | Stops slow propagating | Returns caller capacity | Needs classification | Where it lives |
|---|---|---|---|---|
| Tightened timeout | Bounds it | ✗ still waits | ✓ | Client config |
| Circuit breaker | ✓ | ✓ | ✓ | App or mesh |
| Fallback | ✓ user-visibly | ✓ | ✓ | Application |
| Bulkhead | ✓ contains it | ✓ for others | ✗ | App or mesh |
| Backoff, jitter, retry budget | Prevents amplification | Partly | ✗ | SDK config |
| Queue instead of a call | Removes it | ✓ | Design change | Architecture |
| Readiness vs liveness split | ✓ | n/a | ✓ | Health check |
Two rows need no classification and should be applied everywhere: bulkheads and sane retry behaviour. The rest depend on knowing whether a dependency is essential, which is why the classification is the first work rather than a documentation exercise to do afterwards.
The solution
Classify every dependency as essential or optional, then give the optional ones a tight timeout, a circuit breaker and a fallback, and give every dependency a bulkhead regardless.
The classification comes first and it is a product conversation. For each call in each service, one question: if this returns nothing, does the request still have value? Recommendations, review counts, personalisation and analytics beacons are optional. Inventory, pricing and payment are essential. Write the answer down next to the call, because the timeout, the fallback and the circuit breaker configuration all follow from it and none of them is defensible without it.
Apply bulkheads everywhere, because they need no classification and they are what stops one dependency consuming the service’s whole capacity. A separate connection pool per dependency, sized so that exhausting one leaves the others working, converts a total outage into a partial degradation as a structural property.
For the optional dependencies, set the timeout from the observed p99 plus headroom rather than from a default, add a circuit breaker that opens after a small number of consecutive failures, and implement the fallback explicitly. The recommendations panel gets a 200 ms timeout, a breaker, and a fallback that renders nothing. Under the same incident the panel would have been missing and checkout would have been fine.
For the essential dependencies, the timeout still needs tightening and the fallback is failing the request cleanly with a clear error, which is better than holding a thread for 60 seconds. Where an essential call could be made asynchronous, a queue removes the coupling, though that is a design change rather than a configuration one.
Fix the retry behaviour globally. Exponential backoff with jitter, and a retry budget or adaptive mode so a broadly failing dependency does not receive multiples of its normal load from callers trying to help.
Split the health checks. A liveness check that answers “is this process working” and a readiness check that answers “can this instance serve traffic” should not fail because an optional dependency is slow. In the incident, health checks failing on a recommendations timeout is what turned a slow panel into targets being removed from the load balancer.
Then prove it. A Fault Injection Service experiment injecting latency into the recommendations service, scoped by tag with stop conditions, tests whether the design behaves as designed. Run it in staging, then in production during a quiet period, and expect to find at least one dependency somebody classified wrongly.
Why not just set every timeout to one second. It bounds the damage and it spends a second of every request’s capacity on a dependency that is not working, and it does nothing about the health check interaction that removed targets.
Why not queue everything. It removes this failure mode completely and changes what the application promises, since a synchronous read becomes an eventually-available result. Right where the semantics allow it and wrong as a blanket answer.
Worked example
The classification takes two workshops across thirty services and produces 140 dependency edges, of which 51 are classified optional. The surprise is how many nobody had thought about: an analytics beacon called synchronously in the checkout path, and a currency-conversion service called on every product view when the result changes daily.
Bulkheads go in first, in the shared HTTP client library, with a pool per host. The rollout is a library version bump across services and takes three weeks.
The recommendations call gets a 200 ms timeout against an observed p99 of 40 ms, a breaker opening after five consecutive failures, and a fallback returning an empty list. The panel disappears when the service is unwell, which product accepts immediately once it is framed against a two-hour outage.
The analytics beacon is moved off the request path entirely into a queue, which is both more correct and faster.
The Fault Injection experiment finds two misclassifications. A “reviews” service classified optional turns out to be the source of the product page’s structured data, so omitting it produces a page that renders and fails validation downstream. And an inventory call classified essential is only essential for one of three page types, so it gets a per-path classification rather than a per-dependency one.
Four months later the recommendations service has another bad afternoon, with p99 above ten seconds for about forty minutes. The panel is absent, the breaker opens after the first few requests, checkout is unaffected, and the incident is a ticket rather than a page.
What’s worth remembering
- Slow is worse than down: a dependency returning errors frees the caller’s thread and one that hangs holds it, which is how a non-essential service takes down an application.
- The classification (essential or optional, per call) is the first work, because the timeout, the fallback and the circuit breaker are all undefendable without it.
- Timeouts bound the damage and do not return capacity; a circuit breaker stops calling a broken dependency altogether, which is what gives the caller its threads back.
- Bulkheads need no classification and should be applied everywhere: a connection pool per dependency turns a total outage into a partial degradation structurally.
- Retries without backoff, jitter and a budget amplify a struggling dependency’s load and stop it recovering.
- A health check that fails because an optional dependency is slow tells the load balancer to remove a target that could still serve most requests, which converts degradation into an outage.