Exam-style
A Bedrock-backed assistant starts returning errors: the model in its primary region is failing roughly nine calls in ten, and because every failure runs the full retry ladder before giving up, p99 latency has gone from two seconds to forty. The on-call wants user-facing latency back while the region recovers. What should be added?
Reveal the answer
E. Open a circuit breaker, held in a store the whole fleet shares and keyed per model and region, so calls fail fast to a smaller fallback model until a probe succeeds
Backoff and a breaker assume different failures. Exponential backoff assumes the failure is transient and sparse, so the retry usually succeeds and the extra wait buys a good answer; that is the case the AWS SDK for exponential backoff and jitter is built for. When the dependency is failing nine calls in ten, every retry only adds latency to a request that was going to fail anyway, which is how a two-second p99 becomes forty. A breaker changes the response to a sick dependency. It has three states. Closed passes traffic. Open fails immediately, once a failure threshold trips. Half-open lets one probe through after a timer, and a success closes the breaker again. Failing fast to a smaller model, or to a copy of the same model in a healthy region, is one of the graceful degradation strategies that keeps answers flowing while the primary recovers. The state has to live where every worker can see it, so a DynamoDB item or an AWS AppConfig value keyed per model and region, not an in-process counter that each Lambda execution environment starts fresh and that never sees the rest of the fleet. Of the rest: raising the retry count lengthens the ladder that is already the problem, adaptive mode paces against throttling rather than a hard-down model, and a quota increase and an SQS queue both treat demand when the trouble is a dependency that is out.
Q. The model in the primary region is failing nine calls in ten and p99 has gone from two seconds to forty. More backoff, or a breaker?
A. A breaker. Backoff suits sparse transient failures, where the retry usually succeeds and a throttled call gets through on the second try; against a dependency failing most calls it adds the whole ladder to every doomed request. Trip the breaker on a shared failure threshold, fail fast to a smaller fallback model or the same model in a second region, and let a half-open probe close it when the primary comes back. Keep the state in DynamoDB or AppConfig, keyed per model and region, so the whole fleet trips together.
Why? Retry assumes recovery within seconds; a breaker assumes the dependency is out, and retrying into a hard failure turns one sick dependency into exhausted concurrency across every tool sharing the pool. Keeping the assistant in continuous operation during service disruptions needs both: backoff for the sparse case, circuit breaker patterns for the sick one. AWS Step Functions gives the same shape declaratively. A Choice state reads the breaker item before the model task, next to stopping conditions and per-task timeouts. Safeguarded workflows use circuit breakers to mitigate failures instead of deeper retries.