Exam-style
A Bedrock-backed assistant starts returning errors: the model in its primary region is failing roughly nine calls in ten, and the client is on legacy retry mode with max attempts raised well above the default, so every failure runs the full ladder and p99 has gone from two seconds to forty. What should be added to get user-facing latency back while the region recovers?
Reveal the answer
E. Open a circuit breaker, held in a store the whole fleet reads and keyed per model and region, so calls fail fast to a smaller fallback model until a probe succeeds
Backoff assumes a transient, sparse failure where the retry usually succeeds, which is what the AWS SDKs’ standard retry mode, with exponential backoff and full jitter, is built for. Standard mode also keeps a retry quota, a token budget that stops retries once it drains, but per client instance and returning the error rather than a fallback. Legacy mode has no standardised quota and retries at full rate, so every retry here adds delay to a request that fails anyway and a two-second p99 becomes forty. A breaker has three states: closed passes traffic; open fails immediately once a failure threshold trips; half-open lets one probe through, and a success closes it. The state must sit where every worker reads it: a DynamoDB item keyed per model and region, carrying an expiry timestamp the read compares against, not an in-process counter each Lambda execution environment starts fresh. TTL only clears those rows later: DynamoDB deletes expired items within a few days and serves them in reads until it does. Of the rest: raising max attempts lengthens the ladder; adaptive mode’s rate limiter reacts to throttling, not to a model that is down; a quota increase and a queue address demand, not a failed dependency.
Q. The model in the primary region is failing nine calls in ten and p99 has gone from two to forty seconds. More backoff, or a breaker?
A. A breaker. Backoff suits sparse transient failures, where the retry usually succeeds and a throttled call gets through on the second try; against a dependency failing most calls it adds the ladder to a request that fails anyway. Trip it on a shared failure threshold, fail fast to a smaller model or the same model in a second region, and let a half-open probe close it on recovery. Keep the state in a DynamoDB item per model and region, with an expiry timestamp the read compares against so the fleet trips together; TTL clears those rows later, within a few days.
Why? A breaker assumes the dependency is down, and retrying into a hard failure exhausts concurrency across every tool sharing the pool. Standard mode’s retry quota stops retries once its token budget drains, but per client instance, with no fallback. Both belong in one client: backoff for the sparse case, a breaker for the hard-down one. AWS documents the same shape in Step Functions: a Choice state reads the circuit status, routes to a Fail state while open, and TimeoutSeconds bounds the task.