Exam-style
A Bedrock-backed assistant starts returning errors: the model in its primary region is failing roughly nine calls in ten, and the client is configured with far more retry attempts than the SDK default, so every failure runs the full ladder before giving up and p99 latency has gone from two seconds to forty. The on-call wants user-facing latency back while the region recovers. What should be added?
Reveal the answer
E. Open a circuit breaker, held in a store the whole fleet reads and keyed per model and region, so calls fail fast to a smaller fallback model until a probe succeeds
Backoff assumes the failure is transient and sparse, so the retry usually succeeds; that is what the AWS SDKs’ standard retry mode, with exponential backoff and full jitter, is built for. When the dependency fails nine calls in ten, every retry adds delay to a request that fails anyway, which is how a two-second p99 becomes forty. A breaker has three states: closed passes traffic; open returns an immediate failure once a failure threshold trips; half-open lets one probe through after a timer, and a success closes it again. Failing fast to a smaller model, or to the same model in a healthy region, keeps answers flowing while the primary recovers. The state has to sit where every worker reads it, so a DynamoDB item keyed per model and region, with a TTL that expires the open circuit, rather than an in-process counter that each Lambda execution environment starts fresh. Of the rest: raising max attempts lengthens the ladder that is already the problem; adaptive mode adds a client-side rate limiter that reacts to throttling responses, not to a model that is down; and a quota increase and an SQS queue both address demand, not a failed dependency.
Q. The model in the primary region is failing nine calls in ten and p99 has gone from two seconds to forty. More backoff, or a breaker?
A. A breaker. Backoff suits sparse transient failures, where the retry usually succeeds and a throttled call gets through on the second try; against a dependency failing most calls it adds the whole ladder to every request that fails anyway. Trip the breaker on a shared failure threshold, fail fast to a smaller fallback model or the same model in a second region, and let a half-open probe close it when the primary comes back. Keep the state in a DynamoDB item keyed per model and region, with a TTL that expires the open circuit, so the whole fleet trips together.
Why? Retry assumes recovery within seconds. A breaker assumes the dependency is down, and retrying into a hard failure turns one failing dependency into exhausted concurrency across every tool sharing the pool. Both belong in the same client: backoff for the sparse case, a breaker for the hard-down one. AWS documents the same shape in Step Functions. A Choice state reads the circuit status before the model task and routes to a Fail state while the circuit is open, with TimeoutSeconds bounding the task itself.