Which Bedrock Errors to Retry and Which to Surface

The situation

A subscription retailer runs a support assistant on Amazon Bedrock. It answers questions about deliveries and account state, streams its answers back to a web client, and calls two tools through the model for order lookup and refund eligibility. It handles roughly forty thousand invocations a day, and about two percent of them fail.

The application log records one line per failure: model call failed. Every invocation, interactive and batch alike, goes through the same helper, which catches whatever came back, sleeps a second, and tries again up to three times. When the third attempt fails, the user sees a generic apology.

The symptoms are not evenly spread. Some users wait eleven seconds before the apology arrives. One class of question fails every single time, always after the full retry budget. A capacity incident three weeks ago got measurably worse in the minutes after the on-call engineer raised the retry count from three to five. Nobody can say which of these is which, because the log line is the same for all of them.

What actually matters

The decision to retry is a property of the error, not of the call site. A wrapper around every invocation has the call site and not the failure reason, so one policy covers every failure, and that policy will be wrong for most of them. The distinction is already in the response. Each failure arrives as a named exception with an HTTP status attached, and the name determines whether the same request sent again has any chance of a different outcome. Throwing that name away in a catch block and replacing it with model call failed is why the team cannot answer any of the questions above.

Blanket retry fails in two directions. On a permanent error it converts a fast failure into a slow one: three attempts, three round trips, eleven seconds of a user watching a spinner, and an identical result. Nothing is recovered and the diagnosis is delayed, because the retry hides how deterministic the failure was. On a capacity error it does worse than nothing. A quota is a ceiling on how much work the account may do in a window, so retrying sends more requests at a limit that is already rejecting them, and a fleet of clients backing off in lockstep produces a load spike that outlasts the original one. Raising the retry count during that incident made it worse for exactly this reason.

A second split runs underneath the first: some failures are the request’s fault and can only be fixed by changing the request. A prompt longer than the model’s context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined., an inference parameter the model does not support, a tool schema the model rejects as malformed, an image in a format that model cannot read. These fail identically at attempt one and attempt five hundred. They are code or configuration defects, and the fix belongs in a deploy, not in a runtime policy. The class of question that fails every time is almost certainly one of these, and the retry loop is how the team has avoided finding out which.

Then there is the streaming path, where the recovery problem changes shape. A call that fails before the first token has produced nothing, so retrying it is invisible to the user. A call that fails part-way through a response has already delivered bytes to a browser, and retrying gives the user a second answer stapled to half of a first one. The failure mode and the remedy both depend on whether the failure arrived before or after the stream opened, which is a distinction the one-line log cannot express and the shared wrapper cannot act on.

What we’ll filter on

1. Cause: did the request cause this, did the account’s configuration cause it, or did the service?
2. Effect of retrying: does another attempt have a real chance of succeeding, no chance at all, or a chance of making things worse?
3. Where the fix lives: in the request the code builds, in configuration and quotas, or in the retry policy itself.
4. Timing: can this failure arrive after the response has started streaming to the user?
5. What the caller sees: a retryable blip the user should never learn about, or a condition somebody has to be told about.

The landscape

Bedrock’s runtime exceptions sort into four groups, and the grouping is the taxonomy a retry policy should be written against.

The request is wrong

ValidationException (HTTP 400) covers the request the code built. The prompt plus the conversation history exceeds the model’s context window. An inference parameter sits outside the range that model accepts, or is one it does not implement at all. The tool schema in a function-calling request is malformed, or references a type that model does not accept. An image or document arrives in an unsupported format or over the size limit. Bedrock returns the 400 without running inference, and the same bytes fail the same check every time. Every one of these is fixed by changing what the client sends.

AccessDeniedException (403) is the permission answer, and it splits into two causes that look identical from the client. Either the calling principal’s IAM policy does not allow the action on that model resource, or the account has never been granted access to that foundation model. Both are configuration, both need a human, and neither changes because a loop tried again a second later.

ResourceNotFoundException (404) says the resource ARN you named was not found: a model identifier that is wrong or belongs to another Region, a knowledge base or agent alias that was deleted, a provisioned throughput ARN from a different account. This is nearly always a deployment defect, a stale environment variable or a config file pointing at a Region the resource was never created in.

The account is at its ceiling

ThrottlingException (429) means the account exceeded a Bedrock quota for that model in that Region. Those quotas are token-based: tokens per minute for on-demand invocation, a separate tokens-per-minute quota for calls made through a cross-Region inference profile, and a tokens-per-day ceiling that starts at the per-minute figure multiplied by 1,440. Some models carry a requests-per-minute quota on top and some do not. A minute-scale throttle clears on its own, and backoff with jitter is the right first response. A day-scale one does not clear until tomorrow. Telling a transient throttle from a structural one is its own piece of work, and backoff is only the right answer for the transient half.

ServiceQuotaExceededException is the harder relative, and it arrives as an HTTP 400 rather than a 429, so a policy keyed on status codes files it with the validation errors. Converse and ConverseStream do not return it at all; InvokeModel and InvokeModelWithResponseStream do. AWS says you can resubmit the request later, which is true on the horizon of a quota window and false on the horizon a retry loop works over. A loop does not raise an account quota; a quota increase request does. Treating it as a throttle is how a capacity problem becomes a retry storm.

ModelNotReadyException (429) belongs to Custom Model Import. Bedrock removes imported models that are not in active use, and the first call after a removal starts restoring the model rather than serving it. Restoration time depends on the model’s size and on fleet availability, and AWS documents the request as served within five minutes or returning this exception. It is retryable, and each attempt continues the restoration, but on a horizon of minutes rather than the milliseconds a throttle backoff covers, so a policy tuned for throttles exhausts its attempts long before the model is loaded.

The service or the model failed

InternalServerException (500) and ServiceUnavailableException (503) are the ordinary transient service faults, and they are retryable with backoff in the same way a throttle is. AWS states plainly that a 503 is demand or capacity and not an account quota, and the two get conflated in incident write-ups more often than anything else here. Where they differ from a throttle is what to do when they persist: a Region having a bad few minutes is answered by sending the work somewhere else, which is what a cross-region inference profile does with no change in the client.

A third transient fault is easy to miss because the SDK has no exception class for it. An overloaded_error comes back as HTTP 529 when the model has insufficient serving capacity, separately from the 429 that signals a quota. It is retryable with backoff and jitter, and where the response carries a Retry-After header, wait that long instead of a locally computed delay.

ModelTimeoutException (408) says processing exceeded the model timeout. It is retryable, and it is also a signal about the request, because a long input or a large requested output makes it much more likely. A retry that usually succeeds while the same request times out one call in twenty is a symptom to chase, not a fix. ModelErrorException (424) reports a failure while processing the model, and it carries the original status code and the resource name. Retry it once, then log the input’s shape, because repeated failures on one document point at that document.

The failure arrived mid-stream

ModelStreamErrorException is the one that breaks the wrapper. It is an HTTP 424 that only the streaming operations raise, and it arrives inside the response stream, after the connection succeeded and after tokens have already been delivered. It carries the original status code and message, so the underlying cause is still recoverable from the log. The transport-level answer is retryable; the application-level answer is not automatic, because the client has already rendered half an answer. Anything that streams to a user has to decide in advance whether a mid-stream failure discards the partial output and starts again, or keeps it and appends an explicit truncation notice.

Evaluation

Side by side

Read down the two middle columns and the shape of the answer appears. Eight of the twelve are worth another attempt and four are not, and cutting across that split, three get materially worse when you retry them. Two of those three sit in the never-retry group. The third is ModelStreamErrorException, retryable at the transport level and damaging at the application level, which is why it needs a rule of its own rather than a column. A wrapper that retries everything is correct on seven rows, wasteful on two, and damaging on three. The five rows it gets wrong hold both the failures a user waits eleven seconds for and the failures that turn a busy afternoon into an incident.

The gates in order

The solution

The retry policy is the smallest part of the work. Diagnosis needs three things the current application does not have: error logging worth reading, request validation before the call, and response analysis after it.

Error logging that answers the question

model call failed is a log line that cannot support any decision. Replace it with a structured record carrying the exception name, the AWS request ID from the response metadata, the model identifier or inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. the call went to, the Region, the input and output token counts where the response has them, the attempt number, and the elapsed time. The request ID is what lets a support case go anywhere; the exception name is what lets you count failures by class instead of in aggregate; the token counts are what tell you whether a timeout is correlated with input size.

{
  "event": "bedrock_invoke_failed",
  "exception": "ValidationException",
  "request_id": "b81f2c1a-2f0f-4a2f-9c3d-7f2a1e5b90dd",
  "model_id": "eu.anthropic.claude-sonnet-4-5-20250929-v1:0",
  "region": "eu-west-2",
  "attempt": 1,
  "retryable": false,
  "input_tokens": 214113,
  "max_tokens": 4096,
  "elapsed_ms": 142,
  "detail": "input length exceeds the model context window"
}

With that shape in the log group the application already writes to, the two percent failure rate decomposes in a single query: count by exception name over a day and the mixture of causes stops being a mystery. Emit a metric per exception name alongside it, because a rising ValidationException count is a deploy that broke something and a rising ThrottlingException count is traffic growth, and the two should wake different people.

Request validation before the call

Several of the never-retry errors are checkable before the request leaves the process, and catching them locally turns a round trip and a user-visible failure into a handled branch. Bedrock’s CountTokens operation returns the input token count for an InvokeModel or Converse body without running inference and without a charge. Compare that count, over the system prompt, the conversation history and the retrieved context together, against the target model’s context window, and take the branch you chose for oversized inputs rather than sending it and hoping. Some Claude models offered only through cross-Region inference do not support CountTokens on the bedrock-runtime endpoint, so check the model card before building the check around it. An explicit input budget makes the comparison a subtraction rather than a guess.

The same applies to the tool schemas a function-calling request carries. Validate them against the model’s expected schema at build time or at service start, not per request, so a malformed tool definition fails the deploy instead of failing two percent of production traffic. Check inference parameters against what the target model supports. That matters most when the model identifier is configurable, because a switch to a different provider’s model invalidates a parameter that worked yesterday with nothing failing at deploy time.

Response analysis after it

The failures that never raise an exception are the ones most likely to be misread. A successful Converse response carries a stop reason, and only end_turn and tool_use are the ordinary case. The rest are max_tokens, stop_sequence, guardrail_intervened, content_filtered, malformed_model_output, malformed_tool_use and model_context_window_exceeded. A max_tokens stop means output ended at the configured limit rather than at the end of an answer. Nothing threw. The HTTP status was 200. The answer is wrong anyway, and truncation analysis starts here: log the stop reason on every call, alarm when the max_tokens share of responses rises, and treat a truncated answer as a failure class of its own rather than as a quality complaint.

That list also settles a trap in the taxonomy above. model_context_window_exceeded means an oversized input can land as a 200 with a stop reason instead of as a ValidationException, so a handler that watches only the exception path never sees them. Two more response shapes deserve the same treatment. An empty or missing content block on a 200 is a real outcome, usually a response carrying only a tool-use block or one cut off before any text, and code that assumes text is present will throw somewhere far from the cause. And a guardrail_intervened stop reason means a blocked response rather than a failure, so count it separately from errors and a policy change will not look like a reliability regression.

The retry policy itself

With the taxonomy in hand, the policy is short. Never retry ValidationException, AccessDeniedException or ResourceNotFoundException; surface them with the detail message intact. Never loop on ServiceQuotaExceededException; shed or defer the work and raise the quota. Retry ThrottlingException, InternalServerException, ServiceUnavailableException, the 529 overload, ModelTimeoutException and ModelErrorException with exponential backoff and jitter, and give ModelNotReadyException a longer horizon.

Most of that is already in the SDK, so the hand-rolled loop should go rather than be improved. Standard retry mode is the default across the SDKs. It classifies ValidationException, AccessDeniedException and ResourceNotFoundException as non-retryable and returns them straight to the caller, retries transient and throttling errors with exponential backoff and full jitter, waits longer before retrying a throttle than a transient fault, and stops retrying once its retry quota depletes under sustained failure. The default of three attempts is one request and two retries, adjustable per client. Cap the total wait so an interactive request degrades rather than hangs, and let the batch path take a much larger number of attempts because nobody is watching it. Handle ModelStreamErrorException in the client that owns the stream, where the partial output actually is.

Worked example

Come back to the class of questions that fails every time.

The new logging shows it in one query. Ninety-one percent of the daily failures are ThrottlingException clustered in the afternoon peak, which backoff already absorbs and which the users never see. Six percent are ModelTimeoutException, concentrated on the longest inputs. Three percent, roughly two hundred calls a day, are ValidationException with the detail input length exceeds the model context window, and every one of them carries an input token count over two hundred thousand.

Those are the ones taking eleven seconds. The retry wrapper was sending a request three times that the service rejected on sight, and the user waited out three rejections to receive the same generic apology. Once the wrapper stops retrying them, they fail in 140 milliseconds and the log names the cause.

The cause turns out to be a single subscriber with four years of order history, whose conversation context assembles a prompt well past the model’s two-hundred-thousand-token context window. A CountTokens call before the invocation, and trimming the history to the most recent exchanges, fixes it in a deploy, and the failure class disappears.

Without the taxonomy the fix would have been something else entirely. Before the exception names were logged, the failure looked like a quality problem: long, complicated accounts got a useless reply, and the working theory was that the assistant could not cope with complex histories. A sprint of prompt engineering was already scheduled. The same misreading has a sibling in the responses that succeed. A stop reason of max_tokens marks an answer that ended at the configured output limit. Read as poor quality, it sends a team to rewrite prompts when the fix is a higher limit or a shorter requested answer. Both are the same mistake, which is reading a mechanical limit as a defect in the answer.

What’s worth remembering

1. The retry decision belongs to the error, not to the call site, so a wrapper that retries every failure identically is wrong on five of the twelve classes Bedrock returns.
2. ValidationException, AccessDeniedException and ResourceNotFoundException never change on a second attempt; retrying them adds three round trips of latency and delays the diagnosis.
3. ThrottlingException is a 429 that takes exponential backoff and jitter; ServiceQuotaExceededException is a 400 against an account quota that no loop will move.
4. ModelStreamErrorException arrives after tokens have reached the user, so the client that owns the stream has to decide up front between discarding the partial answer and keeping it with a truncation notice.
5. Log the exception name, request ID, model identifier and token counts on every failure; model call failed supports no decision anybody needs to make.
6. Watch the stop reasons on successful calls: max_tokens is truncation at the configured limit, and model_context_window_exceeded puts an oversized input on the 200 path rather than the exception path.