Which Bedrock Errors to Retry and Which to Surface

The situation

A subscription retailer runs a support assistant on Amazon Bedrock. It answers questions about deliveries and account state, streams its answers back to a web client, and calls two tools through the model for order lookup and refund eligibility. It handles roughly forty thousand invocations a day, and about two percent of them fail.

The application log records one line per failure: model call failed. Every invocation, interactive and batch alike, goes through the same helper, which catches whatever came back, sleeps a second, and tries again up to three times. When the third attempt fails, the user sees a generic apology.

The symptoms are not evenly spread. Some users wait eleven seconds before the apology arrives. One class of question fails every single time, always after the full retry budget. A capacity incident three weeks ago got measurably worse in the minutes after the on-call engineer raised the retry count from three to five. Nobody can say which of these is which, because the log line is the same for all of them.

What actually matters

The decision to retry is a property of the error, not of the call site. A wrapper around every invocation knows where the call was made and nothing about why it failed, so it has to guess, and a single guess applied to every failure will be wrong for most of them. Bedrock already draws the distinction in the response: each failure comes back as a named exception with an HTTP status behind it, and the name says whether the same request sent again has any chance of a different outcome. Throwing that name away in a catch block and replacing it with model call failed is the reason the team cannot answer any of the questions above.

Blanket retry fails in two directions. On a permanent error it converts a fast failure into a slow one: three attempts, three round trips, eleven seconds of a user watching a spinner, and an identical result. Nothing is recovered and the diagnosis is delayed, because the retry hides how deterministic the failure was. On a capacity error it does worse than nothing. A quota ceiling is a limit on how much work the account may do, so retrying pushes more work at the thing already refusing work, and a fleet of clients backing off in lockstep produces a load spike that outlasts the original one. Raising the retry count during that incident made it worse for exactly this reason.

A second split runs underneath the first: some failures are the request’s fault and can only be fixed by changing the request. A prompt longer than the model’s context windowContext windowThe maximum number of tokens an LLM can attend to in a single call – prompt plus output combined., an inference parameter the model does not support, a tool schema the model rejects as malformed, an image in a format that model cannot read. These fail identically at attempt one and attempt five hundred. They are code or configuration defects, and the fix belongs in a deploy, not in a runtime policy. The class of question that fails every time is almost certainly one of these, and the retry loop is how the team has avoided finding out which.

Then there is the streaming path, where the recovery problem changes shape. A call that fails before the first token has produced nothing, so retrying it is invisible to the user. A call that fails part-way through a response has already delivered bytes to a browser, and retrying gives the user a second answer stapled to half of a first one. The failure mode and the remedy both depend on whether the failure arrived before or after the stream opened, which is a distinction the one-line log cannot express and the shared wrapper cannot act on.

What we’ll filter on

1. Cause: did the request cause this, did the account’s configuration cause it, or did the service?
2. Effect of retrying: does another attempt have a real chance of succeeding, no chance at all, or a chance of making things worse?
3. Where the fix lives: in the request the code builds, in configuration and quotas, or in the retry policy itself.
4. Timing: can this failure arrive after the response has started streaming to the user?
5. What the caller sees: a retryable blip the user should never learn about, or a condition somebody has to be told about.

The landscape

Bedrock’s runtime exceptions sort into four groups, and the grouping is the taxonomy a retry policy should be written against.

The request is wrong

ValidationException covers the request the code built. The prompt plus the conversation history exceeds the model’s context window. An inference parameter sits outside the range that model accepts, or is one it does not implement at all. The tool schema in a function-calling request is malformed, or references a type the model will not take. An image or document arrives in an unsupported format or over the size limit. The service rejects the request without running inference, so there is no charge and no wait, and the same bytes will be rejected again every time. Every one of these is fixed by changing what the client sends.

AccessDeniedException is the permission answer, and it splits into two causes that look identical from the client. Either the calling principal’s IAM policy does not allow the action on that model resource, or the account has never been granted access to that foundation model. Both are configuration, both need a human, and neither changes because a loop tried again a second later.

ResourceNotFoundException says the thing you named does not exist: a model identifier that is wrong or belongs to another region, a knowledge base or agent alias that was deleted, a provisioned throughput ARN from a different account. This is nearly always a deployment defect, a stale environment variable or a config file pointing at a region the resource was never created in.

The account is at its ceiling

ThrottlingException means the request exceeded the per-minute request or token rate for that model in that region. It is transient by nature, and it is the one error class where retrying with exponential backoff and jitter is the correct and complete first response. Telling a transient throttle from a structural one is its own piece of work, and backoff is only the right answer for the transient half.

ServiceQuotaExceededException is the harder relative. It reports a limit that retrying will not move, because it is a ceiling on the account rather than a burst against a rate. Backing off and trying again does not raise a quota; a quota increase request does. Treating it as a throttle is how a capacity problem becomes a retry storm.

ModelNotReadyException appears on provisioned throughput that has not finished warming, and it resolves itself given time. It is retryable, but on a scale of tens of seconds rather than the hundreds of milliseconds a throttle wants, so a policy tuned for throttles will exhaust its budget before the capacity is ready. A newly provisioned model unit produces this during every cutover.

The service or the model failed

InternalServerException and ServiceUnavailableException are the ordinary transient service faults, and they are retryable with backoff in the same way a throttle is. Where they differ is what to do when they persist: a region having a bad few minutes is best answered by sending the work somewhere else, which is what a cross-region inference profile does without the client knowing about it.

ModelTimeoutException says the model did not return within the allowed time. It is retryable, and it is also a signal about the request. A long input or a large requested output makes it much more likely. A retry that usually succeeds while the same request times out one call in twenty is a symptom to chase, not a fix. ModelErrorException reports that the model itself failed on this input, which is retryable once but worth logging with the input’s shape, because a model that fails repeatedly on one document is telling you something about that document.

The failure arrived mid-stream

ModelStreamErrorException is the one that breaks the wrapper. It arrives inside the response stream, after the connection succeeded and after tokens have already been delivered. The transport-level answer is retryable; the application-level answer is not automatic, because the client has already rendered half an answer. Anything that streams to a user has to decide in advance whether a mid-stream failure discards the partial output and starts again, or keeps it and appends an explicit truncation notice.

Evaluation

Side by side

Read down the two middle columns and the shape of the answer appears. Seven of the eleven are worth retrying and four must never be retried, and cutting across that split, three get materially worse when you do. Two of those three sit in the never-retry group; the third is ModelStreamErrorException, retryable at the transport level and damaging at the application level, which is why it needs a rule of its own rather than a column. A wrapper that retries everything is correct on six rows, wasteful on two, and damaging on three. The five rows it gets wrong hold both the failures a user waits eleven seconds for and the failures that turn a busy afternoon into an incident.

The gates in order

The solution

The retry policy is the smallest part of the work. Diagnosis needs three things the current application does not have: error logging worth reading, request validation before the call, and response analysis after it.

Error logging that answers the question

model call failed is a log line that cannot support any decision. Replace it with a structured record carrying the exception name, the AWS request ID from the response metadata, the model identifier or inference profileInference profileA Bedrock resource wrapping a model so calls to it can be tagged, routed across regions, or repointed without changing app code. the call went to, the region, the input and output token counts where the response has them, the attempt number, and the elapsed time. The request ID is what lets a support case go anywhere; the exception name is what lets you count failures by class instead of in aggregate; the token counts are what tell you whether a timeout is correlated with input size.

{
  "event": "bedrock_invoke_failed",
  "exception": "ValidationException",
  "request_id": "b81f2c1a-2f0f-4a2f-9c3d-7f2a1e5b90dd",
  "model_id": "eu.anthropic.claude-3-5-sonnet-20241022-v2:0",
  "region": "eu-west-2",
  "attempt": 1,
  "retryable": false,
  "input_tokens": 214113,
  "max_tokens": 4096,
  "elapsed_ms": 142,
  "detail": "input length exceeds the model context window"
}

With that shape in the log group the application already writes to, the two percent failure rate decomposes in a single query: count by exception name over a day and the mixture of causes stops being a mystery. Emit a metric per exception name alongside it, because a rising ValidationException count is a deploy that broke something and a rising ThrottlingException count is traffic growth, and they want different people woken up.

Request validation before the call

Several of the never-retry errors are checkable before the request leaves the process, and catching them locally turns a paid round trip and a user-visible failure into a handled branch. Count the tokens of the assembled prompt, including the system prompt, the conversation history and the retrieved context, against the target model’s context window, and take the branch you chose for oversized inputs rather than sending it and hoping. Context window overflow diagnostics belong on the client side because the client is the only place that knows what it is about to concatenate, and an explicit input budget makes the check a comparison rather than a guess.

The same applies to the tool schemas a function-calling request carries. Validate them against the model’s expected schema at build time or at service start, not per request, so a malformed tool definition fails the deploy instead of failing two percent of production traffic. Check inference parameters against what the target model supports, which matters most when the model identifier is configurable and a switch to a different provider’s model quietly invalidates a parameter that worked yesterday.

Response analysis after it

The failures that never raise an exception are the ones most likely to be misread. A successful response carries a stop reason, and max_tokens means the model ran out of output budget and stopped mid-sentence rather than finishing. Nothing threw. The HTTP status was 200. The answer is wrong anyway, and truncation-related error analysis starts here: log the stop reason on every call, alarm when the max_tokens share of responses rises, and treat a truncated answer as a failure class of its own rather than as a quality complaint.

Two more response shapes deserve the same treatment. An empty or missing content block on a 200 is a real outcome, usually from a model that produced only a tool-use block or was cut off before generating text, and code that assumes text is present will throw somewhere far from the cause. And a guardrail intervention returns a stop reason of its own: a blocked response is the guardrail working, not the model failing, and it should be counted separately from errors so that a policy change does not look like a reliability regression.

The retry policy itself

With the taxonomy in hand, the policy is short. Never retry ValidationException, AccessDeniedException or ResourceNotFoundException; surface them with the detail message intact. Never retry ServiceQuotaExceededException; shed or defer the work and raise the quota. Retry ThrottlingException, InternalServerException, ServiceUnavailableException, ModelTimeoutException and ModelErrorException with exponential backoff and jitter, and prefer the SDK’s own retry mode over a hand-rolled loop, since it implements the backoff, the jitter and the caps correctly. Give ModelNotReadyException a longer horizon. Cap the total wait so an interactive request degrades rather than hangs, and let the batch path use a much larger budget because nobody is watching it. Handle ModelStreamErrorException in the client that owns the stream, where the partial output actually is.

Worked example

Come back to the class of questions that fails every time.

The new logging shows it in one query. Ninety-one percent of the daily failures are ThrottlingException clustered in the afternoon peak, which backoff already absorbs and which the users never see. Six percent are ModelTimeoutException, concentrated on the longest inputs. Three percent, roughly two hundred calls a day, are ValidationException with the detail input length exceeds the model context window, and every one of them carries an input token count over two hundred thousand.

Those are the ones taking eleven seconds. The retry wrapper was sending a request three times that the service rejected on sight, and the user was paying the latency of three rejections to receive the same generic apology. Once the wrapper stops retrying them, they fail in 140 milliseconds and the log names the cause.

The cause turns out to be a single subscriber with four years of order history, whose conversation context assembles a prompt of well over two hundred thousand tokens. Adding a token count before the call and trimming the history to the most recent exchanges fixes it in a deploy, and the failure class disappears.

The interesting part is what the fix would have been without the taxonomy. Before the exception names were logged, the failure looked like a quality problem: long, complicated accounts got a useless reply, and the working theory was that the assistant could not cope with complex histories. A sprint of prompt engineering was already scheduled. The same misreading has a sibling in the responses that succeed. A stop reason of max_tokens shows an answer that stopped because the output budget ran out. Read as poor quality, it sends a team to rewrite prompts when the fix is a larger output budget or a shorter requested answer. Both are the same mistake, which is treating a mechanical limit as a model failing to understand.

What’s worth remembering

1. The retry decision belongs to the error, not to the call site, so a wrapper that retries every failure identically is wrong about roughly half of Bedrock’s exception taxonomy.
2. ValidationException, AccessDeniedException and ResourceNotFoundException never change on a second attempt; retrying them buys three round trips of latency and delays the diagnosis.
3. ThrottlingException wants exponential backoff and jitter, while ServiceQuotaExceededException is an account ceiling that retrying pushes against and cannot move.
4. ModelStreamErrorException arrives after tokens have reached the user, so the client that owns the stream has to decide up front between discarding the partial answer and keeping it with a truncation notice.
5. Log the exception name, request ID, model identifier and token counts on every failure; model call failed supports no decision anybody needs to make.
6. A stop reason of max_tokens is truncation, not a quality problem, and reading it as one sends teams to rewrite prompts when the fix is an output budget.