The situation
A finance team runs an agent overnight to reconcile supplier invoices. For each invoice it fetches the PDF from S3, extracts the totals, looks up the matching purchase order in DynamoDB, calls the supplier portal’s API to confirm the delivery was received, and writes a reconciliation record. Three hundred invoices a night, unattended, finished before anyone arrives.
Last Tuesday the supplier portal started returning 500s. The agent did what an agent does. It called the tool, read the error, reasoned about the error, decided the call was worth retrying with a slightly different parameter, called it again, read the same error, and reasoned again. Around forty iterations per invoice, on every invoice, all night. Nothing crashed. Nothing alerted. The morning brought a reconciliation table with nothing in it, a portal owner asking why one client had generated two hundred thousand failed requests, and a Bedrock bill for the night that was eleven times the usual.
The security review that followed turned up the other half. The tool that fetches invoice PDFs runs under a role with s3:GetObject on the whole bucket, and the bucket also holds scanned employment contracts. Nobody intended that. The role was written during the prototype, when the bucket held six test invoices, and nobody narrowed it afterwards.
What actually matters
The first thing to settle is where a limit is enforced. The team’s initial fix was a line in the system prompt: after three failed attempts on any tool, stop and report. That is a request. The model is free to conclude that this particular fourth attempt is different because the error message changed slightly, and it will conclude that, because reasoning about whether a rule applies is exactly the behaviour we asked for. A bound that has to hold is evaluated by code that counts, outside the model, in a place it cannot reach or reinterpret. Everything else is guidance. That is useful for shaping ordinary behaviour and worth nothing on the night the dependency breaks.
That question decides the second one, which is who runs the loop. An agent is a model plus tools plus a cycle of reason, act, observe. Either the runtime owns that cycle and the model decides each time whether to go round again, or you own the cycle and a piece of code decides. The two arrangements do not differ much on a good night. They differ completely on what you can promise about a bad one. If the runtime owns the loop, your guarantees are whatever the runtime’s configuration exposes. If you own the loop, the iteration count is a number in your state and the stop test is a branch you wrote.
Then there is the shape of the limit, and one is never enough. This run was cheap per iteration and endless; a different failure is short and ruinously expensive; a third finishes fast and quietly reads a bucket it shouldn’t. Iteration count, wall-clock time, and money spent are three independent ways for a run to go wrong, and a cap on one says nothing about the other two. Reach for all three. Put the clock at more than one layer too: a single tool call hanging for fifteen minutes and a whole run grinding for six hours are different failures with different remedies.
The last thing is what happens at the limit, which gets neglected because it only matters after something has already gone wrong. A run that trips a cap and returns nothing is nearly as bad as one that never stops; the operator gets a blank table and no idea why. Returning what was reconciled, plus the reason the run ended and the invoice it was on, turns a silent failure into a five-minute diagnosis. And none of this touches blast radius: a loop that stops after five iterations still reads every object those five calls were permitted to read. Bounding the loop and bounding what the loop can reach are separate jobs, and the second one is IAM’s.
What we’ll filter on
- Is the bound enforced outside the model, by code that counts, or is it an instruction the model may reason its way past?
- Does the arrangement cap all three of iterations, wall-clock time, and spend, and at more than one layer for time?
- Where does run state live, and can a second concurrent worker see that a dependency has already been found broken?
- What does the caller receive when a bound trips: a partial result with a stated reason, or silence?
- What can a single tool call reach if the model is coaxed into making it, and who is able to widen that later?
The landscape
The loop inside the agent runtime
The default arrangement, and the one already in place here, puts the cycle inside the runtime. Bedrock AgentCore hosts the agent, the model plans, the runtime dispatches the tool call, feeds the observation back, and asks the model what to do next. The stopping conditions available are the ones the runtime exposes: a maximum number of turns, a session or invocation timeout, and whatever your agent framework’s own loop configuration offers.
This is genuinely good at the thing it exists for. The path is discovered at run time, so an invoice with a missing purchase order gets handled by the model choosing a different tool rather than by a branch somebody had to anticipate. There is very little to build and nothing to operate. The limits are real limits, enforced by the runtime rather than by the model’s goodwill, so the promise that a run stops after N turns holds.
What you do not get is a per-step grip. The runtime’s turn cap is one number covering the whole session. There is no natural place to hang a per-tool circuit breaker that other concurrent runs can see, no state you can inspect halfway through, and no obvious hook for “before iteration seven, check the spend so far”. The history of a run is a trace to read afterwards rather than a data structure to test against during.
The loop as a state machine
The other arrangement is to write the loop yourself. Using Step Functions to implement ReAct patterns means the reason-act-observe cycle becomes states you can see. A Lambda function asks the model what to do next and returns a decision. A second Lambda executes the chosen tool. A Choice state evaluates the stop test. An iteration counter lives in the execution’s state and increments each time round.
The model still reasons: chain-of-thought reasoning approaches sit inside the reasoning Lambda, where the prompt asks for the working before the decision. The model’s structured reasoning steps come back as a small object the state machine can branch on rather than as free text the runtime interprets. What changes is that the sequence is data. Iteration seven is a number in the execution state, the stop test is a Choice state comparing it against a maximum, and both are visible in the execution history state by state.
The price is honest. You are building and maintaining a state machine, writing the prompt that returns a parseable decision, and handling the case where the model returns something the Choice state cannot read. On a workload where the path is genuinely unknowable and nothing runs unattended, that effort goes nowhere useful.
The brakes, and where each one can live
Four mechanisms do the actual stopping, and they are not alternatives to each other.
Stopping conditions cap the number of times round. In the runtime this is a configured maximum on turns; as a state machine it is an integer in the state and a Choice state that tests it. Timeouts cap elapsed time, and Lambda functions to implement timeout mechanisms are the layer people forget, because a function’s own configured timeout is a bound on the tool call that the orchestrator does not have to be awake to apply. IAM policies to enforce resource boundaries cap reach, and they are the only one of the four that does anything about the bucket. Finally, circuit breakers to mitigate failures cap repetition against a dependency that has already proved itself broken. That one needs state outliving a single iteration and, ideally, a single run.
The first two exist in both arrangements. The third is independent of both. The fourth wants somewhere to keep breaker state, which is a DynamoDB item when the breaker should be shared across concurrent runs and a field in the execution state when per-run is enough.
Evaluation
Side by side
| Property | Runtime-owned loop | State-machine loop |
|---|---|---|
| Bound enforced outside the model | ✓ | ✓ |
| Hard iteration cap you can read mid-run | ✗ (configured, not inspectable) | ✓ |
| Per-tool-call timeout | ✓ (Lambda’s own) | ✓ (Lambda’s own) |
| Whole-run timeout | ✓ (session timeout) | ✓ (execution timeout) |
| Spend ceiling tested between iterations | ✗ | ✓ |
| Circuit-breaker state shared across runs | ✗ (needs bolting on) | ✓ (DynamoDB or state) |
| Per-tool IAM boundary | ✓ | ✓ |
| Partial result with a stated reason on trip | ✗ (whatever the runtime returns) | ✓ (a Fail or Succeed state you wrote) |
| Path discovered at run time | ✓ | ✓ (one step at a time) |
| Effort to build and maintain | Low | High |
The rows that separate the two are the ones about reading and testing state part-way through a run. Both arrangements stop; only one of them lets you decide, at iteration seven, that this run has already spent its budget or that this dependency is already known to be down. That capability costs a state machine’s worth of work, and it is worth paying for on an unattended overnight batch against a third-party API, and not worth paying for on an interactive assistant a human is watching.
Two rows are the same in both columns and deserve saying out loud. Per-tool IAM scoping is orthogonal to the loop question: the invoice-fetch role reads the whole bucket in either arrangement until somebody narrows it. And the Lambda timeout applies wherever the tool runs, because the function’s configured limit is enforced by the platform.
The solution
Run the loop as a Step Functions state machine, and fit five brakes. The finance batch is unattended, runs against a dependency outside the team’s control, and spends real money per iteration, which is the combination that pays for the state machine. The name for this shape is safeguarded AI workflows, and the property they deliver is controlled FM behavior: the model still reasons freely about what to do next, and the run still ends.
Stopping conditions and a maximum iteration cap
Use Step Functions to implement stopping conditions: the execution state carries iteration, the tool-executing state increments it, and a Choice state tests it against a maximum before handing control back to the reasoning step. Set the cap from what the work actually needs, not from what feels generous. This job reconciles one invoice in three or four tool calls on a normal night, so a cap of ten leaves room for a genuinely awkward invoice and stops the forty-iteration spiral cold.
The stop test has more than one arm. It ends the loop when the model signals it is finished, when the iteration count hits the maximum, when the run’s spend crosses its ceiling, or when the breaker for a required tool is open. Each arm routes to a terminal state that says which one fired.
Then decide what comes back. A run that stops at the cap returns the invoices it did reconcile, the invoice it was working on, the iteration count, and the reason it stopped. A partial answer with a reason is a diagnosis; an empty table is a mystery.
Timeouts at three layers
Time gets bounded three times, because the three failures are different. Each tool Lambda’s own timeout bounds one call, and it should be set close to the real work rather than left at whatever the framework defaulted to. A supplier API call that normally answers in two seconds gets a ten-second function timeout, not fifteen minutes. The Task state that invokes it carries a TimeoutSeconds slightly above that, so a lost response does not leave the state machine waiting on a function that has already gone. And the execution itself carries a whole-run timeout, so a batch that has been grinding for six hours ends whether or not any individual step misbehaved.
Lambda functions to implement timeout mechanisms have a second use worth taking. A function that watches its own remaining execution time can return a clean “I ran out of time on the portal call” result instead of being killed mid-flight, which gives the state machine something to branch on rather than an opaque failure.
Resource boundaries in IAM
Give every tool its own execution role. The invoice-fetch function gets s3:GetObject on the prefix that holds invoices, not the bucket. The purchase-order lookup gets read on one table. The reconciliation writer gets write on one table and nothing else. This is what IAM policies to enforce resource boundaries means in practice, and the reason it matters is that a coaxed tool call can only reach what that one tool was permitted to reach, which is the same argument that shapes the tool schemas themselves.
Then stop the roles from drifting back. Attach a permissions boundary to the roles the finance team can create and edit, capping the maximum permissions any of those roles can hold regardless of what policy someone attaches later. Self-service role creation without a boundary is how a prototype’s wildcard survives into production; the boundary makes the narrow version the ceiling rather than the starting point.
A circuit breaker on the failing dependency
Wrap the supplier portal in a breaker. Consecutive failures increment a counter; when the counter crosses a threshold the breaker opens, and while it is open the tool state returns “dependency unavailable” immediately without making the call. After a cool-off the next attempt goes through as a trial, and one success closes the breaker again.
Where the state lives follows from who needs to see it. Three hundred invoices processed by concurrent executions want the counter in a DynamoDB item keyed on the tool name, so the twelfth invoice does not have to rediscover what the first eleven already learned. A single-run breaker can live in the execution state, which is simpler and blind to everything happening in parallel. The finance batch wants the shared version, and the difference on Tuesday night would have been two hundred thousand failed requests becoming a few dozen.
A token and cost ceiling per run
The fifth brake is money, and it is the one the other four do not cover. Have the reasoning step return its token usage, accumulate input and output tokens in the execution state, and add an arm to the stop test that ends the run when the accumulated spend crosses a per-run ceiling. Emit the running total as a metric so the ceiling is visible before it is hit, which is the per-run half of the wider cost guardrails on the workload.
A cap of ten iterations already bounds spend loosely. An explicit ceiling bounds it when a single iteration turns out to be far more expensive than the ones that set your expectations, which is what happens the first time an invoice arrives with sixty pages of attachments.
Where each brake fires
Worked example
Replay Tuesday night with the brakes fitted.
Invoice one. The reasoning step returns a decision to call the portal. The breaker item says closed, so the tool state runs. The portal returns a 500 after 900 milliseconds, well inside the function’s ten-second timeout, and the observe state increments the failure counter to one. The Choice state sees iteration two of ten, spend well under the ceiling, breaker closed, and goes round. Three more failures and the counter reaches five, which opens the breaker with a sixty-second cool-off. The stop test’s breaker arm fires on the next pass. The execution ends with a partial result: no reconciliation, tool unavailable, four attempts, failing dependency named.
Invoices two through three hundred. Every execution reads the same DynamoDB item, sees an open breaker, and ends at iteration one without calling the portal at all. The cool-off lets one trial call through every minute, each of which fails and reopens the breaker. The night’s traffic to the supplier is a few hundred requests rather than two hundred thousand, and the token spend is one reasoning call per invoice rather than forty.
The morning. The batch has finished, which it did not do before. The reconciliation table has three hundred rows, each recording that the invoice was not reconciled and why. A CloudWatch metric on breaker openings has been non-zero since 01:14, and the trace for any one execution shows four tool calls and their errors rather than forty rounds of reasoning about the same 500. Diagnosis takes a couple of minutes instead of a day.
And the contract scans. The invoice-fetch role now carries s3:GetObject on the invoices prefix, under a permissions boundary that caps what any role the finance team creates can hold. That change did nothing for Tuesday night’s loop, and it is the one that matters most on the night somebody works out how to get an instruction into an invoice PDF.
What’s worth remembering
- A limit written into the prompt is guidance the model may reason past; a limit that has to hold is evaluated by code outside the model, which is why the loop’s owner decides what you can promise.
- Owning the loop as a state machine gives you an iteration counter, a spend total, and a breaker check you can test between turns, at the cost of building and maintaining the state machine.
- Cap iterations, wall-clock time, and money separately, and put the clock at three layers: the tool function, the Task state that invokes it, and the whole execution.
- Scope one IAM role per tool and cap those roles with a permissions boundary; bounding the loop does nothing about what a single permitted tool call can reach.
- Hold circuit-breaker state where every concurrent worker can see it, so the twelfth run does not have to rediscover the outage the first eleven already found.
- When a bound trips, return what was completed plus the reason it ended, because a partial answer is a diagnosis and silence is a mystery.