Event-Driven GenAI: Processing Documents Asynchronously

The situation

A back-office team has built a document-processing feature on Amazon Bedrock. Users upload PDFs, contracts, research reports, scanned forms, into an Amazon S3 bucket, and each one needs a generative pass: a structured summary, a set of extracted fields, and a risk classification. A single document runs anywhere from twenty seconds to four minutes through the model, depending on length, and some of the reports are two hundred pages.

The first version put the whole thing behind an API. A user uploaded through a web form, the request called Bedrock inline, and the browser waited. It worked in the demo with a two-page sample. In production it fell over immediately: the API Gateway integration timed out at twenty-nine seconds, the long documents never returned, and the front-end retries re-ran the model call from scratch, so a report that eventually succeeded had been summarised three or four times and billed for every attempt. When a marketing push sent four hundred uploads in an hour, the synchronous path had no way to shed load, and half the requests errored out under Bedrock throttling.

The team now wants the uploads to just work, whatever the volume, without a human watching a progress bar. The document is expensive to process and slow to process, and the user does not need the answer in the same breath as the upload. That combination is the whole design brief.

What actually matters

The deciding property is latency tolerance. A generative pass over a long document is a batch job that happens to be triggered by a person, not an interactive request. Nobody is staring at the screen for four minutes, and even if they were, no sensible HTTP path stays open that long. Once you accept that the answer can arrive seconds or minutes after the upload, the synchronous request stops being a constraint and the whole shape of the system changes: the upload becomes an event, the work becomes a queued task, and the result gets written somewhere the user checks later or gets notified about. Fighting to keep it synchronous is the actual mistake.

The second property is decoupling under bursty load. Uploads do not arrive smoothly; they arrive in clumps, and Bedrock has account-level throughput quotas that a clump will blow straight through. Something has to sit between the flood of uploads and the model and meter the work out at a rate the model will accept. A buffer that holds pending work and lets workers pull from it at their own pace turns a spike that would have thrown throttling errors into a queue that just drains a little slower. Without that buffer, the burst is the user’s problem; with it, the burst is invisible.

The third is failure handling, and it matters more here than in a cheap CRUD system because every retry costs real money. Model calls fail transiently, time out, or hit throttling, and the naive answer, retry, is exactly what tripled the bill in version one. Retries have to be bounded, they have to back off, and repeated failures have to land somewhere you can inspect rather than looping forever or vanishing. That means a dead-letter path for the documents that never succeed, and it means the worker has to be safe to run twice on the same document without producing two charges’ worth of duplicate output, because at-least-once delivery guarantees you will occasionally process the same thing twice.

The fourth is orchestration complexity, which decides how heavy the machinery needs to be. A single summarise-and-store step is one worker. A pipeline, extract text, then summarise, then classify, then write to a database, then notify, with different retry rules at each stage and a branch for documents that fail validation, is a workflow, and trying to cram a multi-stage workflow with per-stage error handling into one function is where worker code turns into an unmaintainable knot. The more stages and the more the stages need independent retries and visible state, the more the orchestration wants to be explicit rather than buried in code.

And the cross-cutting one: sometimes there is no event at all, just a pile. When the job is ten thousand documents sitting in a bucket with no deadline, standing up a queue and workers to trickle them through is more machinery than the problem needs. Bedrock can take a single large asynchronous job, read every record from S3, run them, and write the results back to S3, at roughly half the on-demand price. Reaching for the event-driven plumbing when a batch job would do is its own kind of over-engineering.

What we’ll filter on

1. Latency tolerance, does anyone wait on the answer, or can it arrive minutes later?
2. Volume and burstiness, steady trickle, spiky bursts, or a one-off pile of thousands?
3. Orchestration complexity, one step, or a multi-stage pipeline with branches and per-stage retries?
4. Failure handling, are retries bounded, backed off, and are dead letters captured?
5. Idempotency, is a worker safe to run twice on the same document without double-charging?
6. Cost shape, is the discount of a single async batch job worth giving up per-document immediacy?

The architecture landscape

Synchronous request to Bedrock. The browser or caller waits for the model inline, through API Gateway and Lambda or straight from a server. Fine for short, interactive prompts where the answer comes back in a second or two. For long documents it is the anti-pattern that started this: integration timeouts, client retries that re-run expensive calls, and no way to absorb a burst. Rule it out the moment the work outlasts a comfortable request.

S3 event notifications as the trigger. Configure the bucket to emit an event when an object lands under a prefix, and route it onward. This is the front door for the whole pattern: the upload itself becomes the signal, so there is no polling and no separate submit call. S3 can notify Lambda, SQS, or SNS directly, or fan events through Amazon EventBridge when you want richer routing and filtering. The event carries the bucket and key, not the document, so the worker fetches the object when it runs.

Amazon SQS as the buffer. A standard queue sits between the trigger and the workers, holding pending documents so producers and consumers run at their own pace. This is what tames bursty load and what meters work against Bedrock quotas: workers pull a message, process it, and delete it, and if they are already saturated the queue simply grows and drains later. The visibility timeout hides a message while a worker holds it, and for slow model calls that timeout has to be set longer than the worst-case processing time, or SQS will decide the worker died and hand the same document to a second worker while the first is still running. A dead-letter queue attached to the main queue catches messages that fail past a set number of receives, so a poison document lands somewhere inspectable instead of cycling forever.

AWS Lambda as the worker. A function consumes from the queue, fetches the object from S3, calls Bedrock, and writes the result. It scales out with the queue depth and costs nothing when idle, which fits spiky document traffic well. The constraint to respect is the fifteen-minute maximum execution time: comfortable for most single-document calls, but a genuinely huge job or a long multi-model chain can bump against it, which is a signal to split the work across steps rather than do it all in one invocation. Reserved or provisioned concurrency is how you cap Lambda’s fan-out so it does not scale straight past your Bedrock throughput quota and start throwing throttling errors.

AWS Step Functions for orchestration. A state machine coordinates a multi-step pipeline: extract, summarise, classify, persist, notify, with each state carrying its own retry policy, catch rules, and back-off, and branching for documents that fail a validation gate. The state of every in-flight document is visible and durable rather than implicit in a tangle of function code, and the built-in retry and error handling means you write far less of the plumbing by hand. This is the right weight when the pipeline has several stages that each need independent failure handling; it is overkill for a single summarise-and-store step, where a lone Lambda off the queue is simpler.

Amazon Bedrock batch inference. A single asynchronous job that reads many records from an S3 input location, runs them through a model, and writes the outputs back to S3, priced at roughly fifty per cent of on-demand. There is no queue to run and no worker fleet to manage; you submit the job and collect the results when it completes. This is the fit for high-volume, latency-tolerant work: a nightly enrichment of a whole table, a one-off pass over an archive. It is the wrong tool when documents arrive one at a time and each needs a timely answer, because a batch job is a bulk operation with its own scheduling latency, not a per-upload response.

Side by side

Reading the table against the team’s feature: the upload fires an S3 event, SQS buffers the burst with a DLQ behind it and a visibility timeout tuned to the four-minute worst case, and the choice between a lone Lambda and a Step Functions pipeline comes down to whether the extract-summarise-classify-persist-notify chain wants independent per-stage retries, which it does. The nightly enrichment of the back catalogue is the one piece that wants batch inference instead, because it is a pile with no deadline.

The picks in depth

For the live upload path, the spine is S3 notification into SQS into Lambda, and the details that make or break it are the queue settings. The visibility timeout has to exceed the worst-case processing time plus a margin, so with documents that can take four minutes, a timeout of six or so keeps SQS from re-delivering a message to a second worker while the first is still grinding through a two-hundred-page report. Set it too short and you get duplicate processing under load, which on Bedrock means paying twice. The maxReceiveCount on the redrive policy bounds how many times a failing message is retried before it moves to the dead-letter queue, so a document that reliably crashes the worker, a corrupt PDF, an unsupported format, lands in the DLQ after a few attempts instead of blocking the queue or looping forever. Nothing gets lost, and you get a bucket of poison documents to look at rather than a silent gap.

Idempotency is the piece people skip and regret. SQS is at-least-once, so the same document will occasionally be delivered twice, and every retry, whether from a visibility-timeout mis-set, a redrive, or a transient error, is a chance to run the expensive model call again. The defence is a deterministic result key, typically derived from the object key and version or a content hash, and a check-before-work step: if the result for this document already exists in the output store, the worker skips the model call and returns. That turns duplicate deliveries into cheap no-ops instead of duplicate charges, and it makes the whole pipeline safe to retry aggressively, which is what lets you set generous retry policies without fear.

Concurrency against Bedrock quotas is the other tuning knob. Lambda will happily scale to hundreds of concurrent workers when the queue is deep, and Bedrock will throttle every call past your account’s per-model throughput limit, so an uncapped worker fleet converts a queue backlog into a wall of throttling errors. Reserved concurrency on the worker function caps the fan-out to a number the model quota can sustain; the queue absorbs the rest and drains at that steady rate. Pair the cap with retry-on-throttle and a short back-off so the occasional throttled call recovers on its own rather than falling through to the DLQ. Requesting a quota increase is the move when the sustained rate genuinely needs to be higher, but the cap is what keeps a burst from self-inflicting failures in the meantime.

Step Functions earns its place once the work is a pipeline rather than a step. Extract text, summarise, classify, write to the database, send the notification, each of those can fail independently and each wants its own retry and catch behaviour, and expressing that as a state machine gives you durable, inspectable state for every document instead of a mega-function that swallows its own errors. A Catch on a state routes a failed document to a handling branch, a Retry block applies bounded exponential back-off per state, and the execution history shows exactly where a given document is or why it stopped. The trade is cost and a little ceremony: you pay per state transition and you maintain the definition, so for a genuine single-step job the state machine is weight you do not need. The honest rule is to start with the Lambda-off-a-queue shape and graduate to Step Functions when the stages and their independent failure handling actually appear.

Batch inference is the pick that removes the plumbing entirely, for the workloads that suit it. When the job is enrich this whole table overnight or summarise this archive of ten thousand filings with no per-item deadline, submitting one asynchronous S3-to-S3 job at half the on-demand price beats building and running a queue and a worker fleet to do the same thing slower and dearer. The cost is immediacy: the job schedules, runs as a bulk operation, and completes on its own timeline, so it is exactly wrong for the one-document-at-a-time interactive-ish upload and exactly right for the standing bulk pass. Many real systems run both, the event-driven path for live uploads and a nightly batch job for the backlog, and the skill is knowing which document belongs on which road.

A worked example: the upload pipeline end to end

A user drops contracts/2026/acme-msa.pdf into the ingest bucket. The flow that follows:

S3 (ObjectCreated on contracts/*)
      -> event notification
SQS ingest-queue           (visibility timeout 360s; DLQ after 3 receives)
      -> Lambda event source mapping (reserved concurrency 20)
Lambda worker
      1. derive result key = sha256(bucket, key, versionId)
      2. if result exists in results bucket -> delete message, return
      3. fetch object from S3
      4. call Bedrock (retry on throttling, short back-off)
      5. write summary + fields + classification to results bucket
      6. delete message   (success ack)
Failure past 3 receives -> ingest-dlq  (inspect corrupt / unsupported docs)

The event source mapping deletes the SQS message only on a clean return, so a worker that crashes at step 4 leaves the message to reappear after the visibility timeout and be retried; three failures send it to the DLQ. The idempotency check at step 2 means a duplicate delivery, or a retry of a call that actually succeeded before the ack, costs a cheap S3 lookup rather than a second Bedrock charge. Reserved concurrency of twenty caps the worker fan-out under the model’s throughput quota, so a four-hundred-upload burst becomes a queue that drains at twenty-wide instead of four hundred simultaneous throttled calls.

When the same team later needs extract, then summarise, then classify, then persist, then notify, each with its own retry rules and a branch for documents that fail validation, the single worker becomes a Step Functions state machine triggered off the same queue, and each stage gets its own Retry and Catch. And the untouched back catalogue of fifty thousand old contracts, with no deadline on it, goes through one Bedrock batch inference job reading from S3 and writing back to S3 at half the price, rather than being poured through the live queue.

What’s worth remembering

1. A long generative pass over a document is a batch job triggered by a person, not an interactive request; once you accept the answer can arrive minutes later, the synchronous constraint disappears and the design gets simpler.
2. Keeping a multi-minute model call behind a synchronous HTTP request buys you integration timeouts and client retries that re-run expensive work and bill for every attempt.
3. Make the upload the event: an S3 ObjectCreated notification triggers the work with no polling and no separate submit call, and it can fan to Lambda, SQS, SNS, or EventBridge.
4. Put SQS between the trigger and the workers to absorb bursty uploads and meter work against Bedrock quotas; the queue turns a spike that would throttle into a backlog that just drains slower.
5. Set the SQS visibility timeout longer than the worst-case processing time, or a slow document gets handed to a second worker while the first is still running, and you pay twice.
6. Attach a dead-letter queue with a bounded maxReceiveCount so a poison document lands somewhere inspectable after a few tries instead of looping forever or vanishing.
7. Make the worker idempotent with a deterministic result key and a check-before-work step, because at-least-once delivery guarantees the occasional duplicate and every retry is a chance to re-charge for the same model call.
8. Cap Lambda worker fan-out with reserved concurrency so a deep queue does not scale straight past your Bedrock throughput quota into a wall of throttling errors; pair the cap with retry-on-throttle and back-off.
9. Reach for Step Functions when the work is a multi-stage pipeline that needs per-stage retries, catches, and visible state; a single summarise-and-store step is just a Lambda off the queue.
10. When the job is a high-volume pile with no per-item deadline, Bedrock batch inference runs one asynchronous S3-to-S3 job at roughly half the on-demand price and saves you building a queue and worker fleet at all.