How to Build a Multi-Modal Bedrock Assistant for Insurance Claims

The situation

An insurance company is modernising the first-line claims workflow. A customer submits a claim through any combination of channels: a photo of a damaged laptop, a PDF of the purchase invoice, a voicemail explaining what happened, and a follow-up text message asking when the decision will be made. Today, those artefacts land in separate queues and separate humans stitch them together. The target is a single assistant that accepts any subset of these inputs, understands them, asks clarifying questions where needed, and either resolves the claim or routes it to a human with a clean summary and a recommendation.

Concretely, the assistant needs to:

• Read images. Photos of damaged goods, whiteboard notes from adjusters, screenshots of error messages, identity documents.
• Read PDFs and scanned documents. Invoices, receipts, policy documents, medical notes, a mix of text-over-image and structured PDF.
• Transcribe and understand audio. Voicemails up to three minutes, often with background noise and accents.
• Produce text. Customer-facing explanations, internal summaries, structured decisions for the claims system.
• Optionally produce speech. Accessibility mode reads responses back; some channels (IVR) are audio-only.
• Keep a single conversation. Across modalities, across turns, without losing context.

Five constraints matter: claim acknowledgement within 30 seconds, first substantive response within 2 minutes, decision or routing within 10 minutes, accessibility for audio-first users, and an audit trail for every AI-produced decision.

What actually matters

The phrase “multi-modal” collapses several distinct capabilities that a real system has to handle separately. Understanding an image is not the same as understanding audio, and neither is the same as generating speech. The models that are good at each are different; the failure modes are different; the latency and cost profiles are different.

The first decision is which input modalities go through a single multi-modal model, and which get transcoded to text first. A vision-capable LLMLLMA neural network trained to predict the next token in a sequence, large enough that it generalises to tasks it wasn’t explicitly trained for. takes images directly and answers questions about them in the same promptPromptThe input you hand to an LLM – system instructions, user message, examples, retrieved documents, tool descriptions, the lot. as text. Audio usually doesn’t work that way. The Converse API defines an audio content block, but the model cards for the text-and-vision models on Bedrock list audio input as unsupported, so audio understanding is a separate service that produces text for the LLM. PDFs sit in between. Converse takes a document block with pdf among its formats, so a native PDF needs no pre-processing at all; a scan with no text layer still needs OCR or page images.

The second is the orchestration shape. One prompt with several input blocks, text, image, text, is the simplest case. Several steps (transcribe audio → extract PDF text → combine → send to LLM) is the common case. A stateful agent loop, where each model response selects the next tool, is the most flexible case. Each shape has different latency characteristics.

The third is output modality. Generating text is native to every LLM. Generating speech is a separate service call. Generating images is a separate service call. Whether to bundle these into the model’s response or chain them as a post-step changes what the user experiences.

The fourth is failure modes, per modality. An image might be blurry; an audio file might be inaudible; a PDF might be password-protected; a voicemail might be in a language Transcribe doesn’t cover. Each needs a graceful fallback, “I can’t quite make out the invoice; could you describe the damaged item?”, instead of a hard error.

The fifth is cost shape across modalities. An image consumes input tokensTokenThe unit of text an LLM actually sees – usually a short character sequence, not a whole word. in proportion to its resolution. Transcribe bills batch transcription at USD$0.006 a minute and streaming at USD$0.01 a minute. Polly’s neural voices bill at USD$16.00 per million characters, which puts a minute of synthesised speech in the same range as a minute of transcription. The bill shape depends on which modalities dominate usage.

User expectations differ by channel, too. An accessibility user reading via screen reader expects a different response shape from a claims adjuster reviewing a summary, not a different model, but a different prompt and response length.

What we’ll filter on

1. Modality coverage, which inputs and outputs does this architecture natively support?
2. Latency, first-response and full-response timing for each input shape?
3. Robustness, graceful handling of bad-quality inputs?
4. Operational surface, how many services, SDKs, tools to integrate?
5. Per-modality cost, does the cost model make sense for the expected input mix?

The landscape

Claude Sonnet 5 (vision) + Amazon Transcribe + Amazon Polly. Claude takes text, images, and documents in a single prompt. Transcribe handles audio in, Polly handles speech out. Orchestration is explicit code: a Lambda that receives the request, dispatches to Transcribe if audio, sends everything to Claude, optionally sends Claude’s response to Polly. Claude Sonnet 5 carries a 1M-token context window and a 128K maximum output, and supports tool use, Guardrails and streaming on the Converse API. Transcribe covers over a hundred languages for batch (fewer for streaming), with speaker partitioning and custom vocabularies. Polly offers standard, neural, long-form and generative voices.

Amazon Nova family. Nova Lite and Nova Pro handle text, image, and video input natively through Bedrock, both with a 300K-token context window. Nova Canvas and Nova Reel covered image and video generation and have since been marked legacy (both reach end of life on 30 September 2026), which this assistant never needed anyway, since it reads media rather than making it. Nova Micro is text-only. Audio input is handled by pre-processing through Transcribe. Same orchestration pattern; different vendor.

Amazon Bedrock Data Automation. The one AWS service built for this input mix: documents, images, audio and video through a single API, returning structured output with confidence scores and visual grounding. What comes back is extraction rather than conversation, so it suits the ingestion leg and leaves the multi-turn claims dialogue to a model behind it.

Amazon Quick. The higher-level managed product, and what Amazon QuickSight became. Quick Index grounds answers in an organisation’s own documents, Quick Research returns cited reports, and connectors reach enterprise systems over MCP and OpenAPI. Licensing is a per-user subscription rather than per token. For a well-scoped enterprise-documents case it removes a lot of the plumbing, and it gives less control over the underlying model and less room for claims-specific workflows that mix images, audio, and structured decisions.

SageMaker-hosted multi-modal models. Custom or open-source multi-modal models. LLaVA, Kosmos, Idefics, hosted on SageMaker endpoints. Full control, higher operational cost, justified when the commercial models don’t fit (specialised domains, privacy requirements, custom fine-tuning). Not the default.

An “everything through OCR to text” approach. Run every input through a text transcription (Textract for documents, Transcribe for audio, a vision-to-description step for images), concatenate the text, feed to a text-only model. Simple, and it discards information, since an image described in words drops visual detail the model could have used directly. Cheaper in some cases, wrong when the visual detail matters.

Evaluation

Side by side

Option	Modality coverage	Latency	Robustness	Ops surface	Cost shape
Claude + Transcribe + Polly	Text, image, PDF, audio, speech	Moderate	Per-service fallbacks	3 services + glue	Per-token + per-minute
Nova + Transcribe + Polly	Text, image, video, audio (via), speech	Moderate	Per-service fallbacks	3 services + glue	Per-token + per-minute
Bedrock Data Automation	Document, image, audio, video in	Job-shaped	Confidence scores built in	Minimal	Per unit processed
Amazon Quick	Documents + text	Low	Managed	Minimal	Per-user subscription
SageMaker hosted	Anything we deploy	Variable	Ours	High	Endpoint-hours
Everything-to-text	Text only (post-transcription)	Low	Lossy conversion	Moderate	Cheapest per input

For a claims assistant with four distinct modalities and a need for high visual fidelity (a photo of a damaged laptop carries information that a description drops), build on Claude Sonnet 5 with Transcribe and Polly. Claude covers text, images and documents, Transcribe the audio leg, Polly the speech out. The orchestration is ours to own, but it’s manageable, one Lambda with clean branches per modality.

The orchestration, in shape

The solution

Image and PDF inputs go direct to Claude. A Converse message carries image blocks and document blocks side by side, each as inline bytes or an S3 URI, and a document block has to travel with a text block that prompts against it. A native invoice PDF goes in as a pdf document block; a scan with no text layer splits into page-image blocks instead, because there is no text for the document block to extract; a natural photo of damage goes in as-is. Long documents are where the split-and-embed route through a Knowledge Base becomes the better fit, but a five-page invoice is fine inline.

Audio goes through Transcribe first. The text-and-vision models on Bedrock list audio input as unsupported (Nova 2 Sonic does speech-to-speech for live conversation, and its predecessor Nova Sonic is legacy with an end of life of 14 September 2026, so don’t build on it). Transcribe handles the audio to text conversion, with features that matter for voicemail: speaker partitioning when there are several voices, custom vocabulary for brand names and policy jargon, and a confidence score on every word. Low-confidence stretches get flagged in the prompt, “[transcript, confidence 0.4: mumbling about a laptop]”, so the reliability of the text sits in the prompt beside it. Batch transcription runs as a job against a file in S3 and is polled to completion; Transcribe Streaming returns partial results as the audio arrives, which suits a live channel.

Text messages pass through. No pre-processing needed; text block as-is.

Orchestration as a Lambda. The Lambda receives the claim package (some combination of S3 keys for image/PDF/audio, plus any inline text), dispatches to Transcribe for audio, renders scanned PDFs to page images, assembles a Converse request with all blocks in a stable order (text first, then images, then document pages, then transcribed audio as text with confidence annotations), adds the system promptSystem promptThe instruction block that frames the model’s behaviour for a session, separate from the user’s messages. and session history from DynamoDB, and calls Bedrock.

Output. The model returns text, optionally with tool calls to the claims system (lookup policy by ID, record a provisional decision, route to human). Text goes to the customer’s channel. If accessibility mode is enabled (session attribute on the conversation), the text also routes through Polly. Check SSML tags against the engine before generating them: <break time="500ms"/> works on every voice, but <emphasis> is unavailable on the neural, long-form and generative engines, and Polly returns an error on an unsupported tag rather than ignoring it. Constrain the model’s markup to the tags the chosen engine accepts.

State and audit. DynamoDB holds conversation history keyed by session ID; each turn records the input modalities, the prompt version (from Prompt Management), the model’s text output, and any tool calls made. S3 holds the raw inputs with server-side encryption, evidence the claims system can reference later. CloudWatch records each decision with the session ID, prompt version, and a flag indicating human oversight was required.

Worked example

A customer submits a claim with a photo of a laptop with a cracked screen, a scanned PDF of the original purchase invoice (2 pages), a 45-second voicemail explaining they dropped it, and a text message asking “can this be fixed under my policy?”

1. Lambda receives the package. Starts a Transcribe batch job on the voicemail; renders the scanned PDF to two page images; resizes the photo.
2. Transcribe job completes: “Hi, it’s Sarah, I dropped my MacBook yesterday and the screen is cracked, but it still turns on. Is this covered?” Word confidences 0.95 and above throughout.
3. Converse request assembled: system prompt (claims assistant, empathetic, always name the policy clause), session history (empty, new conversation), user message with blocks: text of text message; image of damage photo; image of invoice page 1; image of invoice page 2; text of transcript.
4. Claude’s response covers accidental damage coverage and makes a tool call, lookupPolicy(policyId from session attributes).
5. Lambda dispatches the tool call, gets the policy, feeds result back.
6. Claude’s next turn: “Hi Sarah, I can see the cracked screen in your photo and I have your invoice from 2025-03-15. Under clause 4.2 of your policy, accidental damage is covered up to AUD$1,500 with a AUD$100 excess. Based on the damage shown, a screen replacement typically costs AUD$400-600. I’ve flagged this for an adjuster to confirm the quote; you should hear back within 48 hours.”
7. Session attribute accessibility: true. Lambda routes the response to Polly (neural voice, en-AU), returns MP3 URL + text.
8. CloudWatch records: session ID, prompt version claims-assistant:v42, input modalities [image, pdf, audio, text], tool calls [lookupPolicy], decision route_to_adjuster, and the measured end-to-end latency against the 2-minute target.

What’s worth remembering

1. “Multi-modal” is a marketing word; real systems route per modality. Images and documents go direct to a vision LLM; audio goes through Transcribe first; speech out goes through Polly. One prompt, several pre-processing paths.
2. Converse carries text, image and document blocks in one user message, with pdf among the document formats. A scanned page with no text layer is the case that still needs rendering to images.
3. Audio in requires Transcribe. The text-and-vision models list audio input as unsupported, and Nova 2 Sonic covers live speech-to-speech as a separate job. Include Transcribe’s per-word confidence in the prompt so the model’s answer can hedge where the transcript is weak.
4. Orchestration is a straight-line Lambda, not an agent. Four pre-processing branches, one Converse call, optional TTS. Deterministic flow; debuggable; no agent loop required unless tools come into play.
5. Robustness is per-modality fallbacks. Low-quality image? Ask for another. Low-confidence transcript? Summarise what was heard and ask for confirmation. Unreadable PDF? Fall back to a text description prompt.
6. Polly’s engines differ in which SSML tags they accept, and an unsupported tag returns an error. <emphasis> is unavailable on neural, long-form and generative voices.