The situation
A general insurer covers home and motor for around 900,000 policyholders. About 1,200 new claims arrive each day, and 240 people work them through to settlement.
Three features sit on the same quarter’s backlog, and all three arrived described the same way: put a foundation model on it. The first is a claims-summary drafter. A claim file accumulates photos, repairer emails, an assessor’s notes and a scanned report or two, and every time a claim changes hands somebody writes a few hundred words on where it has got to. That happens around 900 times a day and takes about twelve minutes each. The draft sits on the file, and the handler who picks the claim up reads it, then reads the underlying documents when something looks off.
The second is the policy-eligibility decision. Given this policy wording, this excess and this event, is the claim covered. The answer is yes or no, it triggers a letter to the policyholder, and two identical claims that come out differently is a complaint the insurer has no answer to.
The third is a customer chat assistant in the mobile app. Six thousand sessions a day. The questions come in the customer’s own words: “is my bike covered if it was locked up outside”, “how do I add my son to the policy”. Around 40 per cent of them end up with a person. Same model, same API, three very different appetites for being wrong.
What actually matters
Take the advantages first, in the terms that keep coming up around generative AI. Adaptability: one pre-trained model does summarising, classifying, rewriting and answering, and moving it from one job to the next means rewriting a prompt rather than assembling a dataset and training something. Responsiveness: a working drafter exists within days, and at request time an answer comes back in a second or two. Conversational capabilities: the model holds a multi-turn exchange, keeps track of what was said three messages ago, and can ask a clarifying question instead of guessing. And the ability to generate content: what comes out is prose a person can read, rather than a label or a score that something downstream has to turn into a sentence.
All four come out of one mechanism, which is why the limitations arrive with them. The model predicts likely continuations of text against a general distribution learned from an enormous corpus, and it will produce a plausible continuation whether or not a true one is available to it. Hence hallucinations: fluent, confident statements that are simply not so, delivered in exactly the same voice as the true parts of the same paragraph. Hence inaccuracy: the shape of the answer right and a detail wrong, an excess of $750 where the policy schedule says $500. Hence nondeterminism: the same claim file summarised twice, two different summaries, neither of them reproducible on demand. And hence weak interpretability, because you cannot open the answer up and see which inputs moved it and by how much, the way you can with a small model over named columns. Nobody gets the adaptability without the nondeterminism; they are the same property looked at from two sides.
So the fit question is what happens to a wrong answer, not how impressive the right ones look in the demo. Two things settle it. Does a person read the output before anything happens because of it, and is a mistake embarrassing or actionable? A summary that misquotes the repairer is caught by the handler reading it, and costs a minute of rework. A declination letter generated from an exclusion the model invented has already left the building, and is now a complaint and a regulatory matter at once. The drafter and the eligibility decision look like the same feature from a distance, and they sit at opposite ends of that scale.
The tempting fix for the middle case is to turn the temperature down. Low temperature narrows sampling and makes output far more stable, and it does not make the system deterministic. Providers version and retire models on their own schedule, serving-side batching and floating-point arithmetic leave small variations even under greedy decoding, and any prompt built from retrieved documents changes whenever those documents do. Treat a low temperature as reducing variation rather than removing it. Where identical inputs must produce identical outputs, and somebody has to be able to demonstrate that they did, what is being described is a rule and not a prediction; that distinction decides more of these arguments than model quality does. Cost pulls the same way at volume. Per-token pricing grows with traffic, while a rule costs the same at 1,200 decisions a day as at 12,000, and the bill follows tokens in and tokens out rather than the number of features built.
What we’ll filter on
- Tolerance for a wrong answer: is a mistake embarrassing and recoverable, or acted on the moment it is produced?
- Repeatability: must identical inputs give identical outputs, and must someone be able to show that they did?
- Explanation: does anybody have to say why this answer, in terms that can be checked against a source?
- Output shape: is the answer open-ended language, or a label, a number, or a yes and a no?
- Cost and latency at the real volume, not at demo volume.
The landscape
Three routes are genuinely available for features of this shape, and the insurer can use different ones for each of the three.
A generative model on Amazon Bedrock
A large pre-trained foundation model called through an API and told what to do in a prompt. No training data, no training job, nothing to size. It handles all three features on day one at some level of quality, which is what makes the backlog conversation so easy to have and so easy to get wrong. Retrieval over the policy wordings improves the factual grounding, guardrails filter what goes in and comes out, and neither turns a generative answer into a checkable one. The related judgement of which jobs suit a pre-trained model at all is worked through in the split between a model you train and a model somebody else trained.
A purpose-built AWS AI service
Amazon Textract pulls text, form fields and tables out of the scanned reports and repairer invoices with a confidence score attached to each extraction. Amazon Comprehend picks entities, key phrases and sentiment out of the emails, and will classify into your own categories once you have labelled examples. Amazon Lex builds a chatbot around named intents and slots, so “add a driver” follows a scripted dialogue that asks for the three things it needs and then calls a Lambda function. Each of these solves one well-defined problem, returns a confidence number you can threshold on, and behaves the same way tomorrow as it does today. The case for reaching past a general model to one of these is made at length in the argument for the narrow service.
A deterministic rule engine
The policy wording encoded as conditions in code or a versioned decision table, evaluated by a Lambda function against the claim’s structured fields. It does nothing with a photograph and nothing with an email written in prose. What it does is return the same answer for the same inputs every time, name the clause that produced the answer, and cost effectively nothing per evaluation. It is also the only one of the three whose behaviour changes when, and only when, somebody deliberately changes it.
Evaluation
Side by side
| Route | Handles open-ended language | Same input, same output | Reasoning can be checked | Moves to a new task by rewriting a prompt | Cost flat as volume grows |
|---|---|---|---|---|---|
| Generative model on Amazon Bedrock | ✓ | ✗ nondeterministic | ✗ generated rationale only | ✓ | ✗ per token |
| Purpose-built AWS AI service | ✓ within its own task | ✓ pinned model version | ✗ confidence scores, not reasons | ✗ | ✗ per request, but cheap |
| Deterministic rule engine | ✗ | ✓ | ✓ names the clause | ✗ | ✓ |
The first two columns never both say yes. Anything that copes with a claim file written in English is a model, and every model in this table that copes with open-ended language gives up either repeatability or the ability to say why. That is the trade being made, and it is made once per feature rather than once per organisation.
Which feature lands where
The gates are ordered so the cheapest thing to check comes first. Whether a person stands between the output and its effect is a fact about the workflow that anybody in the room can answer, and it removes most of the argument on its own.
The solution
The claims-summary drafter goes to a generative model on Amazon Bedrock, and the reason is the handler, not the model. Every draft is read by the person picking up the claim, with the source documents one click away. A hallucinated repairer quote is caught at the desk, and it costs a minute. Adaptability shows up straight away here: the same prompt pattern, adjusted, also drafts the handover note and the settlement summary without anybody training anything. Design the feature so the model’s habits are visible. Have it quote the file rather than paraphrase where a number is involved. Have it cite which document each claim of fact came from. Leave the draft looking like something a handler is expected to edit, not something finished. Log the edits, because the edit rate is the running measure of whether it is any good.
The policy-eligibility decision goes to a deterministic rule engine, and no amount of prompt work changes that. The insurer must answer why one claim was declined and its twin accepted, and answer it the same way in six months against the same file. Rules give an audit trail, a version history, and a clause number in the declination letter. The model still has a job here, which is writing the letter. The engine decides and names the clause. The model turns “excluded under 4.2b, unattended vehicle” into a paragraph a person can understand. Keep those two jobs apart in the code as well as in the design, because a prompt that receives both the facts and the decision will happily produce a different decision if the wording nudges it.
The customer chat assistant goes to a generative model with retrieval over the policy documents, guardrails, and a handoff. Conversational capabilities are the actual requirement here, and Lex intents do not survive contact with “is my bike covered if it was locked up outside”. Three constraints carry it. First, the assistant answers about cover in general terms and refuses to state a figure from a specific policy schedule. Inaccuracy on an excess amount is the failure mode that generates a complaint. The number itself comes from a lookup against the policy record, rendered as data rather than generated. Second, anything that would change the policy, open a claim or promise a payment goes to a person. Third, ground the answers in retrieved wording and measure how often the answer strays from what was retrieved, the way grounding is measured rather than assumed in a system that has to keep doing it.
Two things are worth being blunt about before any of the three ships. Low temperature reduces nondeterminism and does not remove it, so a plan that depends on identical output from identical input needs a rule underneath it and not a setting. And interpretability does not arrive by asking the model to explain itself: the explanation is generated the same way the answer was, so it describes the answer rather than measuring what caused it. Where the reasoning has to be checkable, put the checkable step in something deterministic and let the model do the writing. The broader form of that judgement, applied before a feature exists at all, is worked through in the question of whether generative AI belongs in the design at all.
Worked example
The same claim, summarised twice
A test harness runs the same claim file through the drafter twice at temperature 0.7. Both summaries are accurate, both are useful, and they are not the same summary: one leads with the assessor’s damage estimate, the other with the disputed liability. Nobody minds, because a handler reads it and the two summaries support the same next action.
The same harness runs an eligibility prompt twice against a claim where the vehicle was left unlocked overnight on a driveway. It returns “not covered, exclusion 4.2b” and, on the second run, “covered, the driveway falls within the insured address”. Turning the temperature to zero makes that disagreement rare rather than impossible, and the insurer has no way to prove which answer a given letter came from. That is the moment the feature stops being a model problem and becomes a rules problem.
A fourth ask arrives
Someone proposes bulk-generating renewal letters that explain each customer’s premium change. The gates run quickly. The letter goes out unread by anyone, so the first gate says yes. Must the same customer’s circumstances produce the same explanation? Yes, because two customers comparing letters is a normal Tuesday. So the premium reasons come from the pricing system as structured facts. The model gets those facts and writes them into English, with a template for anything that states a number. Generation is doing the writing; nothing about the answer is being generated.
What’s worth remembering
- The advantages of generative AI (adaptability, responsiveness, conversational capabilities, and the ability to generate content) come from the same mechanism as its disadvantages, so a design cannot take the first set without accepting the second.
- Hallucinations and inaccuracy are tolerable where a person reads the output before it takes effect and expensive where the output is acted on directly, which is what separates a drafting feature from a deciding one.
- Nondeterminism means identical inputs can give different outputs, and low temperature reduces it without removing it, so any requirement to reproduce an answer belongs in deterministic code.
- Weak interpretability is not fixed by asking the model to explain itself, because that explanation is generated the same way the answer was and describes it rather than measuring its causes.
- A generative model is often the right writer for a decision it should not be making, so let deterministic code produce the answer and let the model write it up.
- Open-ended language in or out points at a generative model, a narrow well-defined task points at a purpose-built AWS AI service, and a repeatable answer somebody has to justify points at rules.