Explaining an AI Decision to the Person It Affects

The situation

Karri Mutual is a home-and-contents insurer with about six hundred thousand policies and a claims team of ninety assessors. Two very different models sit in its claims process.

The first is a tabular classifier trained on eight years of settled claims. It scores a lodged claim from zero to one on how likely the claim is to fall outside the policy terms. The features are ordinary claims-desk facts: the claim amount, the event type, the policy age, the excess, the days between the event and the lodgement, and the claimant’s prior claim count. A high score routes the claim to a specialist assessor queue instead of the fast-track queue. It sorts the queue; the decline is an assessor’s. The second is a foundation model on Amazon Bedrock. It drafts the decision letter once an assessor has settled the outcome, and answers the claimant’s follow-up questions in the app, grounded on a knowledge base holding the policy wordings and the product disclosure statement.

Claim 4471 is storm damage to a patio roof, AUD$9,400, lodged sixty-one days after the storm. The classifier scored it 0.82 and sent it to the specialist queue. The assessor read the file, agreed, and declined it: the policy asks for notification as soon as reasonably practicable, and two months with no explanation does not meet that. The letter went out on a Tuesday. By Friday, three separate people had asked why.

The claimant rang, upset, and wanted to know what had actually gone against her and whether anything could be done. The compliance lead is preparing for a review of automated decision-making and needs to show that the models are documented, evaluated and watched. And a developer noticed something odd in the letter: a sentence quoting a flood exclusion, which has nothing to do with a patio roof in a windstorm.

What actually matters

The audience sets the artefact, because each of these three people is about to make a different decision. The claimant has to decide whether to accept the outcome or push back, so what serves her is a reason in words she can check against her own memory of events, plus somewhere to go if she disagrees. The compliance lead has to decide whether the system may keep running, so what serves her is documentation covering the whole system, backed by measurement rather than by assertion. The developer has to decide what to change, so what serves him is a trace of this one request. A feature-importance chart mailed to a claimant is a document she cannot act on, and a friendly one-line summary handed to an auditor is not evidence of anything.

Which model made the call sets what is available at all. For the tabular classifier, a score is a function of named input features, so a per-prediction explanation exists: you can attribute this claim’s 0.82 to the features that pushed it there. For the foundation model that wrote the letter, there is no equivalent. There are no stable input features to attribute, and no chart of token weights that would mean anything to a reader. What stands in for it is traceability: which retrieved passages the sentence came from, what the feature is documented to be for, and a stated rationale. So the difference between the two kinds of model shows up here as a difference in what can be explained.

Then there is who wrote the artefact. Some of what follows is documentation an organisation writes about itself, some is documentation AWS publishes about its own services, and some is a measurement. A document you author can say whatever you type, which is why the compliance lead will ask what backs it. Measurement turns a written claim into a record. The strongest position pairs them: a document stating what the system is for and where it fails, with evaluation results attached to the same page.

Last, an explanation nobody can respond to is a notification. The guide calls this group of concerns the principles of human-centered design for explainable AI, and it covers ground the artefact list does not touch: disclosure, plain language, a route to report a wrong answer, a named human to hear an appeal. That layer is also where the developer’s best debugging signal comes from, because a claimant reporting a wrong answer usually finds a fault before any monitoring does.

What we’ll filter on

1. Audience: who reads it, and what decision does reading it let them make?
2. Scope: does it describe one decision, or the system as a whole?
3. Author and checkability: is it self-written, published by AWS, or measured from real outputs?
4. Model type: does it work on a classic tabular model, on a foundation model, or on both?
5. Timing: is it written once at design time, or produced per request at run time?
6. Next step: does the reader finish it holding something they can do?

The landscape

The guide names its examples of tools for identifying models that are transparent and explainable: Amazon SageMaker Model Cards, Amazon Bedrock Model Evaluations, and open source models, data, licensing. Karri’s situation needs three more alongside them, plus the human-centred layer that none of the artefacts covers.

Amazon SageMaker Model Cards

A structured document you author and keep with the model in SageMaker AI. It records intended use, the training data and process, evaluation results, caveats, an owner and a risk rating of low, medium, high or unknown. Any edit other than an approval-status change creates a new version, and the card exports to PDF, so a reviewer can read the state of the model as at a date. For Karri’s classifier, the card says the model routes claims to a queue and must not be used to decline one, lists the features it consumes, and names the person accountable.

Two things about Model Cards get missed. It is documentation, so nothing checks it against the running system: a card claiming the model only routes, sitting above a pipeline that auto-declines, is a wrong card. And it documents rather than gates. If an unapproved model must not reach production, the approval status in the model registry is what a deployment pipeline tests; the card sits beside it and explains what was approved.

AWS AI Service Cards

AWS’s own transparency documents, published for its managed AI services and its own models. Each sets out intended use cases and limitations, the responsible-AI design choices behind the service, and best practices for deployment and performance. You read one when you adopt a service and cite it in your own records.

The boundary is the thing to hold on to, and it is narrower than the service name suggests. A card covers one capability. Karri uses the Amazon Textract AnalyzeID API to read the driver’s licence a claimant uploads, and the AI Service Card for Textract AnalyzeID covers that API on US driver’s licences and passports. It says nothing about the rest of Textract, and nothing about the triage classifier, the letter, or claim 4471. It is evidence about a component, produced by somebody other than you, which is why an auditor values it.

Feature attribution, and what SageMaker Clarify did with it

The per-prediction explanation for the classic model. SHAP attributes a single prediction to its input features, giving each feature a contribution relative to a baseline, positive or negative, summing to the gap between the baseline score and this one. For claim 4471 that reads as days-to-lodgement contributing +0.31, event type +0.08, claim amount +0.04, policy age -0.02. It answers “why this claim rather than an average one” in a way that a reviewer can argue with.

SageMaker Clarify was the managed way to compute this, along with pre-training and post-training bias metrics. AWS moved it to maintenance on 30 June 2026 and closed it to new customers on 30 July 2026, so do not plan new work on it. Existing customers keep running it; a team starting now computes the same attributions with the open-source SHAP library, which is the engine Clarify was built on and the replacement AWS names. The technique is unchanged and remains the classic-ML answer. Nor has the limit changed: there is no version of this for the foundation model that wrote the letter.

Amazon Bedrock Model Evaluations

Measured evidence about the generative half, named this way in the guide and called Amazon Bedrock evaluations in the console and the documentation. An automatic job takes a prompt dataset from S3, runs the model over it, and scores the outputs with built-in metrics; a variant scores them with a second model acting as judge, which also returns an explanation per response. A human-based job routes the same outputs to a work team you assemble yourself, your own staff or subject-matter experts from the industry, rated against instructions and rating methods you write. There is no AWS-supplied pool of raters here. All of them write per-record scores and a summary back to S3, so the result is data somebody can query later rather than a screenshot in a slide.

For Karri that means three hundred declined claims whose letters an assessor has approved, scored on two things. Does the stated reason match the assessor’s recorded reason? Does every clause quoted actually appear in the policy wording? That is a real number about how often the letter drafting goes wrong, and it belongs in the classifier’s sibling Model Card as evidence. Choosing between models uses the same machinery for a different purpose. An evaluation job describes behaviour across a sample; ask it about one letter and it returns nothing.

Citations from the knowledge base

Per-answer traceability, and the closest thing to a per-decision explanation on the generative side. Retrieval returns passages, the model writes from them, and each citation in the response ties a span of the answer to the chunk behind it. So a letter can quote a clause and link to the paragraph of the product disclosure statement it came from. The claimant, the assessor and the developer can all check it without understanding the model.

Citations are also a diagnosis. The flood sentence has one of two shapes. It carries no citation, so it came from outside the retrieved passages and the fix is grounding and prompt work. Or it cites a chunk retrieval should not have returned, and the fix is retrieval. Requiring a citation for every claim makes that distinction available, and a knowledge base returns the source references with the response rather than as an extra build.

Open source models, data, licensing

The guide’s third named example, and a different sort of transparency. Where a model publishes its weights, and in the strongest cases its training data and evaluation code, anyone can inspect what went into it. A proprietary model gives you the provider’s documentation and the provider’s word; an open model gives you the artefacts. The licence belongs to the same question: field-of-use restrictions, attribution requirements, and whether outputs may be used to train something else. The open-weight and proprietary comparison covers the hosting and evaluation work that follows.

It gives the compliance lead provenance to inspect and the claimant nothing, because public weights say nothing about why her claim went the way it did.

The human-centred layer

The last group is not an AWS artefact but how the decision reaches the person. AI decision transparency starts with telling the claimant that a model was involved, in the letter and in the app, in a sentence she will actually read. Then the reason arrives as a plain-language reason code, drawn from a short list the business wrote and legal approved and mapped from what drove the model. She reads “lodged more than sixty days after the event, and your policy asks you to tell us as soon as reasonably practicable”, not a feature name and a decimal.

User-feedback mechanisms give her a way to say the answer is wrong: a thumbs-down, a correction box, a “this is not what happened” control, wired into a queue a human works through. A feedback control that goes nowhere collects complaints and answers none, worse than not offering one. An appeal path names a person or team, and says how to reach them and by when. And confidence gets stated honestly, so the letter does not read as more certain than the assessor was, and the app carries the model’s qualifiers through rather than flattening them into statements of fact.

Evaluation

Side by side

Artefact	Answers the claimant	Answers the auditor	Answers the developer	About one decision	Works on a foundation model	Measured, not asserted
Amazon SageMaker Model Cards	✗	✓	✗	✗	✓	✗
AWS AI Service Cards	✗	✓	✗	✗	✓	✗
Feature attribution (SHAP)	✗	✓	✓	✓	✗	✓
Amazon Bedrock Model Evaluations	✗	✓	✗	✗	✓	✓
Knowledge base citations	✓	✗	✓	✓	✓	✓
Open weights, data and licence	✗	✓	✓	✗	✓	✗
Reason codes, feedback and appeal	✓	✗	✗	✓	✓	✗

Read down the first column. Five of the seven rows do nothing for the person the decision was made about, and the two that do are both things Karri has to build rather than services it can turn on. Every artefact AWS publishes or hosts lands in the auditor’s column, which describes what those artefacts were designed for and is a bad reason to assume they cover the claimant.

The citation row is the only one ticking both the claimant and the developer, which is a good argument for requiring citations before anyone asks for them. It gives her a clause she can look up and him a trace he can follow, from the same mechanism, with no extra machinery once retrieval is in the path.

The attribution row is the only one with a cross under “works on a foundation model”, and that cross is where teams get into trouble. A design review promising the assistant will show which factors drove its answer has promised something deliverable for the classifier and impossible for the letter. The gap usually surfaces months later, when somebody tries to build the screen.

Routing the question to the artefact

The solution

Give the claimant a letter and an app screen carrying four things and no model internals: a sentence saying a model helped assess the claim and a person made the decision; one or two reason codes from the approved list, tied to the facts of her claim; the clause the decision rests on, quoted, with a link to the paragraph it came from; and a named path to human review with a deadline, beside a control that lets her tell Karri the reason is wrong. None of the AWS artefacts appears on that page; they sit behind it, which is the arrangement to aim for.

Give the compliance lead a Model Card per model, with the measurement attached rather than described. The classifier’s card records intended use, the features it consumes, and its performance broken down by the subgroups the business cares about, with the attribution summary showing which features drive scores across the population. The letter drafting gets its own record, populated by Amazon Bedrock Model Evaluations results run on a schedule rather than once at launch, plus AI Service Cards for the managed services in the pipeline. Then two operational numbers that matter more than either card: how often an assessor overrode the model’s routing, and how many appeals were upheld. A card without those is a statement of intent.

Give the developer a per-request trace, stored from the beginning because it cannot be reconstructed later. Request identifier, the claim, the retrieved chunks with their scores, the prompt template version, the model identifier and version, the citations returned, and the assessor’s final decision. Working the flood clause backwards through that trace is a two-minute job. Without it, the same investigation is a re-run against a prompt that may have changed since.

Date the Model Card, version it, review it whenever the model or its use changes, and have somebody other than the author sign it off. A card that has drifted from the running system is worse than no card, because it will be read as true.

Attribution also answers a narrower question than people hear. A large contribution from a postcode-derived feature says the model used that signal heavily. It does not say the signal is lawful, fair, or causally connected to anything; that is separate work with its own measurements. The same caution applies to the human in the loop. An assessor who signs four hundred model-routed recommendations a day is a rubber stamp, and a decision that has to land on a specific outcome needs better evidence than a signature. Track the override rate; if it sits at nought, the review is not happening. Watching the numbers that show whether the feature is working covers where those figures come from.

Worked example

What the claimant gets

The letter names the decision and the person who made it, then says an automated tool helped sort her claim into the specialist queue. The reason follows in two sentences: her policy requires her to tell Karri about damage as soon as reasonably practicable, and her claim was lodged sixty-one days after the storm on 14 June. The clause is quoted with its section number, and the wording matches the product disclosure statement because the drafting model cited that paragraph and the assessor checked it. At the bottom sit a phone number for the claims review team, a fourteen-day window to ask for review, and a line inviting her to reply if any fact above is wrong. She reads it once and knows what she is arguing about, which is the sixty-one days.

What the compliance lead gets

Two Model Cards. The classifier’s card names its purpose as queue routing, states in terms that it must not be used to decline a claim, and splits accuracy and false-positive rates by policy age and postcode band. The letter drafting’s record carries the last four Amazon Bedrock Model Evaluations runs, each scoring three hundred approved letters on reason-match and clause-accuracy, with the current clause-accuracy figure at 97.6% and every failure logged. Beside them sit the Textract AnalyzeID service card, the override rate for the specialist queue at 11%, and last quarter’s appeals: forty-two lodged, six upheld, all six reviewed for a pattern. The lead is not being asked to take anybody’s word for anything.

What the developer finds

The trace for claim 4471 shows six retrieved chunks. Ranked third is a passage from the flood exclusion section, returned because the storm narrative mentioned water pooling on the patio and the embedding put it close to the flood wording. The letter’s flood sentence carries a citation, and it points at that chunk. So the clause came from the corpus: retrieval returned it, and the prompt set no rule restricting the letter to clauses matching the assessor’s recorded reason. That makes it a retrieval and prompting fix rather than a model swap: filter retrieval to the sections the assessor cited, and require the letter to quote only from those. Had the sentence carried no citation, the same trace would have pointed at a different repair.

What’s worth remembering

1. Match the artefact to who is asking: the person affected needs a plain-language reason and a route to a human, the auditor needs documentation backed by measurement, and the developer needs a per-request trace.
2. Per-prediction feature attribution with SHAP explains one classic-model prediction and has no equivalent for a foundation model, where explainability means traceability; SageMaker Clarify computed it as a managed service until AWS closed Clarify to new customers on 30 July 2026, so reach for the SHAP library directly.
3. Amazon SageMaker Model Cards are documentation you write about your own model and AWS AI Service Cards are AWS documentation about their managed service; neither is measured, and neither enforces anything at run time.
4. Amazon Bedrock Model Evaluations, whether scored automatically, by a judge model, or by a work team you assemble, produce the measured evidence that turns a Model Card from a statement of intent into a record.
5. Open source models, data, licensing gives an auditor provenance to inspect and gives the affected person nothing, so treat it as a procurement and governance decision rather than an explanation.
6. The human-centred layer finishes the job: disclose that a model was involved, give reason codes rather than model internals, wire user-feedback mechanisms into a queue somebody works, and name the human who hears an appeal.