The situation
Karri Mutual is a home-and-contents insurer with about six hundred thousand policies and a claims team of ninety assessors. Two models sit in its claims process, and they are very different animals.
The first is a tabular classifier trained on eight years of settled claims. It takes a lodged claim and scores it from zero to one on how likely the claim is to fall outside the policy terms. The features are ordinary claims-desk facts: the claim amount, the event type, the policy age, the excess, the days between the event and the lodgement, and the claimant’s prior claim count. A high score routes the claim to a specialist assessor queue instead of the fast-track queue. It routes; it does not decide. The second is a foundation model on Amazon Bedrock. It drafts the decision letter once an assessor has decided, and answers the claimant’s follow-up questions in the app, grounded on a knowledge base holding the policy wordings and the product disclosure statement.
Claim 4471 is storm damage to a patio roof, $9,400, lodged sixty-one days after the storm. The classifier scored it 0.82 and sent it to the specialist queue. The assessor read the file, agreed, and declined it: the policy asks for notification as soon as reasonably practicable, and two months with no explanation does not meet that. The letter went out on a Tuesday. By Friday, three separate people had asked why.
The claimant rang, upset, and wanted to know what had actually gone against her and whether anything could be done. The compliance lead is preparing for a review of automated decision-making and wants to show that the models are documented, evaluated and watched. And a developer noticed something odd in the letter: a sentence quoting a flood exclusion, which has nothing to do with a patio roof in a windstorm. All three asked for an explanation of the same decision. Handing any of them what one of the others needs would satisfy nobody.
What actually matters
The audience decides the artefact, because each of these three people is about to make a different decision. The claimant has to decide whether to accept the outcome or push back, so what serves her is a reason in words she can check against her own memory of events, plus somewhere to go if she disagrees. The compliance lead has to decide whether the system may keep running, so what serves her is documentation covering the whole system, backed by measurement rather than by assertion. The developer has to decide what to change, so what serves him is a trace of this one request. A feature-importance chart mailed to a claimant is a document she cannot act on, and a friendly one-line summary handed to an auditor is not evidence of anything.
Which model made the call decides what is even available. For the tabular classifier, a score is a function of named input features, so a per-prediction explanation exists: you can attribute this claim’s 0.82 to the features that pushed it there. For the foundation model that wrote the letter, there is no equivalent. There are no stable input features to attribute, and no chart of token weights that would mean anything to a reader. What stands in for it is traceability: which retrieved passages the sentence came from, what the feature is documented to be for, and a stated rationale. Karri runs one of each kind of model, which is exactly why the difference between the two kinds of model shows up here as a difference in what can be explained at all.
Then there is who wrote the artefact, and whether it can be wrong. Some of what follows is documentation an organisation writes about itself, some is documentation AWS publishes about its own services, and some is a measurement. A document you author can say whatever you decide to type, which is not a reason to skip it, but it is a reason the compliance lead will ask what backs it. Measurement is the thing that turns a written claim into a record. The strongest position pairs them: a document that states what the system is for and where it fails, with evaluation results and monitoring output attached to the same page.
Last, an explanation nobody can respond to is a notification. The guide calls this group of concerns the principles of human-centered design for explainable AI. It covers ground the artefact list does not touch. Was the claimant told a model was involved? Does the reason arrive in ordinary language? Can she report the answer as wrong, and is there a named human who will hear an appeal? That layer is also where the developer’s best debugging signal comes from, because a claimant reporting a wrong answer usually finds a fault before any monitoring does.
What we’ll filter on
- Audience: who reads it, and what decision does reading it let them make?
- Scope: does it describe one decision, or the system as a whole?
- Author and checkability: is it self-written, published by AWS, or measured from real outputs?
- Model type: does it work on a classic tabular model, on a foundation model, or on both?
- Timing: is it written once at design time, or produced per request at run time?
- Next step: does the reader finish it holding something they can do?
The landscape
The guide names its examples of tools for identifying models that are transparent and explainable: Amazon SageMaker Model Cards, Amazon Bedrock Model Evaluations, and open source models, data, licensing. Karri’s situation needs three more alongside them, plus the human-centred layer that none of the artefacts covers.
Amazon SageMaker Model Cards
A structured document you author and keep with the model in SageMaker AI. It records intended use, the training data and process, evaluation results, limitations, known risks, an owner and a risk rating. It is versioned and exportable, so a reviewer can read the state of the model as at a date. For Karri’s classifier, the card says the model routes claims to a queue and must not be used to decline one. It lists the features the model consumes, carries accuracy and false-positive figures broken down by postcode band and policy age, and names the person accountable.
Two things about Model Cards get missed. It is documentation, so nothing checks it against the running system: a card claiming the model only routes, sitting above a pipeline that auto-declines, is a wrong card. And it documents rather than gates. If the requirement is that an unapproved model cannot reach production, the approval status in the model registry is what enforces that; the card sits beside it and explains what was approved.
AWS AI Service Cards
AWS’s own transparency documents, published for its managed AI services and its own models. Each one sets out intended use cases, the design and fairness considerations behind the service, known limitations, and guidance on using it responsibly. You read one when you adopt a service and cite it in your own records to show you understood the envelope you were buying into.
The boundary is the thing to hold on to. A service card describes AWS’s service, and stops there. Karri uses Amazon Textract to lift fields off scanned claim forms; the Textract service card describes what Textract is for and where its accuracy drops, and it says nothing at all about the triage classifier, the letter, or claim 4471. It is evidence about a component, produced by somebody other than you, which is also why an auditor values it.
Feature attribution, and what SageMaker Clarify did with it
The per-prediction explanation for the classic model. SHAP attributes a single prediction to its input features, giving each feature a contribution relative to a baseline, positive or negative, summing to the gap between the baseline score and this one. For claim 4471 that reads as days-to-lodgement contributing +0.31, event type +0.08, claim amount +0.04, policy age -0.02. It answers “why this claim rather than an average one” in a way that a reviewer can argue with.
SageMaker Clarify was the managed way to compute this, along with pre-training and post-training bias metrics. It moved to maintenance in June 2026 and closed to new customers at the end of July. A team already running it keeps its reports; a team starting now computes the same attributions itself with the open-source SHAP library in a processing job. The technique is unchanged and remains the classic-ML answer. What has not changed either is the limit: there is no version of this for the foundation model that wrote the letter.
Amazon Bedrock Model Evaluations
Measured evidence about the generative half. An automatic job takes a prompt dataset from S3, runs the model over it, and scores the outputs with built-in metrics or with a judge model. A human-based job routes the same outputs to reviewers, either a work team you supply or an AWS-managed one, rated against instructions and rating types you define. Both write per-record scores and a summary back to S3, so the result is data somebody can query later rather than a screenshot in a slide.
For Karri that means three hundred declined claims whose letters an assessor has approved, scored on two things. Does the stated reason match the assessor’s recorded reason? Does every clause quoted actually appear in the policy wording? That is a real number about how often the letter drafting goes wrong, and it belongs in the classifier’s sibling Model Card as evidence. Choosing between models uses the same machinery for a different purpose. An evaluation job describes behaviour across a sample; ask it about one letter and it has nothing to say.
Citations from the knowledge base
Per-answer traceability, and the closest thing to a per-decision explanation on the generative side. Retrieval returns passages, the model writes from them, and the response carries which retrieved chunk supported which span of the answer. A letter that quotes a clause and links to the paragraph of the product disclosure statement it came from can be checked by the claimant, the assessor and the developer without any of them understanding the model.
Citations are also a diagnosis. The flood sentence in the letter has one of two shapes. It carries no citation, in which case the model produced it from nothing and the fix is grounding and prompt work. Or it cites a chunk that retrieval should never have returned, and the fix is retrieval. Requiring a citation for every claim is what makes that distinction available at all, and a knowledge base returns the source references as part of the response rather than as an extra build.
Open source models, data, licensing
The guide’s third named example, and a different sort of transparency. Where a model publishes its weights, and in the strongest cases its training data and its evaluation code, anyone can inspect what went into it rather than reading a description of what went into it. A proprietary model gives you the provider’s documentation and the provider’s word; an open model gives you the artefacts. The licence is part of the same question, because it sets what you may do with the model and its outputs: field-of-use restrictions, attribution requirements, and whether outputs may be used to train something else. The open-weight and proprietary comparison covers what that costs you in hosting and evaluation.
This one answers a procurement question and a provenance question. It gives the compliance lead something to inspect and gives the claimant nothing at all, because knowing the weights are public says nothing about why her claim went the way it did.
The human-centred layer
The last group is not an AWS artefact; it is how the decision reaches the person. AI decision transparency starts with telling the claimant that a model was involved, in the letter and in the app, in a sentence she will actually read. Then the reason arrives as a plain-language reason code, drawn from a short list the business wrote and legal approved and mapped from what drove the model. She reads “lodged more than sixty days after the event, and your policy asks you to tell us as soon as reasonably practicable”, not a feature name and a decimal.
Alongside those sit three mechanisms. User-feedback mechanisms give her a way to say the answer is wrong: a thumbs-down, a correction box, a “this is not what happened” control, wired into a queue that a human works through. A feedback control that goes nowhere collects complaints and answers none of them, which is worse than not offering one. An appeal path names a person or a team, says how to reach them and by when. And confidence gets communicated honestly, which mostly means the letter does not sound more certain than the assessor was, and the app hedges where the model hedged instead of smoothing it into a flat statement of fact.
Evaluation
Side by side
| Artefact | Answers the claimant | Answers the auditor | Answers the developer | About one decision | Works on a foundation model | Measured, not asserted |
|---|---|---|---|---|---|---|
| Amazon SageMaker Model Cards | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ |
| AWS AI Service Cards | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ |
| Feature attribution (SHAP) | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ |
| Amazon Bedrock Model Evaluations | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ |
| Knowledge base citations | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ |
| Open weights, data and licence | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
| Reason codes, feedback and appeal | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
Read down the first column. Six of the seven rows do nothing for the person the decision was made about, and the two that do are both things Karri has to build rather than services it can turn on. Every artefact AWS publishes or hosts lands in the auditor’s column, which is a fair description of what those artefacts were designed for and a bad reason to assume they cover the claimant.
The citation row is the only one ticking both the claimant and the developer, which is a good argument for requiring citations before anyone asks for them. It gives the claimant a clause she can look up and the developer a trace he can follow, from the same mechanism, at no extra cost once retrieval is already in the path.
The attribution row is the only one with a cross under “works on a foundation model”, and that cross is where teams get into trouble. A design review that promises the assistant will show which factors drove its answer has promised something deliverable for the classifier and impossible for the letter. The gap usually surfaces months later, when somebody tries to build the screen.
Routing the question to the artefact
The solution
Give the claimant a letter and an app screen that carry four things and no model internals. A sentence saying a model helped assess the claim and a person made the decision. One or two reason codes from the approved list, in ordinary language, tied to the facts of her claim. The clause the decision rests on, quoted, with a link to the paragraph it came from. And a named path to a human review with a deadline, sitting next to a control that lets her tell Karri the reason is wrong. None of the AWS artefacts appears anywhere on that page; they sit behind it, which is the arrangement to aim for.
Give the compliance lead a Model Card per model, with the measurement attached rather than described. The classifier’s card records intended use, the features it consumes, and its performance broken down by the subgroups the business cares about, with the attribution summary showing which features drive scores across the population. The letter drafting gets its own record, populated by Amazon Bedrock Model Evaluations results run on a schedule rather than once at launch, plus AI Service Cards for the managed services in the pipeline. Then two operational numbers that matter more than either card: how often an assessor overrode the model’s routing, and how many appeals were upheld. A card without those is a statement of intent.
Give the developer a per-request trace, and store it from the beginning because it cannot be reconstructed later. Request identifier, the claim it belongs to, the retrieved chunks with their scores, the prompt template version, the model identifier and version, the citations returned, and the assessor’s final decision. Working the flood clause backwards through that trace is a two-minute job. Without it, the same investigation is a re-run against a prompt that may have changed since.
Four things go wrong here often enough to plan around. A Model Card is documentation you write, so date it, version it, and review it whenever the model or its use changes. Have somebody other than the author sign it off. A card that has drifted away from the running system is worse than no card, because it will be read as true. An AI Service Card stops at the boundary of the AWS service, so it never covers your application, your data, or your decision. Feature attribution does not exist for the foundation model. Any commitment to show the claimant which factors drove a generated answer needs correcting before it reaches a design document, because the generated half delivers citations and a stated rationale instead.
And attribution answers a narrower question than people hear. A large contribution from a postcode-derived feature says the model used that signal heavily. It does not say the signal is lawful, fair, or causally connected to anything; that is a separate piece of work with its own measurements. The same caution applies to the human in the loop. An assessor who signs four hundred model-routed recommendations a day is a rubber stamp, and a decision that has to land on a specific outcome deserves better evidence than a signature. Track the override rate; if it sits at nought, the review is not happening. Watching the numbers that show whether the feature is working covers where those figures come from.
Worked example
What the claimant gets
The letter opens by naming the decision and the person who made it, then says that an automated tool helped sort her claim into a queue for specialist assessment. The reason follows in two sentences: her policy requires her to tell Karri about damage as soon as reasonably practicable, and her claim was lodged sixty-one days after the storm on 14 June. The relevant clause is quoted with its section number, and the wording matches the product disclosure statement because the drafting model cited that paragraph and the assessor checked it. At the bottom sit a phone number for the claims review team, a fourteen-day window to ask for review, and a line inviting her to reply if any of the facts above are wrong. She reads it once and knows exactly what she is arguing about, which is the sixty-one days.
What the compliance lead gets
Two Model Cards. The classifier’s card names its purpose as queue routing and states in terms that it must not be used to decline a claim. It lists the features, carries accuracy and false-positive rates split by policy age and postcode band, and attaches an attribution summary showing days-to-lodgement as the dominant driver across the population. The letter drafting’s record carries the last four Amazon Bedrock Model Evaluations runs, each scoring three hundred approved letters on reason-match and clause-accuracy, with the current clause-accuracy figure at 97.6% and every failure logged. Beside them sit the Textract service card, the override rate for the specialist queue at 11%, and last quarter’s appeals: forty-two lodged, six upheld, all six reviewed for a pattern. The lead is not being asked to take anybody’s word for anything.
What the developer finds
The trace for claim 4471 shows six retrieved chunks. Ranked third is a passage from the flood exclusion section, returned because the storm narrative mentioned water pooling on the patio and the embedding put it close to the flood wording. The letter’s flood sentence carries a citation, and it points at that chunk. So the model did not invent the clause; retrieval handed it a passage that looked relevant and the prompt did not tell it to use only clauses matching the assessor’s recorded reason. That makes it a retrieval and prompting fix rather than a model swap: filter retrieval to the sections the assessor cited, and require the letter to quote only from those. Had the sentence carried no citation at all, the same trace would have pointed at a different repair.
What’s worth remembering
- Match the artefact to who is asking: the person affected needs a plain-language reason and a route to a human, the auditor needs documentation backed by measurement, and the developer needs a per-request trace.
- Per-prediction feature attribution with SHAP, which SageMaker Clarify computed as a managed service before it moved to maintenance in June 2026, explains one classic-model prediction and has no equivalent for a foundation model, where explainability means traceability.
- Amazon SageMaker Model Cards are documentation you write about your own model and AWS AI Service Cards are AWS documentation about their managed service; neither is measured, and neither enforces anything at run time.
- Amazon Bedrock Model Evaluations, automatic or human-based, produce the measured evidence that turns a Model Card from a statement of intent into a record.
- Open source models, data, licensing gives an auditor provenance to inspect and gives the affected person nothing, so treat it as a procurement and governance decision rather than an explanation.
- The human-centred layer finishes the job: disclose that a model was involved, give reason codes rather than model internals, wire user-feedback mechanisms into a queue somebody works, and name the human who hears an appeal.