When the Decision Has to Be Explainable

The situation

A consumer lender takes about 60,000 personal loan applications a year. Roughly a third are declined. Every declined applicant receives a letter, and the regulator’s rule about that letter is specific: it has to name the reasons the application failed, in terms the applicant can act on. “Your existing repayments are high relative to your declared income” is a reason. “Our model scored you below the threshold” is not.

The credit team has scored applications for eleven years with a scorecard: a logistic regression over about thirty features, with the coefficients printed on a page that the head of credit keeps in a drawer. Two years ago the data science team trained a gradient-boosted model on the same history. It is measurably better. On a held-out year it reaches an AUC of 0.83 against the scorecard’s 0.79, and at a fixed 3.1 per cent default rate it would approve 61.4 per cent of applicants instead of 59.2. That is about 1,300 more loans a year to people who would have repaid them.

The boosted model has sat unused for two years, because nobody could work out what the decline letter would say. Somebody has now proposed a third option: send the whole application pack to a foundation model on Amazon Bedrock and ask it for a decision and a written reason. It produces fluent reasons immediately, which is what makes it dangerous.

What actually matters

Two words get used as though they mean the same thing, and separating them settles most of this. Interpretability is being able to look inside the model and follow how it computes an output: the coefficients, the split points, the rules. Explainability is being able to account for one particular output after the fact, without necessarily opening the model up at all. A scorecard offers both. A gradient-boosted model offers the second through an attribution technique bolted on afterwards. A foundation model offers neither, and its fluent paragraph about why it declined an applicant is generated text, not a report of its own arithmetic.

The obligation here is per-decision, and that is a stricter demand than it first sounds. Knowing that income-to-repayments is the most influential feature across the whole portfolio does not tell Ms Nguyen why her application failed. Somebody has to produce, for her specific application, the two or three attributes that pushed the score below the cut, ordered by how much each contributed. Global understanding of the model and a per-applicant reason are different artefacts, and a technique that gives you one does not give you the other.

Then reproducibility, which is the constraint people forget until an auditor arrives. A complaint about a decline can land three years after the decline. Somebody has to be able to take that application, the model version that scored it, and produce the same reasons again. A scorecard makes this trivial, because the model is a table of numbers that can be rerun by hand. A post-hoc attribution over a boosted model can be reproduced too, but only if the attribution values were computed and stored at decision time along with the model version, the feature values and the code that produced them. Nothing about the model reconstructs them later on its own.

Last, accuracy is a real cost and it should be written down rather than waved at. Four AUC points is 1,300 families a year who could have serviced a loan and were told no. The AWS exam guide asks you to identify tradeoffs between model safety and transparency and gives “measure interpretability and performance” as its worked example, and measure is the operative word. The choice is only defensible once both sides of it carry a number.

What we’ll filter on

1. Per-decision reason. Can the approach produce, for one named applicant, the specific attributes that decided their outcome?
2. Inspectable internals. Can a reviewer read the model itself and follow how an input becomes a score?
3. Accuracy on this task. How does it rank against the alternatives, in AUC and in approvals at a fixed default rate?
4. Auditable reproduction. Three years later, can somebody rerun the decision and get the same reasons, from artefacts that were stored at the time?
5. Faithfulness. Is the explanation derived from the computation that made the decision, or is it a plausible account produced separately?

The landscape

The options form a ladder, from models whose internals are readable to models whose internals are not available at any price. AWS frames the underlying skill as describing the differences between models that are transparent and explainable and models that are not, and this ladder is where those differences live.

Linear and logistic regression

The scorecard. Every feature has one coefficient, and the score is those coefficients multiplied by the feature values and added up. The contribution of each feature to one applicant’s score is a multiplication you can do on paper, which means the explanation and the computation are the same object. There is no separate explaining step to get wrong. The case for a model this plain is usually argued on cost and maintenance grounds, and here it is argued by an obligation that has nothing to do with either.

The limitation is that a straight line is all it can draw. Interactions between features have to be built in by hand: if the relationship between income and risk changes shape above a certain age, somebody has to notice and add a term for it. That hand-built feature work is where most of a scorecard’s accuracy comes from, and it takes a modeller who knows the domain.

Decision trees and rule sets

A tree asks a sequence of yes-or-no questions and lands in a leaf. The path from root to leaf is the explanation, already in the form a letter wants: repayments above 40 per cent of income, then two missed payments in the last year, then decline. Rule sets read the same way. Trees handle interactions naturally, which regression does not.

They stay interpretable only while they stay small. A tree of depth four has at most sixteen leaves and reads like a policy document. A tree of depth twenty has a million paths, and nobody reads a million paths. Depth is the control, and constraining it costs accuracy in exactly the way you would expect.

Gradient-boosted trees

Hundreds or thousands of small trees, each correcting the errors of the ones before it, added together. This is usually the strongest model on tabular data of this kind, and it is the 0.83 in the scenario. It is not interpretable: no human follows eight hundred trees.

It is explainable, through feature attribution. SHAP is the common technique, and it distributes a prediction across the input features so that the contributions add up to the difference between this applicant’s score and the average score. Amazon SageMaker Clarify packaged that computation as a managed job, across a dataset and for a single prediction, but it moved to maintenance in June 2026 and closed to new customers at the end of July, so a lender starting this build owns the job itself and runs the open-source SHAP library in a processing step beside the scoring run. Either way the output is a list of features with signed contributions, which is enough to write a specific reason. Two cautions come with it. The attribution is a model of the model, so a poor approximation gives a confident and wrong reason. And the values have to be computed and stored at decision time, because recomputing them later against a retrained model answers a different question.

Deep neural networks

Millions of weights across many layers, with no readable structure at all. Attribution methods exist for them, and on tabular data of this size they generally do not beat boosted trees anyway, so the accuracy that would justify the opacity is not on offer. For images and text they are the only thing that works, which is why this rung matters elsewhere and not here.

Foundation models on Amazon Bedrock

The weights are not yours to inspect, and for most models on Bedrock they are not published to anyone. No feature attribution is available, because there are no input features in the sense a scorecard has them; there is a prompt. Ask the model why it declined an application and it will write you a paragraph, and that paragraph is a plausible-sounding narrative generated after the fact, with no mechanical connection to whatever produced the decision. Next-token prediction does not produce a reason and then act on it.

That does not make foundation models useless in this workflow. Reading an application pack, extracting fields, summarising a bank statement: all reasonable. Deciding, or authoring the official reason for a decision, is not.

Evaluation

Side by side

Approach	Per-decision reason	Inspectable internals	Accuracy here	Auditable reproduction	Faithful to the decision
Logistic regression scorecard	✓	✓	0.79	✓	✓
Shallow decision tree or rule set	✓	✓	0.77	✓	✓
Gradient-boosted trees	✓ (via attribution)	✗	0.83	✓ (if stored)	✓ (approximate)
Deep neural network	✓ (via attribution)	✗	0.82	✓ (if stored)	✓ (approximate)
Foundation model on Bedrock	✗	✗	untested	✗	✗

Read the first two columns against each other and the shape of the trade appears. Only the top two rows give both, and they are the two lowest accuracy figures. Everything below them buys accuracy by moving the explanation out of the model and into a separate technique that approximates it. The bottom row gives up both columns and offers nothing in the accuracy column to pay for it, which takes it off the table before any of the harder arguments start.

Which gate the obligation trips

Lending trips all three gates, so it lands on the bottom row and the four AUC points are the price of landing there. Note how narrow that outcome is. A model that ranks a marketing list, forecasts warehouse demand or drafts an internal summary stops at the first gate, and arguing about interpretability for those is time spent on a constraint nobody has.

The solution

Keep the scorecard as the decisioning model, spend the modelling effort on closing the accuracy gap by hand rather than by switching architectures, and compute and store a per-application SHAP attribution alongside every decision.

The hand-closing is worth more than it sounds. Most of the boosted model’s advantage on tabular credit data comes from interactions it discovers and the scorecard cannot express: repayment burden behaving differently for applicants with thin credit files, employment tenure mattering more below a certain income. Those can be found by inspecting the boosted model’s own attributions across the population, then added to the scorecard as explicit terms. Doing that here took the scorecard from 0.79 to 0.81, which halves the cost of the decision to roughly 600 approvals a year. Write that number in the decision record, because a trade nobody quantified is a trade somebody will reopen every eighteen months.

Attribution still has a job even with an interpretable model. Per-instance SHAP values give a consistent, ranked, signed set of contributions in one format, computed the same way for every decision, rather than a bespoke script that reimplements the scorecard arithmetic and drifts from it. Clarify ran that computation for the teams already on it; this build, starting after it closed to new customers, runs SHAP in its own processing job, which is a few dozen lines and a scheduled step rather than a project. Store the attribution output with the application record, along with the model version and the exact feature values that went in. That is the artefact an auditor reads three years later, and it exists only if somebody wrote it at the time.

The boosted model does not go in a drawer. Run it in shadow against every application, compare its ranking with the scorecard’s monthly, and use the divergence as a standing measure of what the transparency obligation is costing. When the gap widens, the credit committee has an evidenced conversation about whether the obligation still outweighs it, rather than a preference-based one. Use the boosted model directly wherever no decision falls on an identifiable person: prioritising which files a human reviews first, forecasting portfolio losses, sizing the provisioning line. Same model, different gate.

Bedrock stays well away from the decision. Where it helps is reading unstructured attachments into structured fields the scorecard consumes, and every field it extracts is a field a human can check against the source document. The generated text never becomes the reason for anything.

Where transparency and safety pull apart

The second half of this task statement is the one people skip, because it runs against the instinct that more disclosure is always better. Publishing a model’s weights, its training data and its evaluation results genuinely raises transparency: an outside researcher can reproduce your evaluation, test for behaviours you did not test for, and hold you to what you claimed. Open weights, open data and clear licensing are the strongest form of the transparency claim anyone can make.

They also hand over the ability to undo your safety work. Safety behaviour in a released model sits in the weights, and a few hours of fine-tuning on a small contrary dataset removes most of it. Publish the weights and you have published a model whose refusals are optional. Publish the training data and you have told anyone interested exactly which examples produced those refusals.

Documentation carries the same problem in miniature. A model card that says “this system has been evaluated for prompt-injection resistance and passed at 94 per cent” is useful accountability. A model card that lists the six prompt patterns that got through is a working set of instructions, published under the heading of responsible disclosure. Detail an auditor needs and detail an attacker needs are frequently the same detail.

The resolution is tiered, not a middle setting on one dial. Publish enough for accountability: what the system is for, what it was trained on in general terms, how it was evaluated, what the aggregate results were, the groups it works less well for, and the licence. Hold the operational specifics for people with a reason and an obligation, which is the regulator, the internal audit function and the security team, under the access controls that already govern anything else sensitive. And write the split down as a decision, so the next person can see it was chosen rather than assumed.

For the lender, the same shape appears in smaller form. The scorecard’s readability is a virtue right up until it is published, at which point every applicant learns which three fields to arrange before applying, and the model measures a different thing than it did. Applicants get their own reasons and the actions that would change them. Nobody gets the full coefficient table.

Worked example

An application is declined: 42 years old, £58,000 declared income, existing repayments of £1,940 a month, one missed payment fourteen months ago, credit file six years deep, employed three years.

The scorecard puts the applicant 31 points below the cut. The arithmetic is legible on the page. Repayments-to-income at 40 per cent contributes minus 44 points, the missed payment minus 12, and employment tenure and file depth contribute plus 18 and plus 7. The SHAP attribution over that model returns the same ordering with signed contributions, because with a linear model the attribution and the arithmetic agree by construction. The letter names repayment burden first and the missed payment second, and both are things the applicant can change.

The boosted model declines the same applicant, and its attribution puts repayments-to-income first with a similar magnitude. It also surfaces a third contributor: an interaction between file depth and employment tenure that the scorecard has no term for. Useful, and unusable in a letter, because explaining it means explaining an interaction inside eight hundred trees to somebody who wants to know what to fix. It goes into the modelling backlog as a candidate term for the next scorecard revision, which is how the accuracy gap gets closed.

Bedrock is given the same file and asked to explain the decline. It writes three fluent paragraphs about affordability and payment history. Both paragraphs about payment history describe a second missed payment that is not in the file.

What’s worth remembering

1. Interpretability means the internals can be followed; explainability means one output can be accounted for afterwards, and a model can offer the second without the first.
2. When each decision has to carry its own reason, a population-level view of the model does not satisfy it: the artefact required is a per-decision attribution.
3. Accuracy given up for transparency should be stated in outcomes, not asserted, which is what AWS means by “measure interpretability and performance”.
4. Post-hoc SHAP attribution makes a boosted model explainable but not interpretable, and it is auditable only if the values are computed and stored at decision time with the model version; SageMaker Clarify packaged that computation and closed to new customers at the end of July 2026, so a new build runs it itself.
5. A foundation model’s account of its own reasoning is generated text, so a decision that must be justified belongs to a model whose reason comes from the arithmetic that made it.
6. Publishing weights, data and evaluation detail raises transparency and lets someone fine-tune the safety behaviour back out, so disclose the aggregate account publicly and hold the exploitable specifics for the auditors and regulators who need them.