What does I Audited an AI Loan Rejection Five Months Later mean in practice?
When Dr. Jonah Tebaa audited a five-month-old AI rejection for a forty-five-thousand-dollar loan, reconstructing the decision exposed an unlogged automated decline. The applicant received a score of 47 out of 100, which fell squarely inside the historical 40 to 55 manual-review band. Auditing such decisions requires capturing the pinned model version, frozen inputs, and timestamped policy thresholds at decision-time, aligning with Article 12 of the EU AI Act and the NIST AI Risk Management Framework.
On 10 August 2026, a lawyer's letter lands on a compliance desk. It asks a simple question: on what basis was a $45,000 loan application declined, five months and one week earlier, on 3 March. The bank has ten business days to answer before the matter goes to the regulator.
Take a case of the kind I see regularly in my work with banks, insurers, and fintechs across the region. The details here are illustrative, but the shape of the problem is not. Somewhere in a mid-sized Beirut commercial bank, an AI-assisted credit-scoring tool screens SME working-capital applications. On 3 March, it scores a textile importer's application at 47 out of 100. Under the policy in force that day, anything below 40 is auto-declined, anything above 55 is auto-approved, and everything from 40 to 55 is routed to a human underwriter for manual review. The application is declined the same day it is scored.
Five months later, someone has to explain why.
A Letter About a Decision Made Five Months Ago
This is where most institutions discover that "the model can explain its decisions" and "we can explain a decision made five months ago" are two different claims. The first is a property of the model at the moment of scoring. The second is a property of the organization's records, and it degrades a little every day that passes. Systems get retrained. Policies get revised. The underwriter who might have remembered the case moves teams, or leaves. None of that is unusual. What is unusual is expecting the file to still hold together five months on if nobody built it to.
The compliance lead handling the letter cannot start with an opinion about the decision. She has to start by reconstructing exactly what happened, using only what the system captured at the time. That is a harder task than it sounds.
What Reconstructing One Decision Actually Requires
Reconstructing a single AI-assisted decision from five months back is a small forensic exercise, and it only works if six specific things were captured at the time, not assembled after the fact:
- The model version that actually scored the application on that date, not the current version, and not an assumption. In this case the scoring model has been retrained twice since, in April and in July, for routine feature drift. Naming the wrong version quietly invalidates everything that follows.
- The input snapshot as it existed that day. The applicant's file was updated in May with additional collateral information. That update is real and relevant to today, but it must not be allowed to contaminate the reconstruction of what the model actually saw in March.
- The score and the reason codes the model produced: in this case, 47 out of 100, cited for insufficient collateral coverage and a short banking history.
- The policy threshold in effect on that specific date, not the policy in effect today. The manual-review band here had itself been narrowed in May, from 40-55 to 40-50. Judging a March decision against an August policy is not a technicality. It is a different standard entirely.
- The human sign-off record, if the decision required one: the underwriter's name, the date, and the notes.
- A reconciliation against current state: a short note explaining what has changed since, in the model, the policy, or the data, so that anyone reviewing the file later understands why today's numbers differ from March's without confusing the two.
The NIST AI Risk Management Framework treats this kind of traceability as a governance function, not an engineering afterthought: the ability to map an outcome back to the model, the data, and the decision context that produced it. The EU AI Act's Article 12 goes further for high-risk systems, requiring automatic logging built into the system itself for this exact purpose. Neither framework has to apply to your institution directly for the underlying standard to matter. They are simply the clearest public statements of what "defensible" looks like when someone eventually asks.
The Gap the File Revealed
Here is where the reconstruction stops being a compliance exercise and becomes something else.
47 out of 100, scored on 3 March, falls inside the 40-55 manual-review band under the policy that was in force that day. This application should have gone to a human underwriter. It did not. The file contains an auto-decline letter and nothing else: no underwriter name, no review notes, no sign-off. The system auto-declined an application that its own policy required a person to look at.
Nobody lied. Nobody cut a corner on purpose. The gap existed because nobody had ever reconstructed a single decision end to end until a lawyer's letter forced the exercise. The compliance lead now has two problems stacked on top of each other: answering the original question, and explaining a deployment gap that the act of answering just uncovered.
That second problem is the more expensive one. A wrong decision, caught and corrected quickly, is a cost of doing business. A wrong decision that goes undiscovered for five months, because nobody could see it, is a governance failure, and it tends to surface at the worst possible time, in front of the worst possible audience.
Build the File Before the Letter Arrives
The difference between this becoming a crisis and becoming a non-event was never the model. It was whether the governance file existed at decision-time or had to be assembled at challenge-time. Built at decision-time, the same facts surface quietly in March, to the team that owns the process. Assembled at challenge-time, they surface in August, in writing, in front of a lawyer.
This is not unique to credit scoring. The same four things need to exist, at the moment of decision, for any AI-assisted process a board or regulator might one day ask about: claims denials, underwriting, hiring screens, anything where a score or a model output changes what happens to a person.
- A pinned model version, attached to the decision record itself, not something inferred from a deployment log later.
- A frozen input snapshot, so what the model actually saw is never confused with what the file looks like now.
- A timestamped policy threshold, so the decision is judged against the rule that existed then, not the rule that exists today.
- A hard block on any auto-decision that falls inside a band the policy itself says requires a human, enforced by the system, not by a reviewer's memory.
That last point is the one I would put in front of any board. It converts a policy that lives in a document into a constraint the system cannot violate. Had it existed here, the gap would have been caught by the system in March, not by a lawyer in August, and the ten-day deadline would have been a formality instead of a scramble.
The real test of an AI governance program is never how well it explains a decision on the day it is made. Every vendor demo clears that bar. The real test is whether the decision can be reconstructed and defended from evidence alone, months later, after the model has been retrained twice, the policy has moved, and the people who touched the file have forgotten it existed. It is worth finding out where your own systems stand on that question before a letter arrives, not after.
