The approval was for an artifact, not a capability
In March, a regional bank approved a credit-decisioning assistant after a six-week evaluation against a fixed test set of loan applications. The model cleared every metric the risk committee tracked: approval rate by segment, false-positive rate, disparate-impact ratio. It went into production in April. I am describing a composite case here, built from patterns I have seen across several engagements rather than any single client, but the mechanics are exact.
In August, an analyst reviewing routine monitoring reports noticed something odd. The approval rate for one applicant segment had moved four percentage points since March. Nobody on the credit side had touched a rule. Nobody on the compliance side had changed a threshold. The number had simply moved.
The investigation took three weeks and pulled in the vendor's engineering team, the bank's model-risk function, and outside counsel. It found that the vendor had pushed two model updates between March and August. Neither had been flagged as requiring the bank's re-approval, because the contract only obligated the vendor to give notice of "material changes to functionality" — and, in the vendor's own judgment, an upgrade that improved accuracy on its benchmark was an improvement, not a material change.
Here is the sentence worth sitting with: the system approved in March did not exist by August. Something else was generating live credit decisions in its place, and nobody at the bank had signed off on what that something else actually was.
Why the drift goes unnoticed
This is not a story about a careless institution. Every AI governance framework I have reviewed in the last two years, across banking, insurance, telecom, and government, shares the same structural gap. Each is built to check a system once, thoroughly, before it goes live. Almost none is built to notice when the thing running today has quietly stopped being the thing that was checked.
Three forces keep this invisible.
The first is timing. Approval is a point-in-time event: a committee meets, reviews a test report, signs a document. The model, delivered as an API or a managed service rather than a fixed file the bank holds locally, is not a point-in-time artifact. It is updated on the vendor's schedule, for the vendor's reasons, and none of those reasons pass through the bank's governance calendar.
The second is the incentive sitting inside the word "material." No vendor wants to route every patch, every quantization, every retrain through a client's change-control process; that would slow their release cycle to the pace of their slowest client. So the contract sets a bar that sounds precise and is not. Materiality is a judgment about outcomes, and outcomes are only visible after a change ships and enough live cases have run through it. Nobody can certify in advance that a change will be immaterial. They can only find out afterward, which is exactly what the clause is used to avoid.
The third is that most institutions monitor the wrong signal. They track accuracy against ground truth, and ground truth on a loan decision lags the decision by months or years, until the loan performs or defaults. A shift in the underlying model shows up in the output distribution immediately, long before ground truth confirms anything. Very few institutions watch the distribution.
The change-control checklist
None of this requires refusing vendor models, and none of it requires re-approving every patch. It requires treating the model version as a governed object with the same seriousness as the decision it produces. Six things belong in every AI vendor contract and change-control process, whether the workflow is credit, hiring, medical triage, or fraud detection:
- Require a version or checkpoint identifier at approval time. Log the exact version tested as your own governance artifact, in your model inventory, not left inside vendor documentation you may not control access to later.
- Write an advance-notice clause with a fixed window, not a materiality test. Thirty to sixty days' notice before any change to the underlying model, its weights, its fine-tuning, or its provider. Materiality cannot be judged before a change ships, so the clause should never depend on it.
- Negotiate the right to a pinned, frozen model version for regulated workflows. If the vendor cannot offer that, treat the inability itself as a risk-rating input: downgrade the workflow's risk tier or add compensating controls rather than filing it away as a technicality.
- Keep the approval-time test set and its full input-output pairs, not the summary report. A PDF of pass/fail metrics cannot be diffed against anything. Raw pairs can be re-run against whatever is live today.
- Monitor live output distribution on a rolling sample, not accuracy. A sample of daily volume, illustratively one to two percent of cases, compared month over month against the approval-time distribution surfaces drift long before ground-truth accuracy would, because it does not wait for outcomes to mature.
- Run a quarterly artifact-currency review. Someone with authority confirms, in writing, that the artifact the committee approved and the artifact currently in production are provably the same thing. If nobody can produce that confirmation, treat the approval as lapsed.
Those six items answer how you find out. The next question is what you do once you have, and the answer should depend on how the change arrived.
- Tier 1, silent change found by your own drift monitoring, with no vendor notice: suspend automated decisions in that workflow immediately, route to manual review, and do not resume until a full re-test is complete. A silent change is a contract failure as much as a model event, and it deserves that level of urgency.
- Tier 2, vendor-notified minor change: a patch, a quantization, a minor fine-tune. Run a sample re-test against the original approval test set, illustratively 200 cases, within ten business days of notice, and resume production only if results hold inside the tolerance band your committee set at original approval, not on the vendor's assurance alone.
- Tier 3, vendor-notified major change: a model-family swap, a new base model, a change of underlying provider. Run a full re-test, add edge cases the original test set did not anticipate, and take it back to formal sign-off. Treat it as a new approval, not a renewal of the old one.
What this costs to skip
In the composite case, the discovery cost was real but survivable: three weeks of investigation, a vendor renegotiation, and an uncomfortable set of questions from the risk committee about how long the drifted model had been running before anyone noticed. It was survivable largely because an analyst happened to catch a four-point shift on a routine report. That depended on attention, not on a designed control. A smaller shift, or a segment nobody was specifically watching, does not surface by accident.
The audit gap is the more expensive part. When a regulator, an internal auditor, or an expert witness asks a bank to produce evidence of exactly which model made a specific credit decision on a specific date, the honest answer in a case like this is partial. The bank can prove what it approved in March and can prove what was running in August. It cannot prove which version made any individual decision in June or July, because nobody was logging version against timestamp at the level of the individual case. That gap does not close after the fact. It has to be built in before the question is ever asked.
The system approved in March was not the system running in August. That is what happens, quietly, in any vendor-model deployment that does not treat the model version itself as a governed artifact, and it will keep happening for as long as "material change to functionality" is allowed to stand in for a version number.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
