Forty percent. That's how much a business credit line dropped overnight in a case I want to walk through here — a composite, anonymized illustration built from patterns I've seen across several fintech and banking engagements, not a single real account. The borrower had not missed a payment in three years. No complaint sat on file. No dispute, no default, nothing he could point to as the reason.
He called his relationship manager. The honest answer he got was the only honest answer available: the model had flagged elevated risk. Nobody could name the one fact that triggered it, because no single fact did. The line had been cut on a pattern, not an event. There was no stated reason to give him because the reason wasn't a fact anyone had told the bank. It was a conclusion the bank's model had drawn on its own.
The System Wasn't Wrong
Here is the part that should worry operators more than a model error would. When the team traced it back, the system had worked exactly as designed. The inputs were legitimate. The correlation was real. The risk score was, by every technical measure, accurate. This was not a bug ticket. There was nothing to patch.
That is precisely why it's dangerous. We've spent years building the instinct that harm comes from wrong outputs — the false positive, the mislabeled transaction, the model that hallucinates a fact. We test for those. We monitor for those. But this borrower wasn't hurt by an error. He was hurt by a correct inference he never knew the system was capable of drawing, applied to an outcome he had no way to see coming and no path to contest before it landed.
Accuracy was never the risk. Invisibility was.
What "Informed" Never Covered
Every consent form this borrower signed covered the data his bank collected: transaction history, payment timing, account balances. He knew that data existed. He'd agreed to its collection in the ordinary sense that anyone agrees to fine print.
What he never consented to, because no form on earth asks this question, was the conclusion the model drew from that data. Stated data is what a customer tells you or knowingly hands over. Inferred data is what a model concludes from patterns in that data, sometimes patterns the customer isn't aware are patterns at all — a shift in payment timing that correlates with sector-wide stress, a purchasing rhythm that resembles accounts that later defaulted, a geographic or behavioral proxy for risk that was never a line item on any form.
The gap is not a legal technicality. It's the actual site of harm. A customer can dispute a stated fact on file — a missed payment, an incorrect balance, a wrong address. They can call, correct it, and watch the correction take effect. An inference sits nowhere on the file. There is nothing to look at, nothing to correct, and often nothing to even know is there. That asymmetry is the entire problem, and it's the one question almost no deployment checklist asks before launch: what is this model allowed to conclude on its own, separate from what it's allowed to be told?
The Inference Boundary
In my work with operators running credit, pricing, priority, and risk-classification systems, I now ask teams to run every model-driven inference through the same three questions before it's allowed to act on a real outcome. It takes ten minutes per use case and it catches what accuracy testing never will.
- Would the person recognize this conclusion as something we actually know about them, or would it surprise them that we "know" it?
- Can they see the inference and contest it, the same way they could contest a stated fact on file?
- Does the inference change a real outcome for them — price, access, priority, risk classification?
The rule that follows is plain, and it's the one part of this framework I won't soften: if the answer to question three is yes and the answer to question two is no, the inference is forbidden from acting on that outcome. Not flagged for later. Not scheduled for the next model refresh. Forbidden until the disclosure is fixed. This is a policy fix, not a modeling fix — retraining the model doesn't touch the actual harm, because the harm isn't inaccuracy. It's the absence of a way for the person in front of you to see and answer back.
Write the Forbidden Column First
The practical output of that test is not a policy document. It is a two-column list, and I ask teams to build it before a system goes near production. The left column is what the model is permitted to infer and act on. The right column is what it may compute internally but is forbidden to let touch a customer outcome. Most teams have never written the right column at all, which is why it fills up quickly the first time they try.
What surprises people is how often something moves to the forbidden column not because the inference is unfair, but because nobody could explain it to the customer in a sentence. That is a useful filter. If your own team cannot articulate in plain language what the system concluded and on what basis, the customer certainly cannot contest it, and an inference nobody can explain is an inference nobody can defend. The column has a second job as well: it tells you exactly what disclosure work would be required to move an item back to the left. That turns a vague ethical worry into a scoped piece of work with an owner and a deadline.
The list is short. In my experience it runs to eight or ten lines for a single decisioning system, and it takes an afternoon. What it buys is the ability to answer the question that relationship manager could not answer, before the call happens rather than during it.
This isn't a call to strip models of inference. Inference is most of what makes these systems useful. It's a call to stop treating "the conclusion is accurate" as the finish line. Frameworks like the NIST AI Risk Management Framework are already pushing the industry toward this distinction between what a system knows and what it's allowed to act on unilaterally. The operators who get ahead of it won't be the ones with the most accurate models. They'll be the ones whose customers never have to wonder what their AI knows about them that they were never told.
