How do you know when an AI system is ready to run without human review?
Set the cutover on a matched-transaction count, not a calendar date. Run the AI on live transactions while the human process continues untouched, and compare every output. Five decisions make that parallel run real: define agreement field by field in writing beforehand; name one person who reviews every discrepancy within 24 hours; set the threshold, such as 500 consecutive matched transactions or 15 business days, whichever takes longer; let one high-severity mismatch reset the count to zero; and give someone not incentivized by the launch date authority to delay.
$14,000. That's the rough cost of a misrouted payment that sat unnoticed for three weeks inside a system everyone had already decided was working. I want to walk through how that happens — not as a factual case study, but as a composite of a pattern I have seen repeatedly in my work, with the numbers made concrete so the mechanism underneath is visible instead of abstract.
Two Hundred Clean Invoices, Then Silence
The company processed international freight and trading invoices, matching each one against purchase orders and receiving reports before releasing payment. They had spent four months building an AI system to do this matching automatically, and they tested it hard: roughly 200 invoices, pulled from the last two years, run through the model and checked line by line against what a human matcher would have done. The match rate was excellent. The team was proud of it, and they should have been. I see this category of work automated first almost everywhere I go, and the labour data says the same thing: the ILO's refined global index of occupational exposure to generative AI finds that "Clerical occupations continue to have the highest exposure levels." Invoice matching sits squarely in that band.
On go-live day, the manual reviewer who had been checking every match was reassigned to a different queue. Not gradually — immediately. The logic was simple: the system had passed testing, the testing had been rigorous, and keeping a human checking behind it looked like redundant cost.
Invoice 340 came from a new vendor whose export tool merged two cells in a way none of the 200 test invoices had. The vendor field and a tax-line field collapsed into one string. The model, trained to be decisive, resolved the ambiguity with high confidence — and resolved it wrong. It matched the invoice to a similar purchase order from a different shipment and routed payment accordingly.
Nothing flagged it. There was no reviewer left to notice the mismatch, no second set of eyes reading the confidence score against the source document, no process step where a human's intuition — "this vendor name looks off" — could interrupt the flow. The payment cleared. It took three weeks and a routine finance reconciliation, not the AI system or anyone watching it, to surface the error. By then the cost of unwinding it, chasing the vendor, and correcting the ledger came to roughly $14,000.
Passing Testing Is Not the Same as Being Trusted to Run Alone
The instinct after an incident like this is to blame the model. That is almost always the wrong diagnosis. The model did not fail a test it was given — it succeeded on 200 cases and then met a 201st kind of case it had never seen, which is a completely different problem.
Test data is curated. Someone pulled it, cleaned it, made sure it represented the cases the team thought mattered. Production data is not curated by anyone — it is whatever a vendor's export tool, a scanner's OCR pass, or a clerk's typing habits happen to produce that day. In practice that means merged cells, inconsistent date formats, mixed Arabic and English fields in the same document, vendor names spelled four different ways across four different invoices, and edge cases nobody thought to write a test for because nobody had seen one yet.
A system can be 99% accurate on a clean test set and still fail in a way that test set never once represented, because the failure mode lives in the 1% of production reality the test set didn't sample. That is not a technical flaw in the model. It is a procedural gap in how the team decided the system was ready — they measured readiness against history, then handed it live data with no one checking whether the two matched.
The mistake in the example above was not building the model. It was skipping the step between "the model works in testing" and "the model runs unsupervised" — the step where you find out, on real transactions, whether the two are the same thing.
Set the Cutover on an Agreement Count, Not a Date
The fix is not more testing. It is a parallel run: for a defined stretch, the AI processes real, live transactions and the existing human process keeps running alongside it, untouched. Every output gets compared. The system earns the right to run alone by matching the human, repeatedly, on the actual mess of production data — not by passing a curated exam once.
The part most teams get wrong is treating the parallel run's end date like any other project milestone — something you put on a calendar in advance because the launch deck needs a date. That is exactly backwards. The end of the parallel run should be a threshold the data has to earn, not a date the calendar assigns. In my work, five decisions turn that idea into something operational:
- Define "agreement" field by field before the run starts. Some fields need an exact match (vendor ID, invoice number). Others tolerate a defined band (amount within a rounding threshold, date within a processing window). Decide this in writing before a single transaction runs — not while looking at a disagreement and deciding in the moment whether it counts.
- Name who reviews every discrepancy within 24 hours. A specific person, by name, not "the operations team" or "someone on shift." Discrepancies that sit for a week don't get root-caused; they get waved through.
- Set the threshold that ends the run in advance — volume or time, whichever comes second. For example: 500 consecutive matched transactions or 15 business days, whichever takes longer. Never a fixed calendar date alone, because a slow week of volume can make a date meaningless.
- Define what resets the count to zero. One high-severity mismatch — the kind that would have cost real money or damaged a relationship — restarts the clock, even if the count was at 480. The threshold has to mean something, or it's theater.
- Name who has the authority to say "not yet." Specifically, someone whose bonus or performance review is not tied to the go-live date. If the only person who can delay cutover is also the person most incentivized to hit it, the threshold will bend.
None of this is exotic. It is closer to how a hospital brings a new surgical protocol online, or how a bank tests a new fraud rule before switching off the old one — run it alongside the existing process, count the agreements, and only remove the safety net when the count, not the calendar, says it's earned. I have written before about the wider set of checks that belong in a last-mile shipping review; the parallel run is the one I see skipped most often.
The Number That Should Decide
The $14,000 in the example is not the point — it is a stand-in for the gap between "it passed testing" and "it is trusted to run alone." That gap doesn't close with more testing, and it doesn't close with confidence in the model. It closes with a count: the number of times, in a row, on real data, the AI and the human process reached the same answer. Set the cutover there, and you catch the merged-cell invoice before it becomes a reconciliation surprise three weeks later. Set it on a date, and you find out the hard way.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
