What should already be in place before an AI system gets something wrong?
Three things, decided in advance rather than improvised under pressure: a named, reachable owner rather than a committee or a shared inbox; pre-granted pause or rollback authority assigned to a specific role, since the gap between technical capability and standing authority is where the damage accumulates; and an automatic timestamped log written at the moment of detection, not reconstructed later. Rehearse it as a tabletop exercise: can you reach the owner, pause, and log inside ten minutes?
At 2:14 PM on a Tuesday, an automated refund engine at a mid-size online retailer processes a routine batch tied to an expired promotional code. A retry bug in the payment webhook causes 214 orders to be refunded three times each. No one sees it happen. The only safeguard is a nightly reconciliation report that runs at 11 PM — nine hours away.
2:14 PM
By 2:17 PM, the first support tickets appear — three or four, easy to dismiss as customers confused by a duplicate email.
By 2:19 PM, a support agent handling that ticket queue notices the pattern and forwards it to a shared "finance-ops" inbox, the informal place unusual things go when there is no formal one.
By 2:21 PM, someone in that inbox asks the only question that matters: is this expected, and if not, who owns the refund engine? No one answers for two minutes, because the honest answer is that several people touched that system when it was built, and none of them currently has "own this in production" written anywhere.
By 2:23 PM, the engineer who wrote the refund logic is found — on a client call, unable to act immediately. The retry loop keeps firing.
By 2:26 PM, someone reaches that engineer's manager, who verbally authorizes pausing the webhook — an action the engineer had the technical ability to take at 2:19 PM but not the standing authority to take alone.
By 2:27 PM, the system is paused. Thirteen minutes have elapsed. Refund volume, which should have plateaued at a detectable few thousand dollars, reached $340,000 in duplicate payouts — most of it in the final six minutes, after the problem had already been spotted but before anyone with the authority to stop it had been located.
There was no timestamped record of any of this. The timeline above was reconstructed two days later from Slack messages and email timestamps, with gaps no one could fully account for.
Why "Who Approved This?" Is the Wrong First Question
Most of the AI governance frameworks I review are built almost entirely around one question: who authorized this system to exist. Model risk committees, vendor due diligence, sign-off chains, documentation requirements. This work is necessary. It is also, on its own, decorative — because it only governs the moment of deployment. It has nothing to say about the moment, weeks or months later, when the system does something no one intended.
Mature engineering organizations solved a version of this problem years ago through incident response and on-call discipline — a named responder, a defined severity ladder, an authority to act before consensus is reached. Most organizations deploying AI systems today have no equivalent muscle. They have an approval gate and, past that gate, nothing but hope.
The distinction matters because the two disciplines answer different questions. Pre-approval governance asks: should this system be allowed to run? Incident response governance asks: given that it is running and has just done something wrong, who owns the next ten minutes? The NIST AI Risk Management Framework treats these as separate functions for a reason — "Govern" and "Manage" are not the same muscle, and a framework's Manage function is explicitly about detecting and responding to problems after a system is live, not just certifying it before launch. The EU AI Act's post-market monitoring obligations point at the same gap: approval is a moment, monitoring and response are ongoing.
In my work with leadership teams, the pattern is consistent. The governance policy exists, often a substantial document. When I ask who gets paged if the system misfires at 2 AM and whether that person can pause it without convening a committee, the room usually goes quiet. The policy answers "who approved this." It has never once been asked to answer "who owns the fix."
The Three Things Every AI Incident Needs Before It Happens
Every gap in the scenario above traces back to one of three missing pieces — none of which require new technology, only a decision made in advance rather than improvised under pressure.
- A named, reachable owner — not a committee or an inbox. "Finance-ops" is not an owner. A shared inbox is not an owner. An owner is one specific person, known in advance, whose contact information is somewhere faster to find than a Slack search, and who knows this system is theirs to answer for at 2 AM as much as 2 PM.
- Pre-granted pause or rollback authority, assigned to a specific role in advance. The engineer in the scenario had the technical means to stop the loop for eight minutes before anyone gave them permission to use it. That gap — between capability and authority — is where almost all of the financial damage in a real incident accumulates. Decide, before anything goes wrong, exactly which role can pull the switch without a meeting.
- An automatic, timestamped log created at the moment of detection — not reconstructed later. A record built from memory and email threads two days after the fact is not a record of what happened; it is a record of what people remember, filtered through the discomfort of the previous forty-eight hours. Detection should write a timestamp automatically, the same instant a threshold is crossed.
None of these require a platform overhaul. They require a decision, made once, before there is any pressure attached to making it well.
Rehearsing the Ten Minutes
The only way to know whether these three things actually work is to test them before a real incident forces the question. I recommend a tabletop exercise: pick a plausible failure — a pricing error, a duplicated transaction, a model producing an output that should never reach a customer — and walk the response in real time, with a clock running and the actual named owner in the room.
The standard is simple and unforgiving: from the moment the simulated failure is "detected," can the organization reach the right person, exercise pause authority, and produce a timestamped record — all inside ten minutes, without anyone needing to improvise who is allowed to do what. If the answer is no, that is the finding. Better to discover it in a conference room than in the thirteen minutes that follow a real 2:14 PM.
Governance built only for the approval gate will always look complete on paper. It is only in the first ten minutes after something goes wrong that you find out whether it was ever real.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
