My phone rang on a Saturday, nineteen days into a planned three-week shadow run. The system had been operating silently in parallel to the client's own team the entire time — reading every case, generating a recommendation, touching nothing customer-facing. No one outside the project had seen it act. That was the point.
The ops lead on the other end of the call wasn't asking me a technical question. She was asking me to move the date. Monday's shift was going to be short two people, and flipping the system live two days early would solve a staffing problem that had nothing to do with the AI itself. Everything in the room felt ready. The number on my screen wasn't.
A Shadow Run With No Finish Line Written Down
Here is the part of the story that matters more than the phone call. When we set up this trial, we agreed on a start date and a rough duration — three weeks, give or take. We did not agree, in writing, on what "ready" meant before a single day of shadow data existed.
That gap is not a minor oversight. It is the single point where a technically sound pilot turns into either a stalled launch or a premature one. Without a number to check against before the trial begins, a team has exactly two ways to end a shadow run: on a calendar date, or never. Neither is a decision. Both are defaults.
I have watched teams sit in "still testing" for months because no one wrote down what passing looked like, so no one could say the system had passed. I have also watched teams flip live on the date printed in the kickoff deck, because a date is concrete and a threshold is not — unless someone wrote the threshold down first. The Saturday phone call was the second failure mode showing up in real time, and the only reason I could resist it was that we had, if imperfectly, done some of the first kind of work already.
The Number Almost Got Overruled by the Calendar
The metric we were tracking was simple to describe and hard to fake: on each day of the shadow run, how often the system's recommended action matched what the human team actually did. We logged it daily, not as an average at the end, but as a running series — because a single good week can hide two bad ones, and a single bad week can hide real drift.
On day nineteen, the match rate was good. It was not yet stable. It had crossed our target threshold on individual days, dropped back below it, crossed again — the kind of pattern that looks like success if you only check it once, and looks unresolved if you're watching the line rather than the last data point. We had told ourselves, before the trial started, that we needed the threshold held for a run of consecutive days, not a single strong day pulled from the middle of a noisy series. On day nineteen, we did not have that run yet.
Every qualitative signal in the room argued the other way. The team liked working alongside it. Nobody had flagged a serious miss in over a week. The staffing problem was real and immediate, and the system looked, by every conversational measure, ready. That is exactly the moment a written number earns its keep — not when everyone agrees the system is failing, but when everyone in the room, including me, wants to say yes and the number has not yet said yes back.
Why I Delayed It Anyway
I held the original date, not because the match rate needed more days to compute — it needed more days to include something we had not yet seen. Ending it on day nineteen would have closed the window without a single peak-volume day inside it. No unusually high case volume, no unusually complex cluster arriving all at once. Every day we had logged was, in that sense, an average day.
That is precisely the condition most likely to hide a real gap. Systems that perform well under normal load are not the same systems that perform well when volume spikes and the queue backs up — the failure modes that show up under pressure are often different in kind, not just degree, from the ones that show up on a quiet Tuesday. Holding the original date meant deliberately waiting for a day we knew, from historical patterns, was likely to be a peak day.
It arrived on day twenty. The match rate held. But two edge cases surfaced that had never appeared in three weeks of average-day data — both involving how the system handled cases arriving faster than the team could review each recommendation before falling back on their own judgment. Neither was disqualifying. Both would have gone into production undetected if we had launched two days early, because the condition that revealed them had not yet occurred inside the trial window.
The Exit Criteria I Should Have Written on Day One
None of this required a more sophisticated system. It required a more complete definition of "done," written before the trial started rather than negotiated during a phone call under staffing pressure. Here is what I now put in writing before any shadow-mode trial begins, regardless of client, sector, or system:
- A minimum sustained match threshold, held for a consecutive number of days — not one good day pulled from a noisy series.
- At least one full peak-volume or peak-complexity day inside the trial window, not only average days, because that is where gaps hide.
- Every divergence case reviewed by a named person, with none left as "we'll get to it later."
- A rehearsed rollback path — tested once in practice, not merely documented in a plan nobody has run.
- A single named accountable reviewer who signs off on the number, not a group consensus that quietly diffuses the decision until no one owns it.
Each of these is small on its own. Together, they turn "does this feel ready" into a question with a checkable answer — which is the only kind of question that can survive a Saturday phone call from someone with a real, immediate, unrelated problem to solve.
Shadow-mode testing is not a new idea. Running a system in parallel before letting it touch customers is standard practice, and most teams I work with already do some version of it. What most teams skip is deciding, in advance, what evidence would make them say no to their own launch date — before the pressure to say yes is standing in the room with a good reason attached.
It is far cheaper to write that list on day one than to construct it, under pressure, on day nineteen.
