Ninety-four percent. That was the accuracy number that got an AI ticket classifier approved for a support operation I worked with at Webspot, my AI strategy firm working across Lebanon and the wider MENA region — refunds, shipping questions, product queries, and complaints, arriving across WhatsApp and web chat, roughly forty agents deep. Ninety-four percent ends a debate in a steering meeting. It looks like a green light. Two weeks after launch, a manual audit of two hundred real incoming messages put live accuracy at sixty-one percent. Same model. Same routing logic. Same team. Thirty-three points gone, and none of it explained by anything wrong with the model itself.
The Number That Got the System Approved
The classifier's job was simple to describe and hard to do well: read an incoming message and route it to the right queue — refund, shipping, product, or complaint — before a human ever touched it. Misroutes meant delay, and delay in support is the fastest way to turn a minor issue into a public complaint. So before anyone signed off, the system was tested. Five hundred tickets went in. Ninety-four percent came out correctly classified. In my experience, that is the point where most teams stop asking questions. The number is high, the sample size sounds respectable, and everyone in the room has other things to approve that day. I don't fault anyone for that instinct. I fault the test set.
Where the 94% Actually Came From
The five hundred tickets used for testing came from the helpdesk's "resolved" archive — a clean, structured, searchable log of past cases. What almost nobody in the approval meeting registered was that these records were not the customer's original messages. They were the agent's written summary, logged at the moment the ticket was closed. Complete sentences. Correct grammar. One language per ticket. A clause structure the model could parse cleanly, because a trained human had already done the parsing.
This is the trap. A test set built from historical records that a person has already cleaned up is not measuring the AI system. It is measuring how well a model can classify text that has already been through a human editor. The 94% was real. It just wasn't answering the question the business needed answered, which was: how will this perform on what actually arrives?
What Live Traffic Looked Like That the Test Set Didn't
Live input is not a summary. It's the raw thing — typed fast, on a phone, often mid-frustration, in whatever language and script feels fastest to the person typing it. Once the system was in production, five patterns showed up constantly that had zero presence in the curated test set:
Messages in Arabizi or mixed script — Arabic written in Latin letters and numerals, sometimes switching mid-sentence into English or French. Messages that were one to three words long with no clause structure at all: "refund??", "wain package", "not working." Transcription artifacts from voice notes converted to text, where the tool guessed at words the speaker never actually used. Forwarded threads — a WhatsApp chain or a copied email — carrying two or three layers of quoted history bundled into what the system read as a single message. And submissions that were only an emoji, a screenshot, or a photo, with no text for a classifier to read at all.
None of that existed in the resolved-ticket archive, because by the time a human agent wrote the closing summary, they had already done the work of interpreting all of it. The test set had been pre-digested. The live channel hadn't been.
The Five-Question Raw-Sample Audit
The fix started with a short audit, and I'd recommend it before trusting any accuracy number tied to a system that will face open, unedited input. Pull a genuinely raw sample from the live channel — not a log a human has touched — and ask:
- Does the sample include mixed-language or mixed-script messages, or is everything in one clean language?
- Does it include very short, fragment-style messages, or does every entry have full sentence structure?
- If any input arrives as voice, has it passed through transcription, and does the sample include the transcription errors, not just the intended meaning?
- Does it include forwarded or quoted threads bundled as a single message, or has each case already been isolated to one clean exchange?
- Does it include image-only, emoji-only, or otherwise textless submissions, or does every case in the sample have usable text?
If the honest answer to most of these is "no, the test set doesn't have that," the accuracy number is not describing the system. It's describing the archive it was drawn from.
Rebuilding the Test Set — and Living With a Lower, Honest Number
The actual fix in this case was not a bigger or better model. It was three weeks of unglamorous work: pulling raw, unedited messages directly from the live input channel and building a new test set from those, artifacts and all. Against that harder, truer sample, accuracy dropped to 79% before any tuning was done. Seventy-nine is a worse-looking number than ninety-four. It is also the only one of the two that turned out to be true, and it's the one that held once the system went back into production, because it had finally been measured against the thing it would actually face.
The broader lesson has stayed with me across every AI system I've helped bring into production since, and it is now the first thing I check before any AI e-mployee goes live on BrianServes, where I deploy autonomous AI e-mployees. Before you trust an accuracy number, don't ask how high it is — ask where the test set came from. Pull a raw, unedited sample from the live input channel, not a cleaned historical log, and check whether it structurally resembles what the model was actually tested on. If the test set was built from records a human already cleaned up before the model ever saw them, the number you're looking at is measuring the human's editing, not the system's judgment.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
