Five hundred messages. That's what I pulled from a Beirut consumer brand's WhatsApp inbox the week before their leadership sat down with an AI vendor to sign a support contract. I read every one of them by hand before that call, because I've learned not to trust a demo I didn't design.
Of the 500, only 195 — 39 percent — were written in anything close to formal, MSA-style Arabic. The other 305, 61 percent of real customer traffic, were Lebanese dialect, Arabizi, or a sentence that started in Arabic and finished in English without the customer noticing. "Chou el prix?" "3endkon delivery la Baabda?" "Kifak, b2addesh el shipping ya rab ykoun ma3oul." That last one isn't an edge case. That's a Tuesday.
The Number Nobody Checks Before Buying
Most teams evaluating an Arabic-language AI tool never do this exercise. They sit through a vendor demo, watch it handle a set of sample conversations cleanly, and move to procurement. Nobody asks the one question that actually determines whether the tool will work: what language do our customers write in, measured, not assumed?
In this case the support lead assumed the mix was mostly formal Arabic with some English mixed in — the way it might read in a company's own outbound messaging. The inbox told a different story. Customers don't write to a brand the way a brand writes to them. They write the way they'd text a friend: dialect first, Arabizi when typing Arabic script is slower, English dropped in mid-sentence for delivery, pricing, or product terms that never got a natural Arabic equivalent in daily use. That 61 percent wasn't noise. It was the majority of the business.
What the Vendor Demo Showed vs. What the Real Traffic Did
The vendor's intent-classification demo, run on a curated sample of formal Arabic text, scored 92 percent accuracy. Genuinely impressive on paper. I asked to run the same model against the 305 dialect and Arabizi messages we'd already pulled from the client's own inbox — not a synthetic test set, actual customers, actual traffic. The score dropped to 54 percent.
This isn't a mystery, and it isn't a flaw specific to one vendor. It's mechanical. Most Arabic-language models are trained on corpora built from news archives, government documents, and formal media — text that is, almost by definition, written in MSA. That corpus never taught the model what "3endkon" means, how "b2addesh" maps to "how much," or how to parse a sentence that code-switches into English halfway through without warning. The tokenizer itself often breaks down on Arabizi, because it's not really Arabic and it's not really English — it's a transliteration convention that exists almost entirely in casual messaging, which formal training data doesn't capture. A model can be excellent at MSA and still be nearly blind to the language your customers are actually typing.
The Fix Wasn't a Bigger Model, It Was a Smaller Job
The instinct at this point is usually to look for a "better" Arabic model, or to push the vendor for a custom fine-tune. We did something narrower instead. We scoped the AI down to the traffic it could handle reliably — clean MSA and clearly structured English — and routed everything else, anything flagged as dialect, Arabizi, or ambiguous code-switching, straight to a human queue with a sub-two-minute service-level agreement.
On paper, this made the numbers look worse. Overall containment dropped from the vendor's quoted 92 percent to a production figure of 58 percent. But 58 percent honest beats 92 percent theatrical. The original number was never a description of what would happen with real customers; it was a description of what happens with hand-picked text. The lower, real figure stopped the tool from silently misrouting the majority of the inbox — which is what had actually been generating complaints before anyone measured the language mix at all. An AI that knows what it doesn't understand, and hands it off in under two minutes, outperforms one that guesses confidently in the wrong language.
Five Questions to Ask Before You Scope Any Arabic AI Deployment
- What percentage of your real inbound traffic is Arabizi or code-switched — have you actually sampled it, or are you guessing based on how your brand writes, not how customers write back?
- Did the vendor's benchmark use a sample of your dialect and your customers' phrasing, or a generic MSA test set built for a different market?
- What does the model do when its confidence is low — does it guess and answer anyway, or does it escalate to a human?
- Who reviews a sample of escalated and low-confidence messages every week, and what is the SLA for that human queue?
- Is the contract measured against demo accuracy, or against your production containment rate 30 days after go-live?
Any vendor who can't answer the third and fourth questions with specifics hasn't built for this market. They've built for a market where everyone types the way the news does.
What Changes on Monday
The project I've described here didn't fail because the model was weak. It failed, initially, because nobody had written down, in advance, which slice of the real customer language the AI was actually responsible for. That's not a modeling problem. It's a scoping problem, and it's the kind of gap that a 15-minute demo will never surface.
If you're evaluating or renewing an Arabic-facing AI tool right now, the useful action isn't another vendor call. It's this: pull the last 200 real messages from your primary support channel and tag each one by hand — MSA, dialect, Arabizi, or code-switched. That spreadsheet, built from your own traffic, is the real benchmark. It will tell you more about what you're buying than any demo the vendor puts in front of you.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
