Of 1,200 chats the dashboard called 71 percent resolved, 163 customers wrote again within a week. That is 13.6 percent of the volume, and the dashboard had no column for them.
The company is an illustrative composite, not a real client, and every figure here is illustrative, not research. Picture a 40-person home-appliance retailer that switched on a chat assistant a few months ago. It answers questions about warranties, delivery, returns and whether a product will fit. The monthly report shows 852 conversations marked resolved, a tidy number, and the owner is pleased. Then someone on the showroom floor mentions that the same customers keep ringing back.
The decision in front of that owner is small and a little awkward. Trust the dashboard, or trust the 163. I would trust the 163, and I would read them.
The Dashboard Said Resolved, and 163 Customers Disagreed
A resolved flag usually means one of two things: the customer pressed a button, or went quiet. Neither proves the matter is closed. Silence can be satisfaction. It can also be a customer who concluded that a person would do the job better and picked up the phone.
The returners are the only group who settle the question. They have already voted, in the most honest way available, by spending more of their own time.
My claim is plain. A customer-facing AI is only as good as the share of its correct answers that end the matter, and you can find the ones that do not by reading one week of return visits and sorting each into one of four causes. I call it the Return-Visit Autopsy. It takes an afternoon.
Pull the Pairs, Not the Averages
An average tells you how many came back. A pair tells you why. List every customer who contacted you a second time within seven days of a conversation the assistant handled, matched by phone number, email or account. Each one gives you two transcripts: the first with the assistant, and the second, which may be with the assistant again or with a person.
Put them side by side and read the second one first. Its opening line is usually the customer's own summary of what the first conversation failed to do. Then go back and read the first with that in mind.
Do not read all 163. Take 40 to 60 pairs, chosen without cherry-picking. In the composite I read 60. The sample is small on purpose: you want a direction for the next fix, not a statistic to publish, and the pattern tends to show well before the last pair.
The Four Causes, With One Fragment Each
Tag the first conversation, the one the assistant handled, with exactly one cause. The question is never what the customer did. It is what the assistant did that sent them back.
Wrong question answered. The assistant replied to the words, not the need. A customer wrote, "My fridge is making a loud noise, is that normal?" The assistant quoted the model's noise rating in decibels. Two days later: "I asked if it is broken." In the composite, 21 of the 60 pairs.
Right answer, unusable. "Is the repair covered?" "Yes, the two-year warranty covers the compressor." Correct, and a dead end. The return: "Fine, where do I send it?" There was no address, no form and no timeframe for the customer to act on. 18 of 60.
Right answer, unbelieved. "Can I return it if I opened the box?" "Yes, within 14 days." True. The customer wrote back: "Can someone confirm that?" The answer named no policy and pointed to no source, so a person about to spend real money checked with a person. 9 of 60.
Right answer, incomplete. "Will the dishwasher fit a 60 cm gap?" "Yes, it is 60 cm wide." True, but the next question was always going to be about door clearance and the hose. It arrived the following morning. 12 of 60.

Your Afternoon Checklist
- Define the return window. Seven days is my default. Pick one and keep it fixed so the later comparison is fair.
- Pull the pairs. Every customer who contacted you again inside the window after an assistant-handled conversation, matched by phone, email or account.
- Remove genuine new topics. A customer who bought a dishwasher and now asks about an invoice has not returned for the same matter. Set those pairs aside.
- Tag one cause per first conversation. If two seem to apply, choose the one the customer would name first.
- Count. Four numbers that add up to your sample. If they do not, a pair was tagged twice or skipped.
- Assign by cause owner. Wrong question goes to whoever owns the conversation design. Unusable and incomplete go to the person who writes the answers, with the sales floor supplying the follow-up questions. Unbelieved goes to the policy owner.
- Re-pull in four weeks. Same window, same method, and compare the return share with today's.
What the Count Tells You to Fix First
In the composite the split was 21, 18, 9 and 12, which adds to 60. Wrong question was the largest at 35 percent, but it is also the slowest to fix, because it means the assistant misreads intent and the conversation design needs rework.
The cheap win was the second class, 30 percent of the sample. In all 18 cases the assistant had said the warranty covered the repair and then stopped. The fix was two lines added to that one answer template: where to send the device, and how long a repair takes. Applied to the 163 returners, 30 percent is about 49 customers a month. I would estimate that the change removes close to a third of the returns, and I would treat that as an estimate until the four-week re-pull confirms it.
The mix also turns the review into assignments. Four numbers, three kinds of owner, and no meeting about whether the assistant is good.
What This Will Not Tell You
The customers who gave up in silence never return, so they never appear. Pair the review with a short prompt at the end of a chat, "Did this solve it?", so that some of those are heard too. Where much of this happens on WhatsApp, as it does in Lebanon, one person may also write from two numbers, so matching takes some patience.
Privacy comes before reading. Redact names, phone numbers, addresses and payment details before anyone opens a transcript, keep access to the few people doing the review, and tell customers in your notice that conversations may be reviewed by staff to improve the service. Rules differ by country, so check yours.
The last step is small and reversible. On Thursday afternoon, pull 40 pairs, tag them, and write four numbers on a page. Nothing breaks if you stop. Then tell me which cause dominated. I would be surprised if it was the one you expected.
Related evidence: First contact resolution measures whether a customer's issue is settled without follow-up contacts, and it is best judged from the customer's side, because the customer is the only one who knows whether the issue was resolved. (the first-contact-resolution literature, which holds that only the customer knows whether their issue was actually resolved)
Incentivized experiments find customers avoid a chatbot that may hand them on to a second, expert stage ('gatekeeper aversion'), and recommend transparency about what the bot can handle plus faster live-agent access for customers who experienced chatbot failure. (experimental research on customer-service chatbots, which recommends faster live-agent access for customers who experienced a chatbot failure)
