Back to Blog

Correct, and the Customer Came Back Anyway: Read the Return Visits

A resolved flag tells you the chat ended. The customers who wrote again within a week tell you whether the matter did. Here is how I sort them in one afternoon.

Editorial still-life photograph on a deep navy desk. A steel desk ticket spike on a round base holds a stack of blank cream slips, the top slip marked with a small amber dot. In front of it lie four returned slips, each with a torn hole through its centre and the same amber dot. Headline: Resolved. Then they wrote again. Sub-line: 163 of 1,200 came back.
Direct answer

Why do customers come back after an AI assistant gave the correct answer?

Because a correct answer does not always end the matter. In my Return-Visit Autopsy, I pull every customer who contacted the business again within seven days, read the two transcripts side by side, and tag the first conversation with one cause: wrong question answered, right answer unusable, right answer unbelieved, or right answer incomplete. Forty to sixty pairs is an afternoon, and the cause mix shows who should fix what.

Of 1,200 chats the dashboard called 71 percent resolved, 163 customers wrote again within a week. That is 13.6 percent of the volume, and the dashboard had no column for them.

The company is an illustrative composite, not a real client, and every figure here is illustrative, not research. Picture a 40-person home-appliance retailer that switched on a chat assistant a few months ago. It answers questions about warranties, delivery, returns and whether a product will fit. The monthly report shows 852 conversations marked resolved, a tidy number, and the owner is pleased. Then someone on the showroom floor mentions that the same customers keep ringing back.

The decision in front of that owner is small and a little awkward. Trust the dashboard, or trust the 163. I would trust the 163, and I would read them.

The Dashboard Said Resolved, and 163 Customers Disagreed

A resolved flag usually means one of two things: the customer pressed a button, or went quiet. Neither proves the matter is closed. Silence can be satisfaction. It can also be a customer who concluded that a person would do the job better and picked up the phone.

The returners are the only group who settle the question. They have already voted, in the most honest way available, by spending more of their own time.

My claim is plain. A customer-facing AI is only as good as the share of its correct answers that end the matter, and you can find the ones that do not by reading one week of return visits and sorting each into one of four causes. I call it the Return-Visit Autopsy. It takes an afternoon.

Pull the Pairs, Not the Averages

An average tells you how many came back. A pair tells you why. List every customer who contacted you a second time within seven days of a conversation the assistant handled, matched by phone number, email or account. Each one gives you two transcripts: the first with the assistant, and the second, which may be with the assistant again or with a person.

Put them side by side and read the second one first. Its opening line is usually the customer's own summary of what the first conversation failed to do. Then go back and read the first with that in mind.

Do not read all 163. Take 40 to 60 pairs, chosen without cherry-picking. In the composite I read 60. The sample is small on purpose: you want a direction for the next fix, not a statistic to publish, and the pattern tends to show well before the last pair.

The Four Causes, With One Fragment Each

Tag the first conversation, the one the assistant handled, with exactly one cause. The question is never what the customer did. It is what the assistant did that sent them back.

Wrong question answered. The assistant replied to the words, not the need. A customer wrote, "My fridge is making a loud noise, is that normal?" The assistant quoted the model's noise rating in decibels. Two days later: "I asked if it is broken." In the composite, 21 of the 60 pairs.

Right answer, unusable. "Is the repair covered?" "Yes, the two-year warranty covers the compressor." Correct, and a dead end. The return: "Fine, where do I send it?" There was no address, no form and no timeframe for the customer to act on. 18 of 60.

Right answer, unbelieved. "Can I return it if I opened the box?" "Yes, within 14 days." True. The customer wrote back: "Can someone confirm that?" The answer named no policy and pointed to no source, so a person about to spend real money checked with a person. 9 of 60.

Right answer, incomplete. "Will the dishwasher fit a 60 cm gap?" "Yes, it is 60 cm wide." True, but the next question was always going to be about door clearance and the hose. It arrived the following morning. 12 of 60.

A single stacked bar titled The Return-Visit Autopsy, labelled illustrative composite, showing 60 hand-read sample pairs by cause: Wrong question 21, Unusable 18, Unbelieved 9, Incomplete 12. The Unusable segment is highlighted in amber as the cheap first fix; the other three segments are in shades of blue. Each cause has a one-line definition and the owner of its fix.

Your Afternoon Checklist

  1. Define the return window. Seven days is my default. Pick one and keep it fixed so the later comparison is fair.
  2. Pull the pairs. Every customer who contacted you again inside the window after an assistant-handled conversation, matched by phone, email or account.
  3. Remove genuine new topics. A customer who bought a dishwasher and now asks about an invoice has not returned for the same matter. Set those pairs aside.
  4. Tag one cause per first conversation. If two seem to apply, choose the one the customer would name first.
  5. Count. Four numbers that add up to your sample. If they do not, a pair was tagged twice or skipped.
  6. Assign by cause owner. Wrong question goes to whoever owns the conversation design. Unusable and incomplete go to the person who writes the answers, with the sales floor supplying the follow-up questions. Unbelieved goes to the policy owner.
  7. Re-pull in four weeks. Same window, same method, and compare the return share with today's.

What the Count Tells You to Fix First

In the composite the split was 21, 18, 9 and 12, which adds to 60. Wrong question was the largest at 35 percent, but it is also the slowest to fix, because it means the assistant misreads intent and the conversation design needs rework.

The cheap win was the second class, 30 percent of the sample. In all 18 cases the assistant had said the warranty covered the repair and then stopped. The fix was two lines added to that one answer template: where to send the device, and how long a repair takes. Applied to the 163 returners, 30 percent is about 49 customers a month. I would estimate that the change removes close to a third of the returns, and I would treat that as an estimate until the four-week re-pull confirms it.

The mix also turns the review into assignments. Four numbers, three kinds of owner, and no meeting about whether the assistant is good.

What This Will Not Tell You

The customers who gave up in silence never return, so they never appear. Pair the review with a short prompt at the end of a chat, "Did this solve it?", so that some of those are heard too. Where much of this happens on WhatsApp, as it does in Lebanon, one person may also write from two numbers, so matching takes some patience.

Privacy comes before reading. Redact names, phone numbers, addresses and payment details before anyone opens a transcript, keep access to the few people doing the review, and tell customers in your notice that conversations may be reviewed by staff to improve the service. Rules differ by country, so check yours.

The last step is small and reversible. On Thursday afternoon, pull 40 pairs, tag them, and write four numbers on a page. Nothing breaks if you stop. Then tell me which cause dominated. I would be surprised if it was the one you expected.

Related evidence: First contact resolution measures whether a customer's issue is settled without follow-up contacts, and it is best judged from the customer's side, because the customer is the only one who knows whether the issue was resolved. (the first-contact-resolution literature, which holds that only the customer knows whether their issue was actually resolved)

Incentivized experiments find customers avoid a chatbot that may hand them on to a second, expert stage ('gatekeeper aversion'), and recommend transparency about what the bot can handle plus faster live-agent access for customers who experienced chatbot failure. (experimental research on customer-service chatbots, which recommends faster live-agent access for customers who experienced a chatbot failure)

Frequently asked questions

How do I know whether my AI assistant's correct answers actually settle the customer's question?

Look at the customers who contacted you again within seven days of an assistant-handled conversation. Read each pair of transcripts side by side and tag the first conversation with one of four causes: wrong question answered, right answer but unusable, right answer but unbelieved, or right answer but incomplete. The share of resolved chats that come back, and the cause mix among them, tells you far more than the resolved percentage alone.

How many transcript pairs do I need to read?

Forty to sixty pairs is enough to show a direction, and it takes roughly an afternoon. The aim is to decide what to fix next, not to publish a statistic. If the return volume is small, read all of it. If it is large, sample without cherry-picking and re-pull the same way four weeks after the fix to compare the return share.

Is it acceptable for staff to read customers' chat transcripts?

It can be, if you handle it carefully. Redact names, phone numbers, addresses and payment details before anyone reads the transcripts, limit access to the few people doing the review, and tell customers in your notice that conversations may be reviewed to improve the service. Data-protection rules differ by country, so check yours. This is an operating method, not legal advice.

Who should fix each of the four causes?

The cause mix turns the review into assignments. Wrong-question cases go to whoever owns the conversation design or the product. Unusable and incomplete answers go to the person who writes the answer content, with input from staff who know the follow-up questions. Unbelieved answers go to the policy owner, who can name the policy or supply the source the customer needs to see.

Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.