Every Friday, a finance lead reads 20 of the 500 payment-reminder letters her company's AI drafted that week. She has done it for a month and found nothing. If 2 letters in every 100 carry a wrong balance, as I will assume for this example, that clean month happens about one time in five. Every figure here is hypothetical, built to show a method, not taken from a client.
The firm in my example is an invented 60-person building-supplies distributor in Beirut. The finance lead is doing exactly what she was asked to do. The flaw is not in her. It is in the size of the job, and I want to show why with nothing more than multiplication.
A clean month that proves very little
Start with what is really happening. The AI drafts 500 letters a week, and 2 percent carry a wrong balance: 0.02 x 500 = 10 wrong letters a week. Say each one costs USD 45 in credit notes and staff time (hypothetical). Unseen, that is 10 x 45 = USD 450 a week.
She reads 20 letters at 2 minutes each: 20 x 2 = 40 minutes. The chance that any one letter is clean is 0.98. The chance that all 20 she reads are clean is 0.98 multiplied by itself 20 times, which is 0.6676, about 67 percent. Two Fridays in three, she sees nothing while 10 wrong letters go out. The chance she sees at least one is 1 - 0.6676 = 0.3324, about 33 percent.
Now stretch it to a month. Four clean Fridays in a row: 0.6676 x 0.6676 = 0.4457, and 0.4457 x 0.4457 = 0.1987, about 20 percent, one month in five. Over those four weeks, 4 x 10 = 40 wrong letters were sent, and 40 x 45 = USD 1,800 was spent without anyone seeing the cause.
Her Friday note says "reviewed, no issues". It is true of the 20 letters she read. It is misleading about the 500 that went out. A review that is too small does not just fail to protect you. It produces a comforting record, and that record is the risk.
How a review sees an error
The arithmetic is the same each time. Every letter read is either clean or not. If the true error rate is 2 percent, the chance a read is clean is 0.98. The chance that n reads in a row are all clean is 0.98 multiplied by itself n times, written 0.98^n. The more you read, the faster that number falls.
The question for a manager is how low it must fall. My rule of thumb, derived from this one formula and not taken from any study, is to read about 3 divided by the error rate you cannot accept. At 2 percent, that is 3 / 0.02 = 150 reads. With 150 reads, 0.98^150 is about 0.048, so the chance of seeing nothing is about 4.8 percent and the chance of seeing at least one error is about 95 percent. The same rule gives 300 reads for a 1 percent line and 60 reads for a 5 percent line.
One plain caveat. These probabilities assume each letter is wrong independently of the others, like separate coin tosses. Real errors are not like that. They cluster around a new product line, a new template or a changed price list. So treat this as planning arithmetic, not research, and not a statement about any market or any tool.
The Reading Quota, five steps
The Reading Quota is my working method for sizing a human review. It takes one page.
- Name one error. Pick a single checkable mistake: a wrong amount, a wrong date, a promise the company does not make. "Quality" is not an error you can look for.
- Price a miss. Write what one such error costs when it reaches a customer, in your currency: refund, goodwill, staff time.
- Set the unacceptable rate. Decide the share of outputs with that error you would refuse to live with, for example 2 in 100.
- Compute the quota. Reads per period are about 3 divided by that rate. At 2 percent, 150. A clean run of 150 would happen only about 4.8 percent of the time if the rate were truly 2 percent.
- Draw it blind, and record "not seen". Draw the reads at random from the whole week, not the newest, not the easiest, not the ones staff flagged. A clean quota is written "no error seen at the quota", never "no errors". If the quota is unaffordable, split it across people or weeks and accept the stated delay, or let a mechanical check take the checkable part so people read only for judgement.
This is a planning method, not statistics research. Because real errors cluster, raise the quota right after any change to the template, the data or the model. And if the true rate is lower than your line, the quota will tell you that too.
The same Friday, rebuilt
Apply the quota to the distributor. The error is a wrong balance printed in the letter, the price of a miss is USD 45, and the rate it refuses is 2 percent. The quota is 3 / 0.02 = 150 reads a week. At 2 minutes each that is 150 x 2 = 300 minutes, or 5 hours. That is too much for one finance lead on a Friday, so there are two honest ways to carry it.
First, split it. Three people read 50 letters each, 150 in all, and each spends 50 x 2 = 100 minutes. Second, take the sums out of human hands: a formula recomputes the balance on every letter, so the wrong-balance error is covered for all 500, and the team reads 50 letters for tone and wording, 100 minutes in total. The formula covers the calculable error entirely. People read for what a formula cannot judge.
| Hypothetical plan | Reads a week | Minutes | Chance a week looks clean | Chance four weeks all look clean |
|---|---|---|---|---|
| Current: one person reads 20 | 20 | 40 | 67% | 20% |
| Quota: 150 reads, split in three | 150 | 300 in all, 100 each | 4.8% | under 1 in 100,000 |
The last cell is 0.0483 multiplied by itself four times, which is 0.0000054. In the first week under the quota, the 150 reads show 3 wrong balances: 3 / 150 = 2.0 percent, exactly the rate I assumed. The team holds new sends until the template is fixed. The next week, 150 reads show none, and the note says "no error seen at the quota". That sentence now carries weight, because a rate of 2 percent would have hidden itself from 150 reads only about 4.8 percent of the time.
What to do this week
On Monday, open last month's review notes. Write down the one error you fear, its price, and the rate you refuse to accept. Divide 3 by that rate. Then compare the answer with the number of outputs your team really reads.
If the gap is large, say so to your board in plain words: it is a detection gap, not a clean bill. The reviewer has done nothing wrong. The routine was sized by the time available, not by the risk. Draw the reads blind, write "not seen" instead of "none", and double the quota for two weeks after any change.
That is the whole exercise: one error, one price, one rate, one number. If you want a second pair of eyes on a review routine, write to me.
