Back to Blog

Twenty Reads a Week Will Miss a Two Percent Error

A weekly read of 20 AI letters looks clean 67% of the time at a 2% error rate. Size the read from the error you cannot accept. Hypothetical.

A round copper sieve with a wooden rim rests on two wooden battens laid across a cream stoneware bowl of chickpeas on navy linen; a few small dark pebbles drop from the sieve into the bowl, and two lie on the chickpeas below. Text: Clean review. Can it even see? Dr. Jonah Tebaa.
Direct answer

How many AI outputs does a weekly review need to read to catch a 2 percent error?

In my hypothetical example, reading 20 of 500 letters a week looks clean about 67 times in 100 at a 2 percent error rate, and four clean weeks in a row happen about one month in five. Dividing 3 by the error rate you cannot accept gives 150 reads, which leaves about a 4.8 percent chance of seeing nothing. Draw them blind and write "no error seen at the quota".

Every Friday, a finance lead reads 20 of the 500 payment-reminder letters her company's AI drafted that week. She has done it for a month and found nothing. If 2 letters in every 100 carry a wrong balance, as I will assume for this example, that clean month happens about one time in five. Every figure here is hypothetical, built to show a method, not taken from a client.

The firm in my example is an invented 60-person building-supplies distributor in Beirut. The finance lead is doing exactly what she was asked to do. The flaw is not in her. It is in the size of the job, and I want to show why with nothing more than multiplication.

A clean month that proves very little

Start with what is really happening. The AI drafts 500 letters a week, and 2 percent carry a wrong balance: 0.02 x 500 = 10 wrong letters a week. Say each one costs USD 45 in credit notes and staff time (hypothetical). Unseen, that is 10 x 45 = USD 450 a week.

She reads 20 letters at 2 minutes each: 20 x 2 = 40 minutes. The chance that any one letter is clean is 0.98. The chance that all 20 she reads are clean is 0.98 multiplied by itself 20 times, which is 0.6676, about 67 percent. Two Fridays in three, she sees nothing while 10 wrong letters go out. The chance she sees at least one is 1 - 0.6676 = 0.3324, about 33 percent.

Now stretch it to a month. Four clean Fridays in a row: 0.6676 x 0.6676 = 0.4457, and 0.4457 x 0.4457 = 0.1987, about 20 percent, one month in five. Over those four weeks, 4 x 10 = 40 wrong letters were sent, and 40 x 45 = USD 1,800 was spent without anyone seeing the cause.

Her Friday note says "reviewed, no issues". It is true of the 20 letters she read. It is misleading about the 500 that went out. A review that is too small does not just fail to protect you. It produces a comforting record, and that record is the risk.

How a review sees an error

The arithmetic is the same each time. Every letter read is either clean or not. If the true error rate is 2 percent, the chance a read is clean is 0.98. The chance that n reads in a row are all clean is 0.98 multiplied by itself n times, written 0.98^n. The more you read, the faster that number falls.

The question for a manager is how low it must fall. My rule of thumb, derived from this one formula and not taken from any study, is to read about 3 divided by the error rate you cannot accept. At 2 percent, that is 3 / 0.02 = 150 reads. With 150 reads, 0.98^150 is about 0.048, so the chance of seeing nothing is about 4.8 percent and the chance of seeing at least one error is about 95 percent. The same rule gives 300 reads for a 1 percent line and 60 reads for a 5 percent line.

One plain caveat. These probabilities assume each letter is wrong independently of the others, like separate coin tosses. Real errors are not like that. They cluster around a new product line, a new template or a changed price list. So treat this as planning arithmetic, not research, and not a statement about any market or any tool.

The Reading Quota, five steps

The Reading Quota is my working method for sizing a human review. It takes one page.

  1. Name one error. Pick a single checkable mistake: a wrong amount, a wrong date, a promise the company does not make. "Quality" is not an error you can look for.
  2. Price a miss. Write what one such error costs when it reaches a customer, in your currency: refund, goodwill, staff time.
  3. Set the unacceptable rate. Decide the share of outputs with that error you would refuse to live with, for example 2 in 100.
  4. Compute the quota. Reads per period are about 3 divided by that rate. At 2 percent, 150. A clean run of 150 would happen only about 4.8 percent of the time if the rate were truly 2 percent.
  5. Draw it blind, and record "not seen". Draw the reads at random from the whole week, not the newest, not the easiest, not the ones staff flagged. A clean quota is written "no error seen at the quota", never "no errors". If the quota is unaffordable, split it across people or weeks and accept the stated delay, or let a mechanical check take the checkable part so people read only for judgement.

This is a planning method, not statistics research. Because real errors cluster, raise the quota right after any change to the template, the data or the model. And if the true rate is lower than your line, the quota will tell you that too.

The same Friday, rebuilt

Apply the quota to the distributor. The error is a wrong balance printed in the letter, the price of a miss is USD 45, and the rate it refuses is 2 percent. The quota is 3 / 0.02 = 150 reads a week. At 2 minutes each that is 150 x 2 = 300 minutes, or 5 hours. That is too much for one finance lead on a Friday, so there are two honest ways to carry it.

First, split it. Three people read 50 letters each, 150 in all, and each spends 50 x 2 = 100 minutes. Second, take the sums out of human hands: a formula recomputes the balance on every letter, so the wrong-balance error is covered for all 500, and the team reads 50 letters for tone and wording, 100 minutes in total. The formula covers the calculable error entirely. People read for what a formula cannot judge.

Hypothetical planReads a weekMinutesChance a week looks cleanChance four weeks all look clean
Current: one person reads 20204067%20%
Quota: 150 reads, split in three150300 in all, 100 each4.8%under 1 in 100,000

The last cell is 0.0483 multiplied by itself four times, which is 0.0000054. In the first week under the quota, the 150 reads show 3 wrong balances: 3 / 150 = 2.0 percent, exactly the rate I assumed. The team holds new sends until the template is fixed. The next week, 150 reads show none, and the note says "no error seen at the quota". That sentence now carries weight, because a rate of 2 percent would have hidden itself from 150 reads only about 4.8 percent of the time.

What to do this week

On Monday, open last month's review notes. Write down the one error you fear, its price, and the rate you refuse to accept. Divide 3 by that rate. Then compare the answer with the number of outputs your team really reads.

If the gap is large, say so to your board in plain words: it is a detection gap, not a clean bill. The reviewer has done nothing wrong. The routine was sized by the time available, not by the risk. Draw the reads blind, write "not seen" instead of "none", and double the quota for two weeks after any change.

That is the whole exercise: one error, one price, one rate, one number. If you want a second pair of eyes on a review routine, write to me.

Frequently asked questions

What if I cannot afford to read 150 outputs a week?

Split the reads across several people or across weeks, and state the delay in learning plainly. Where the error can be calculated, such as a balance, let a formula check every output so people read only for judgement. Give the full quota only to the costly error; a low-stakes mistake can carry a much smaller one.

Does a clean quota mean my AI makes no mistakes?

No. It means no error of that one type was seen at that quota. Write it as "no error seen at the quota", never "no errors". Rarer mistakes, or other kinds, can still exist. A clean result is evidence about the rate you set as your line, and nothing more.

How do I pick the error rate I cannot accept?

Start from the weekly loss you would refuse to carry. Divide it by the price of one miss and by your weekly volume. In my hypothetical example, USD 450 a week divided by USD 45 is 10 letters, and 10 of 500 is 2 in 100, the line the firm refused. Your own price of a miss sets your own line.

When should I raise the quota?

After any change to the template, the data, the model, the product line or the season. Errors cluster at change points, so a quota that was enough last month may not be now. I would double it for two weeks after a change, then return to the normal figure if nothing is seen.

Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.