Back to Blog

Five Runs of One Contract: Review Where Your AI Disagrees With Itself

Run the same job through your AI tool five times on the same input. Where the runs agree, spot-check; where they disagree, review every time, and prove the steady fields once against an answer you know.

Overhead still life of a slim light panel with five translucent drafting sheets stacked on it, their edges stepping out at the top-left corner. Each sheet carries the same nine rows of plain bars. The top five rows show one crisp navy bar each; the lower four show overlapping amber bars that spread apart, widest in the last row. No readable text.
Direct answer

How do you decide which AI outputs to check?

Use the Five-Run Spread. Run one real job through the tool five times on identical input and count, field by field, how many runs agree. Fields that agree 5 of 5 are Locked: spot-check one input in five. 3 or 4 of 5 are Loose: check every time. 2 of 5 or fewer are Split: narrow the instruction or keep the field with a person. Then test the Locked fields once against an answer you know, because steady is not the same as right.

A contracts coordinator at a mid-size distributor ran one supplier agreement through her AI tool five times, each in a fresh session, and laid the five answers side by side. Supplier name, start date, end date, governing law and payment terms were identical every time. The notice period read 60 days, 60, 30, 60 and 90. Four of nine fields changed between runs. She had been checking all nine fields on every agreement, and until that afternoon she could not have said which ones deserved it. She is an illustrative composite, not a client; the scene is built to show a method.

A tool that disagrees with itself is telling you where to look

Ask an AI tool the same question twice and you may not get the same answer twice. As I understand it, these tools build each answer fresh, with some built-in variation, rather than retrieving a stored one. Often the variation is harmless: the same point in different words. Sometimes the substance changes, and a notice period of 60 days becomes 30.

That difference is useful information. When a tool gives the same answer in five fresh attempts, it is at least steady on that point. When it gives three different answers, it is showing you, at no cost, that it does not reliably know. A field the tool answers differently each time is a field it cannot be left alone with.

You do not need technical skill to gather this. You need one input, a fresh start each time, and a way to line the answers up. Most review plans never use it. They treat every field as equally risky, so everything is checked, or equally safe, so nothing is. Neither matches how the tool behaves.

The Five-Run Spread

The Five-Run Spread is the method I use to decide where checking time goes. It has five steps, in this order.

Step 1. Pick the job and the input. Choose one real job with a known shape: a document goes in, a fixed list of fields or points comes out. Use one real input that you have already read yourself.

Step 2. Run it five times. Use identical input and an identical instruction. Start a fresh session each time, so that one run cannot lean on another.

Step 3. Score each field. Line the five outputs up field by field. For each field, count how many of the five runs give the same answer. Same in substance, not in wording. That count is the field's spread score: 5 of 5, 4 of 5, and so on.

Step 4. Sort the fields into bands. Each band carries its own review rule.

The three bands of the Five-Run Spread and the review rule for each.
BandRuns that agreeReview rule
Locked5 of 5Spot-check one input in five against the source.
Loose3 or 4 of 5Check every time against the source. The tool can do it; it cannot be left alone.
Split2 of 5 or fewerDo not accept the field as phrased. Narrow the instruction, or have a person do this field.

Step 5. Run one known-answer check on the Locked band. Agreement is steadiness, not correctness. A tool can be consistently wrong. Compare the Locked fields with the input you already read yourself before you trust the band.

The line I would keep from all of this: the spread tells you where to look; the known answer tells you whether to believe what you are not looking at.

When a field is Split, rewrite the instruction narrower and re-run the five. For example, ask the tool to quote the sentence that sets the point, or to write “not found”. Then report the new score. Do not claim the tool is now correct. Claim it is steadier and cheaper to check. Redo the spread whenever the instruction changes or the job moves to another tool.

The Five-Run Spread, illustrative composite: nine contract fields each run five times. Supplier name, start date, end date, governing law and payment terms agree in all five runs (Locked); auto-renewal agrees in four, notice period and liability cap in three (Loose); the price-change clause agrees in two (Split). The end date is marked steady but wrong. Rules: Locked, spot-check one input in five; Loose, check every time; Split, narrow the instruction or have a person do it.
The Five-Run Spread: where five runs agree, spot-check; where they disagree, review every time. Illustrative composite.

The worked example: nine fields, one agreement

Illustrative composite, not a client engagement: the coordinator, her company and every figure below are constructed to show the mechanics. She handles 60 supplier agreements a quarter and pulls nine fields from each. She takes one agreement she has read closely and runs the extraction five times in fresh sessions. Here is the spread.

Illustrative composite, not a client engagement. Runs agreeing out of five, by field, before any fix.
FieldRuns agreeing (of 5)Band
Supplier name5Locked
Start date5Locked
End date5Locked
Governing law5Locked
Payment terms5Locked
Auto-renewal4Loose
Notice period3Loose
Liability cap3Loose
Price-change clause2Split

Five Locked, three Loose, one Split: 5 + 3 + 1 = 9 fields. The four fields that changed between runs are the four that were not Locked, at 4, 3, 3 and 2 of 5. The notice period shows the pattern in miniature: 60 days appeared three times, which is the 3 of 5.

Now the step that matters most. She compares the Locked fields with her own reading. Four are right. The end date is wrong in all five runs. The tool read the original term and missed a later addendum that extended it. That field was Locked and wrong, and no amount of re-running would have shown it. Only the known answer did.

She adds one line to the instruction, “check for addenda that change dates”, and re-runs. The end date now carries the addendum in 5 of 5 runs. For the Split field, she narrows the price-change instruction to “quote the sentence that sets how prices may change, or write ‘not found’”. The score moves from 2 of 5 to 4 of 5. That is now Loose, not Locked, and it stays that way until it earns more. After the fixes the bands read Locked 5, Loose 4, Split 0, which is 9 fields again.

The rule that follows is not “trust the Locked fields”. It is “spot-check them, and prove them once”. Here is what that does to her quarter.

  • Before: every field on every agreement, 60 x 9 = 540 field checks.
  • After: the four Loose fields checked every time, 60 x 4 = 240; the five Locked fields spot-checked on one agreement in five, 12 agreements x 5 = 60. Total 240 + 60 = 300 field checks.

That is 240 fewer checks, about 44 percent (240 of 540). As an assumption, not a measurement: at a minute a check, the difference is 240 minutes, or four hours a quarter. I would not weigh the saving too heavily. The larger benefit is that the checks now fall where errors actually occur, and the one error that was hiding in a Locked field was found before it cost anything.

Your first Five-Run Spread, in one afternoon

  1. Choose a job with a fixed list of fields and an input you have already read yourself.
  2. Run it five times in fresh sessions. Save the five outputs side by side.
  3. Count agreement per field and write the band beside each one.
  4. Check the Locked fields against what you know. Fix the instruction for any that are wrong and for any that are Split, then re-run the five.
  5. Write the review rule for each band on one line and use it for next month’s batch.

What the spread does not show

Five runs is a sample, not proof. A different five could score a field differently, so treat a score of 4 and a score of 3 as neighbours rather than rivals.

A field can agree five times and still be wrong. That is why step 5 exists, and why one input you have read yourself is worth more than a fifth rerun.

Variation in wording is not disagreement in substance. “Sixty days” and “60 days” are the same answer; “60” and “30” are not. Count only the second kind. And a changed instruction or a different tool needs a new spread.

The takeaway is simple: spend review time where the tool disagrees with itself, and prove the steady parts once against something you know. This week, take one job you already check line by line and run it five times. The fields that will not hold still are where your review belongs.

Related evidence: The SelfCheckGPT paper finds that when a language model knows a fact, repeated sampled answers tend to agree, while for hallucinated facts the sampled answers are likely to diverge and contradict one another, so disagreement across runs flags unreliable content without any external database. (the SelfCheckGPT research showing that a language model's repeated answers agree on what it knows and contradict one another on what it invents)

The Nature paper detects a class of hallucinations it calls confabulations by sampling several answers to the same question and measuring how much they disagree in meaning rather than in wording; high disagreement signals an answer that should not be trusted. (the Nature study that flags unreliable AI answers by measuring how much repeated answers disagree in meaning rather than wording)

Frequently asked questions

Why does my AI tool give different answers to the same question?

As I understand it, these tools build each answer fresh, with some built-in variation, rather than looking up a stored one. Most of the time that shows up as different wording. On some points the substance changes, such as a notice period reading 60 days in one run and 30 in the next. Those are the points that deserve your checking time.

Why five runs, and not two or ten?

Five is enough to separate a steady field from an unsteady one, and few enough to line up by hand in one afternoon. Two runs can agree by luck. Ten adds work without changing the band for most fields. It is a practical sample, not a statistical proof, so treat neighbouring scores such as 3 and 4 of 5 as close.

If all five runs agree, can I stop checking that field?

No. Agreement means the tool is steady, not that it is right. In my worked example an end date agreed in all five runs and was wrong in all five, because a later addendum had been missed. Prove Locked fields once against an input you have read yourself, then spot-check them, for instance one input in five.

What should I do with a field that splits across the runs?

Do not accept it as phrased. Narrow the instruction, for example by asking the tool to quote the sentence that settles the point or to write 'not found', then re-run the five and record the new score. If it still will not settle, keep that field with a person. Claim steadier and cheaper to check, not correct.

Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.