A contracts coordinator at a mid-size distributor ran one supplier agreement through her AI tool five times, each in a fresh session, and laid the five answers side by side. Supplier name, start date, end date, governing law and payment terms were identical every time. The notice period read 60 days, 60, 30, 60 and 90. Four of nine fields changed between runs. She had been checking all nine fields on every agreement, and until that afternoon she could not have said which ones deserved it. She is an illustrative composite, not a client; the scene is built to show a method.
A tool that disagrees with itself is telling you where to look
Ask an AI tool the same question twice and you may not get the same answer twice. As I understand it, these tools build each answer fresh, with some built-in variation, rather than retrieving a stored one. Often the variation is harmless: the same point in different words. Sometimes the substance changes, and a notice period of 60 days becomes 30.
That difference is useful information. When a tool gives the same answer in five fresh attempts, it is at least steady on that point. When it gives three different answers, it is showing you, at no cost, that it does not reliably know. A field the tool answers differently each time is a field it cannot be left alone with.
You do not need technical skill to gather this. You need one input, a fresh start each time, and a way to line the answers up. Most review plans never use it. They treat every field as equally risky, so everything is checked, or equally safe, so nothing is. Neither matches how the tool behaves.
The Five-Run Spread
The Five-Run Spread is the method I use to decide where checking time goes. It has five steps, in this order.
Step 1. Pick the job and the input. Choose one real job with a known shape: a document goes in, a fixed list of fields or points comes out. Use one real input that you have already read yourself.
Step 2. Run it five times. Use identical input and an identical instruction. Start a fresh session each time, so that one run cannot lean on another.
Step 3. Score each field. Line the five outputs up field by field. For each field, count how many of the five runs give the same answer. Same in substance, not in wording. That count is the field's spread score: 5 of 5, 4 of 5, and so on.
Step 4. Sort the fields into bands. Each band carries its own review rule.
| Band | Runs that agree | Review rule |
|---|---|---|
| Locked | 5 of 5 | Spot-check one input in five against the source. |
| Loose | 3 or 4 of 5 | Check every time against the source. The tool can do it; it cannot be left alone. |
| Split | 2 of 5 or fewer | Do not accept the field as phrased. Narrow the instruction, or have a person do this field. |
Step 5. Run one known-answer check on the Locked band. Agreement is steadiness, not correctness. A tool can be consistently wrong. Compare the Locked fields with the input you already read yourself before you trust the band.
The line I would keep from all of this: the spread tells you where to look; the known answer tells you whether to believe what you are not looking at.
When a field is Split, rewrite the instruction narrower and re-run the five. For example, ask the tool to quote the sentence that sets the point, or to write “not found”. Then report the new score. Do not claim the tool is now correct. Claim it is steadier and cheaper to check. Redo the spread whenever the instruction changes or the job moves to another tool.

The worked example: nine fields, one agreement
Illustrative composite, not a client engagement: the coordinator, her company and every figure below are constructed to show the mechanics. She handles 60 supplier agreements a quarter and pulls nine fields from each. She takes one agreement she has read closely and runs the extraction five times in fresh sessions. Here is the spread.
| Field | Runs agreeing (of 5) | Band |
|---|---|---|
| Supplier name | 5 | Locked |
| Start date | 5 | Locked |
| End date | 5 | Locked |
| Governing law | 5 | Locked |
| Payment terms | 5 | Locked |
| Auto-renewal | 4 | Loose |
| Notice period | 3 | Loose |
| Liability cap | 3 | Loose |
| Price-change clause | 2 | Split |
Five Locked, three Loose, one Split: 5 + 3 + 1 = 9 fields. The four fields that changed between runs are the four that were not Locked, at 4, 3, 3 and 2 of 5. The notice period shows the pattern in miniature: 60 days appeared three times, which is the 3 of 5.
Now the step that matters most. She compares the Locked fields with her own reading. Four are right. The end date is wrong in all five runs. The tool read the original term and missed a later addendum that extended it. That field was Locked and wrong, and no amount of re-running would have shown it. Only the known answer did.
She adds one line to the instruction, “check for addenda that change dates”, and re-runs. The end date now carries the addendum in 5 of 5 runs. For the Split field, she narrows the price-change instruction to “quote the sentence that sets how prices may change, or write ‘not found’”. The score moves from 2 of 5 to 4 of 5. That is now Loose, not Locked, and it stays that way until it earns more. After the fixes the bands read Locked 5, Loose 4, Split 0, which is 9 fields again.
The rule that follows is not “trust the Locked fields”. It is “spot-check them, and prove them once”. Here is what that does to her quarter.
- Before: every field on every agreement, 60 x 9 = 540 field checks.
- After: the four Loose fields checked every time, 60 x 4 = 240; the five Locked fields spot-checked on one agreement in five, 12 agreements x 5 = 60. Total 240 + 60 = 300 field checks.
That is 240 fewer checks, about 44 percent (240 of 540). As an assumption, not a measurement: at a minute a check, the difference is 240 minutes, or four hours a quarter. I would not weigh the saving too heavily. The larger benefit is that the checks now fall where errors actually occur, and the one error that was hiding in a Locked field was found before it cost anything.
Your first Five-Run Spread, in one afternoon
- Choose a job with a fixed list of fields and an input you have already read yourself.
- Run it five times in fresh sessions. Save the five outputs side by side.
- Count agreement per field and write the band beside each one.
- Check the Locked fields against what you know. Fix the instruction for any that are wrong and for any that are Split, then re-run the five.
- Write the review rule for each band on one line and use it for next month’s batch.
What the spread does not show
Five runs is a sample, not proof. A different five could score a field differently, so treat a score of 4 and a score of 3 as neighbours rather than rivals.
A field can agree five times and still be wrong. That is why step 5 exists, and why one input you have read yourself is worth more than a fifth rerun.
Variation in wording is not disagreement in substance. “Sixty days” and “60 days” are the same answer; “60” and “30” are not. Count only the second kind. And a changed instruction or a different tool needs a new spread.
The takeaway is simple: spend review time where the tool disagrees with itself, and prove the steady parts once against something you know. This week, take one job you already check line by line and run it five times. The fields that will not hold still are where your review belongs.
Related evidence: The SelfCheckGPT paper finds that when a language model knows a fact, repeated sampled answers tend to agree, while for hallucinated facts the sampled answers are likely to diverge and contradict one another, so disagreement across runs flags unreliable content without any external database. (the SelfCheckGPT research showing that a language model's repeated answers agree on what it knows and contradict one another on what it invents)
The Nature paper detects a class of hallucinations it calls confabulations by sampling several answers to the same question and measuring how much they disagree in meaning rather than in wording; high disagreement signals an answer that should not be trusted. (the Nature study that flags unreliable AI answers by measuring how much repeated answers disagree in meaning rather than wording)
