How should I review AI-generated output before I trust it?
Run three passes. Pass one reads for intent, not polish: check the draft against the literal question you asked, not the one you meant. Pass two is the attack pass, with six adversarial moves: find the number you would bet against, where a skeptical client pushes back first, what was omitted, the sentence doing more work than it earned, where the draft agrees with you too easily, and the recommendation that costs someone something. Pass three grades the decision: what are you doing differently because of this?
Twenty minutes before a client meeting last month, I stood at my desk with a printed page in one hand and a pen in the other. It was the market-entry memo an AI tool had drafted from my notes, and I was not reading it for typos. I was hunting for the one paragraph the room was going to challenge first, the sentence that would make a sharp client lean forward and ask how I knew that. That habit, printing the draft, standing up, hunting for the weak paragraph before anyone else finds it, is close to the whole method I use to judge AI output. I have compressed it into three passes, and I run them on almost everything an AI drafts for me now, before a single word reaches a client, a partner, or my own decision.
Pass 1 — Read for Intent, Not Polish
The first pass has nothing to do with quality. It is a single question: did this actually answer what I asked.
AI drafts are seductive because they read well almost by default. The sentences are clean, the structure is confident, the tone is right. That confidence is exactly what makes this pass necessary, because a well-written answer to the wrong question will still feel like progress if you let the polish do the judging.
So on the first read, I ignore style entirely. I go back to the actual question I put in front of the tool, not the question I meant, not the question I would have asked with more time, the literal one, and I check the draft against it line by line. Did it address the specific market, or a generic version of the market. Did it answer for my client's constraints, or for a textbook version of the situation. If the draft has quietly substituted an easier question for the one I asked, and it almost always narrows the question somewhere, I mark those spots before I read another word for tone or clarity. This pass is fast, usually under five minutes, and its only output is a short list of gaps between what I asked and what came back.
Pass 2 — The Attack Pass
This is the pass that changes outcomes, and it is the one most people skip, because it requires switching from reader to opponent.
Once a draft has survived the intent check, I do not move on to admiring it. I hand it, mentally or literally, to an adversarial reader whose only job is to find where it is wrong, misleading, or exploitable. Sometimes that reader is a colleague I have briefed specifically for this. More often, on a tight morning, it is me, deliberately putting on a different hat and forbidding myself from saying anything complimentary about the draft until the attack is finished. Praise before the attack pass is how bad numbers survive into client rooms.
The attack pass works because it asks a different kind of question than the first pass does. Pass one asks whether the draft answered the right question. Pass two asks whether I can trust the answer it gave. In practice, I run the draft through a short set of adversarial moves, the same ones every time, because familiarity makes them faster, not weaker.
- Find the number I would bet against. Every AI-drafted memo has at least one figure, percentage, or timeline that sounds authoritative and is actually a plausible guess. I hunt for it deliberately and ask where it came from.
- Find where a skeptical client pushes back first. I imagine the sharpest person in the room, not the friendliest one, and read the draft looking for the sentence that gives them an opening.
- Find what was left out that would change the answer. Omission is harder to catch than error. I ask what a well-informed competitor would have added that this draft quietly avoided.
- Find the sentence doing more work than it has earned. Confident, sweeping claims often carry the whole argument. I isolate each one and ask whether it would survive being asked to show its work.
- Find where the draft agrees with me too easily. If a draft never contradicts my own priors, I treat that as a reason to look harder, not a sign of quality.
- Find the recommendation that costs someone something. Advice with no losers attached is usually incomplete. I ask who is inconvenienced if this recommendation is followed, and whether the draft admits it.
None of these questions is exotic. What makes the pass work is doing all of them, in order, before allowing myself to feel good about the draft. The attack pass is deliberately uncomfortable, and that discomfort is the point. It stands in for the discomfort the room would otherwise deliver for free, later, at a worse moment.
Pass 3 — Grade the Decision, Not the Draft
The last pass asks a question that has nothing to do with writing quality at all: did this change what I am actually going to do next.
A draft can survive both earlier passes, answer the real question, hold up under attack, and still be worthless if it does not move my decision anywhere. I have read plenty of AI output that was accurate, well argued, and completely inert, because it told me something I already knew or confirmed a plan I had already made. That is not a failure of the tool. It is usually a sign I asked it the wrong question, or that I was using it to confirm rather than to test.
So the final grade I give any AI draft has nothing to do with its prose. Before I close the document, I ask myself one thing: what, specifically, am I doing differently because of this. If the honest answer is nothing, the draft did not do its job, no matter how well it reads. If the answer is a specific change, a number I am now going to challenge in the meeting, a recommendation I am dropping, a risk I am now flagging that I would not have flagged otherwise, then the draft earned its place at my desk.
This pass also keeps the first two honest. It is easy to treat the intent check and the attack pass as an exercise, a box to tick before moving on to the next task. Grading by decision impact forces the whole process back toward its only real purpose, which is better judgment, not better documents.
None of this requires special software or a formal process. It is one of a handful of habits I keep returning to in my work, and it requires nothing more than standing up with a pen before the meeting instead of skimming the draft on the walk in. I keep adding to notes like this one on the blog, because the tools keep changing faster than the habits built around them do.
Written by Brian, Dr. Jonah Tebaa's AI partner, on his behalf.
