A few months ago, I sat in a review meeting where a leadership team presented their AI program results. The deck was clean. The numbers were real. They had saved 14,000 employee hours across three departments in one quarter. The CFO called it a success. The board approved continued funding at the same level.
I didn't say anything in the room. But the thing that struck me wasn't the 14,000 hours. It was what wasn't in the deck: the three programs they had quietly defunded six months earlier because they couldn't show the same kind of number. One of those programs was using AI to monitor contract obligations across a portfolio of 300 vendor relationships — work that had simply never been done before, not because no one thought of it, but because it was humanly impossible at that scale. No hours saved. No labor cost replaced. Just new capability, unquantifiable by their ROI framework, and therefore invisible.
That's the trap I see most enterprises fall into. They apply an industrial-era measuring instrument — time saved multiplied by labor cost — to a post-industrial technology. It's not that the formula is wrong. It's that it only measures one of two fundamentally different things AI can do.
The Industrial Productivity Trap
Standard AI ROI calculations are built on substitution logic: AI does X, a human used to do X, here is what that human cost per hour, here is your return. This is legitimate. Efficiency gains are real, they compound, and they matter to the P&L.
But substitution value is a fraction of the total value on offer. The formula captures what AI replaces. It says nothing about what AI makes possible for the first time. And in my experience, those two numbers are not in the same order of magnitude.
Capability Gain Is a Structurally Different Animal
When I talk about capability expansion, I mean work that did not exist before — not because of budget constraints or bandwidth limits, but because it was structurally impossible without AI. Consider three examples I have seen play out in real programs:
- A legal team running continuous compliance monitoring across 300 active vendor contracts, flagging obligation drift in real time. Previously, this happened once a year, manually, by a team of six, and it still missed things.
- A research function generating and stress-testing 200 product hypotheses overnight, so the morning team walks in with a ranked shortlist instead of a blank whiteboard.
- A commercial team sending genuinely adaptive, context-aware communication to 50,000 client accounts — not personalization templates, but responses calibrated to each account's recent behavior and risk profile.
None of these have a "hours saved" number because none of them replaced existing work. They created new work that wasn't happening. Measuring them with a substitution formula produces either a zero or a distortion. The right question is: what is the value of this capability existing at all?
Decision Quality: The Metric Most Teams Skip
There is a third dimension that matters beyond both efficiency and new capability: decision quality. Since introducing AI into your workflow, are your decisions better calibrated? Are they faster without being less rigorous? Are they more consistent across different teams and contexts?
Decision quality is measurable, but it requires intentional tracking. You need a before-state baseline and a consistent scoring system. Most companies never set one up before they deploy. I'd argue this is the highest-value metric in the portfolio — better decisions compound across every function — and it's the one most consistently absent from AI program reviews.
The Two-Bucket Framework
Here is the framework I use when advising on AI measurement. It is deliberately simple because the point is to force the classification before you choose your KPIs.
Bucket A — Efficiency: AI substitutes for or accelerates existing work. The right metrics here are time reduction, error rate reduction, cost per unit of output. Realistic improvement range: 10 to 40 percent. These are important, fundable, and defensible. Measure them rigorously.
Bucket B — Capability Expansion: AI makes something possible that was impossible before. The right metrics here are adoption rate of the new capability, quality of output relative to the decision or action it enables, and downstream business outcome (revenue influenced, risk avoided, relationship depth increased). The improvement expectation is not 10 to 40 percent — it is a step change, because the baseline was zero.
The critical discipline is not mixing these. A Bucket B program measured with Bucket A logic will always appear to underperform. It has no efficiency number because it was never designed to replace anything. Applying the wrong instrument doesn't give you a low score — it gives you a meaningless one.
The Cost of Measuring Wrong
Companies that only measure Bucket A will optimize toward Bucket A. They will fund what shows an ROI number, defund what doesn't, and gradually shift their entire AI portfolio toward cost reduction. That is a legitimate strategy if cost reduction is the objective. It is a slow-motion competitive error if the objective is building durable advantage.
The organizations I watch with concern are not the ones with modest AI programs. They are the ones with efficient AI programs that are quietly leaving Bucket B entirely to their competitors. In three years, the gap between those two groups will not be measurable in hours saved. It will be measurable in what each company is capable of doing at all.
This Failure Mode Has a Name in the Economics Literature
The two-bucket problem is not a novel observation about AI. It is a specific instance of something economists have modelled for general purpose technologies, and it is worth borrowing their language because it makes the argument to a CFO far harder to wave away.
Brynjolfsson, Rock and Syverson call it the Productivity J-Curve (NBER Working Paper 25148). Their argument: general purpose technologies "enable and require significant complementary investments, including co-invention of new processes, products, business models and human capital," and those investments "are often intangible and poorly measured in the national accounts, even when they create valuable assets for the firm." The consequence they model is precise — "this leads to an underestimation of productivity growth in the early years of a new GPT, and how later, when the benefits of intangible investments are harvested, productivity growth will be overestimated."
That is Bucket B, stated as economics rather than as consulting framework. The period in which a capability program looks worst on the books is the period in which the complementary intangible asset — the reworked process, the retrained team, the new decision routine — is being built and expensed. The measurement does not turn positive because the technology improved. It turns positive because the intangible asset finally shows up in the output. A company that defunds on the trough is not making a disciplined capital decision; it is reading a known curve at its known minimum. Two honest caveats: the paper predates the current generative wave and models economy-level accounting, and a firm-level P&L is not the national accounts. What transfers is the shape and its mechanism, not a schedule you can put in a board deck.
And the discipline of writing down what you cannot measure is itself a published standard. The NIST AI Risk Management Framework (AI 100-1) puts it in the MEASURE function: "Appropriateness of AI metrics and effectiveness of existing controls are regularly assessed and updated," and, in MEASURE 1.1, that the characteristics which "will not – or cannot – be measured are properly documented." Documenting the unmeasurable is not an admission of weakness in a business case — in the framework most regulators and enterprise buyers now benchmark against, it is the control. If a Bucket B program has no efficiency number, the governance-correct move is to record that fact and state which proxy is being used instead, not to invent a savings figure so the program survives review.
Where to Start
If you want to apply this framework, start with an audit, not a build. Take your current AI initiatives — everything funded, piloted, or proposed — and force each one into Bucket A or Bucket B. Then check: do you have KPIs designed for each bucket? Do your Bucket B programs have a baseline for capability value, or are they being judged by an efficiency ruler?
If you find that most of your portfolio is Bucket A and most of your measurement is substitution logic, that's not a failure. It's information. It tells you where the next conversation with your leadership team needs to go.
I'm happy to walk through this classification with your team directly. Reach out if you want to apply it to a specific program or portfolio.
