A client of ours ran a successful AI pilot last year. Deflection rates improved. The vendor demo had gone well. Internal stakeholders were optimistic. By the end of Q1, leadership had approved a full rollout.
By Q3, the program was under review. Not because the technology failed. Because nobody had built accountability into what happened after the pilot.
That pattern is more common than most organizations want to admit. And it has very little to do with which AI tool you chose.
The Business Case for AI Gets Built Backward
Most AI business cases start with the technology and work backward to justify the cost. A vendor shows a capability. Someone internally builds a slide deck around it. The numbers get shaped to fit the approval threshold.
What gets skipped is the harder question: what specific operational problem are we solving, and how will we know in 12 months whether we actually solved it?
When I review AI proposals across our engineering and operations engagements, the ones that survive year two share one thing: the business case was written by someone who owned a P&L line, not someone who owned a tech roadmap. Those are different documents. They ask different questions. And they hold different people accountable.
What a P&L Owner Asks That a Tech Team Doesn't
The technology team's business case asks: what does this capability do, and what does it cost to build?
The P&L owner's business case asks: what specific cost comes down, what specific revenue goes up, and by when? And who loses their bonus if it doesn't happen?
Those questions feel uncomfortable in early AI conversations. Vendors don't like them. Internal champions don't like them. But they're the only ones that matter when the CFO asks for a year-two budget review.
Here's the gap I see consistently:
| What gets measured in the pilot | What should be measured at 12 months |
|---|---|
| Deflection rate / automation rate | Cost per resolved outcome |
| Model accuracy | Process cycle time (end to end) |
| User adoption percentage | Revenue or cost impact per FTE |
| NPS from early users | Customer satisfaction vs. pre-AI baseline |
| Implementation on time | ROI vs. business case projection |
The pilot measures what the AI does. The year-two review measures what the business got. Those are not the same measurement.
Why Year Two Is Where It Falls Apart
Pilots are funded differently. They run in controlled conditions, with vendor support, dedicated project teams, and executive attention. Year two is just operations.
In year two, the vendor's implementation team is gone. The internal champion has moved to the next initiative. The process changes that were supposed to happen in parallel haven't happened. And the AI system, which was optimized for the pilot dataset, is now running on real-world variation it wasn't trained on.
None of this is the AI's fault. It's a governance problem.
The delivery organizations that get year two right build the operational governance model before the pilot ends, not after. They define who owns the system in production, who reviews the output, what the escalation path looks like when it gets something wrong, and how the model stays current as the business changes.
That work is unglamorous. It doesn't show up in the pilot dashboard. But it's the difference between a capability that compounds over time and one that slowly degrades until someone puts it on a list of failed investments.
Building a Business Case That Survives the Board
If you're taking an AI investment to a board or leadership team, the case needs three things that most decks don't include.
A clear owner. Not a project manager. Someone whose compensation is linked to the outcome. If no one in the room is willing to own the number at month 12, the program isn't ready to scale.
A defined measurement model. Agreed before the money is approved. Which metrics go up, which go down, and what the baseline is today. Not after six months of implementation, when the baseline has conveniently shifted.
A realistic year-two cost. Most AI business cases include the implementation cost and the licensing cost. They don't include the ongoing cost of model maintenance, retraining, quality review, and the human oversight layer that most AI systems still require to operate reliably at scale.
When those three things are in the business case, the conversation changes. Vendors become more specific about what they can actually guarantee. Internal teams become more honest about what they can realistically absorb. And the investment decision gets made on commercial terms, not enthusiasm.
The Honest Version of the AI ROI Conversation
I've been in rooms where the AI business case looked excellent on paper and the actual outcome was a write-down two years later. I've also seen cases where the business case was modest, the governance was tight, and the program quietly compounded value for years without anyone needing to run a slide deck about it.
The difference wasn't the technology. It was whether the person who approved the budget was the same person who had to account for it at year two.
If you own a P&L and you're evaluating an AI investment right now, the most useful thing you can do is ask your vendor to show you a client whose program is in year two or three, performing against the original business case. Not a testimonial. The actual numbers. How they respond to that request will tell you a great deal about what you're buying.
Intelegencia works with organizations on AI and automation implementations where the commercial accountability sits with us alongside the technical delivery. If you're building the business case or reviewing an existing program, that conversation is worth having.
FAQs
Frequently Asked Questions
Long enough to see it operate without vendor support, which is usually 60 to 90 days. The pilot period that matters isn't the demo phase — it's the two months after the implementation team steps back and your team is running it alone. If performance holds, you have something real. If it degrades, you've learned something important before committing the full budget.
