First of three. We believed more output would create more value. Measuring outcomes rather than activity changed what we believed, and the biggest lesson turned out not to be about AI.
We started our AI work believing that more output would create more value. More analysis, more content, more recommendations, more visible activity.
We were wrong, and the way we found out is worth setting down, because the mistake was not a technical one.
Everything we had built worked, and that was the surprise. The models performed and the pipelines held. Nothing we measured suggested a capability problem at any point.
What was wrong was what we believed about it. We had assumed that producing more, and producing it faster, would show up somewhere that mattered. Nobody had tested the assumption, because the assumption was doing what assumptions do, which is sitting underneath the work looking obvious.
It is a comfortable position. Everything visible is improving. The improvements are real. And there is no way to tell, from inside it, whether any of it is reaching the business.
We started measuring outcomes rather than activity.
That sounds like a small distinction and it is the whole thing. Activity is what the system did. Outcomes are what the organisation could then do that it could not do before. The two quantities do not move together, which is exactly why measuring the first one feels like progress and tells you very little.
The change was one question, asked consistently, with an activity metric refused as an answer.
What changed in the business because it produced that?
Not how much was produced. Not how quickly. Not how good it was. What changed.
Asked once, that question is awkward. Asked of everything, for a period, it is genuinely uncomfortable, because a good deal of very capable work turns out to have no answer at all. Nothing changed. It was produced, it was competent, and the business carried on exactly as it would have.
The numbers get worse before anything gets better. Activity metrics flatter you. Outcome metrics are smaller and harder to produce, and nobody enjoys presenting them. Somebody has to be willing to show a leadership team a smaller number and explain why it is the more useful one.
And some of the work people were proud of does not survive the question. That is a management problem rather than an analytical one, and treating it as anything else is how an organisation quietly goes back to counting documents.
It changed how we close client engagements as well. Go-live used to be where the account of the work stopped. Every engagement we run is now planned with a value review after go-live, measured against the objectives the client signed at the start, and what we learn from it will be fed back into the record the next engagement starts from.
Two findings came out of it, and each one overturned something we had believed.
The first was about supply and demand. We were producing intelligence faster than anyone in the business was asking for it, and we could not connect most of what we had produced to a question anybody had actually raised.
The second was about speed, and the honest measurement of it was not the one we had been quoting.
The next piece takes the first of those. It is the finding that changed how we decide what to work on.
Part of Insights series. Get new articles by email or follow the RSS feed.
Second of three. We compared what we had produced against what anyone had actually asked for. A large share of it could not be matched to a question, and that changed how we choose what to work on.
Most AI initiatives stop at the pilot. The reason is usually not the model. It is that nothing connects an AI output to a decision anyone is willing to be accountable for.
The first correction in our AI work had nothing to do with AI. What an organisation knows is usually scattered, undated and contradictory, and a model working from that will answer confidently and wrongly.
