A vendor delivered an impact evaluation of an economic development program run with municipal governments. I was asked to validate it. The real job turned out to be different: showing leadership what the data could and could not support.
The request was simple: check an impact report produced by an outside consultancy before it was used. In practice, the underlying question was whether the organization could rely on those numbers to decide about the program and about the evaluation contract itself.
Reviewing the method, I found methodological and technical inconsistencies that undermined the conclusions.
Pointing out flaws would only solve half the problem. Decision makers would be left with no reliable report and no alternative. So I replaced "is the report right?" with a different question: with the data we have, what can we say about the relationship between program participation and municipal economic indicators, and how confident can we be?
We built the alternative analysis together with the team responsible for the program, who knew the operation and the municipalities.
The work was presented to executive leadership and informed the decision not to continue the evaluation contract.
The result that stayed with me was a different one: making the problem clear to the people who had to decide. A correct technical critique, on its own, would not have changed anything.
A methodological critique only changes a decision when it comes with an alternative the decision maker can use.