How Evidence Changes What You Believe

5 Apr

In my previous posts, we’ve looked at why most organisational measurement is noise and why you need far less data than you think to find the signal. But once you have a piece of evidence—a "signal"—what do you actually do with it?

Most leaders treat new data as a standalone verdict: "The numbers are up; we’re doing great!" or "The numbers are down; hit the panic button!" This is a mistake. Practical measurement isn't about finding an exact number; it is a "quantitatively expressed reduction of uncertainty based on observation".

To lead effectively, you need a formal mechanism for learning from experience. This is the world of Bayesian Inference.

The Prior: Start with What You Already Know

In Bayesian thinking, probability isn't a property of the world; it’s a feature of your own uncertainty. You never start from a place of "zero knowledge".

Before you look at this week’s dashboard, you have a "Prior"—a baseline belief based on past experience or a "Reference Class" (data from similar past initiatives).

Example: Suppose you are trying to predict if you’ll hit your roadmap outcome this quarter. Looking at your team’s history, 6 out of 10 similar initiatives have succeeded.

Your Prior: A 60% probability of success.

The Evidence: Observe a Signal

Now, a new event occurs. Let's say your team reports that cycle time is decreasing.

In a traditional "feature-team" model, you might just report this as a "productivity gain" and move on. But as an empowered leader, you treat this as a signal that needs to be weighed against your prior belief.

The Update: The Power of the Likelihood Ratio

This is where the magic happens. To update your belief, you don’t just ask, "Is this good news?" You ask a specific, statistical question: "How much more likely is this signal if we are on track to succeed versus if we are failing?".

Likelihood if we succeed: If we are genuinely on track to hit the outcome, the probability of seeing cycle times improve is high (let's say 80%).

Likelihood if we fail: If we are actually failing to deliver value, cycle times might still improve (perhaps due to cutting corners), but it’s less likely (say 30%).

The ratio between these two (80% vs 30%) is your Likelihood Ratio. It represents the "strength" of the evidence.

The Posterior: Your New Truth

When you combine your Prior (60%) with the Evidence (the Likelihood Ratio), you get your "Posterior" belief—your updated probability of success.

In our example, this math shifts your belief from a 60% chance of success to an 80% chance.

That 20-point shift is the value of the measurement. If the signal hadn't changed your belief, the metric would have been a "vanity metric"—worthless for decision-making.

Why This Matters for Leaders

This loop: Prior → Evidence → Update is how you transform a metric into a leading indicator.

Uncertainty is not ignorance: You can make informative inferences from very small samples (like the Rule of Five) as long as you are willing to update your beliefs.

Stop "Discovery-Driven" tracking: Don't surround yourself with dashboards hoping to find a pattern. Start with a "Decision-Driven" mindset: identify a choice you need to make and measure only what has the power to change your mind.

Empowerment through context: When you share these updated success probabilities with your team, you provide the Strategic Context they need to solve problems autonomously.

The goal of measurement isn't to be "right"; it's to be less wrong. Every new piece of data is an opportunity to refine your mental map of the project. If you aren't using data to update your posterior beliefs, you aren't measuring—you're just reporting history.