Lagging, leading, and proxy indicators

This probably seems to be a fairly boring topic, or at least one you fully understand. But I've been thinking a lot about different indicators (i.e. KPIs) this week, the differences between them, and the potential value and risks of using them.

I'm going to use two different examples in this article. The first is from outside of a work environment for anyone new to the topic: trying to lose weight. The second from the world of technology: delivering features and functionality faster to customers.

Lagging indicators - what matters

Lagging indicators should be the things that measure what matters. This is a measure of the thing that you care about changing.

Weight loss

By default, the lagging indicator here is, of course, your weight. The goal of weight loss is to make this number smaller. But even just with this simple example, we can see two potential problems with lagging indicators:

• It changes slowly: no matter what I do today (within reason) I'll be the same weight tomorrow. It's hard to know if I'm actually on track, day-to-day, by looking at my weight because it's a lagging indicator. I.e. I have to do all the right things first, and then the results will come second. As anyone who's ever drunk a glass of water knows, you can do all the right things and still end up with a higher number on the scale the next day (i.e. there is noise in the metric).
• It might not be the 'right' metric: do we care about weight? The answer might be yes but for a lot of people what they care about is their level of health, how they feel day to day, whether they can do all the things they want to do, and how they look. There's a pretty strong argument to be made that weight isn't actually what matters, even if it's a decent proxy in a lot of instances (see later).

Tech delivery

If the simple example of weight loss already made us doubt our lagging indicator then Tech delivery is going to be way worse. Is the lagging indicator actually about speed? Is delivering new stuff to customers daily necessarily a good metric that you're doing the right thing? I could build an app and change it every 15 minutes, but it would still be rubbish.

Speed of delivery isn't a lagging indicator, it's a leading indicator (see below). What matters is how much value you deliver to your users, and that's your lagging indicator. The more value you deliver, and the earlier you deliver it (i.e. the longer it's in their hands) the better you're doing.

The problem is that's not something we can measure directly, and that's where we come on to leading and proxy indicators.

Leading indicators - how we think we can drive change

Leading indicators represent a hypothesis. They are a hypothesis for how we think we can drive the actual lagging indicators that we care about. Again let's think about the examples:

Weight loss

Leading indicators here could be, for example, the number of healthy (vs unhealthy) meals consumed in a week, the number of days we stayed below a certain calorie count, or the number of minutes exercised in a week.

Each of these encodes an idea about how to achieve our weight loss goal. For example, by setting the number of minutes exercised as a leading indicator, we are making a (reasonable) hypothesis that doing more exercise will cause us to lose more weight. The point is our goal here is not to do more exercise, it's to lose weight.

Leading indicators like this are great because we can typically impact them day-to-day: I can make my number of minutes exercised this week go up immediately by heading to the gym or going for a walk. However, there are a couple of pitfalls to watch out for with leading indicators:

• They're not actually what matters: a leading indicator isn't the change we're trying to drive. It's vitally important to test the impact of the leading indicator on the lagging indicator over time. If the latter doesn't start moving, we might have the wrong hypothesis.
• They can almost always be overdone: leading indicators almost always work initially; if you're not exercising and you start going for a 10-minute daily walk, it will help you lose weight. But rather than just being something you should maximise, there is typically an optimal value for a leading indicator and going beyond it at least leads to diminishing returns, and may even set back your progress. Here for example, if you already exercise 90 minutes every day is it going to be beneficial to increase that further instead of, for example, spending some time preparing healthy meals?

Tech delivery

The world of Tech, and in particular agile, abounds with leading indicators. Change lead time, deployment frequency, and change fail percentage are just some examples that I've taken from the DORA website.

Like above each of these is a hypothesis for how to deliver what matters: value to customers. And just like the weight loss example, those hypotheses are sensible and backed by a lot of research. Most organisations will deliver more value to their customers by improving these metrics, particularly if these lead metrics are currently poor. But they are not a measure of what matters, they are just a well-established way to get there.

The question then becomes: does this matter? I would argue the answer is yes, it does. Fully abstracting away the thing that you're trying to do, can eventually lead to the wrong behaviour. I want to come back to this in the 'Does this matter?' section below, but let's quickly cover proxy indicators first.

Proxy indicators - when we can't measure what matters

Ideally, we would just measure the lagging indicator directly, but sometimes that just isn't possible. Maybe it's a nebulous concept that can't be measured, or maybe it's just not realistic to ask your users to spend more time filling out feedback forms than interacting with your service. This is where proxy indicators come in; they are the closest we can get to measuring the lagging indicator.

Weight loss

We've already covered this above. Weight loss often is not what people care about, it's health, how they look, etc. Weight therefore isn't a lagging indicator for a lot of people, so much as it's a proxy indicator. For most people if your weight is going down that's a proxy indication that you're becoming healthier, etc.

Tech delivery

Similarly, it's very hard to accurately measure the value being delivered to your customers; doing so is at best slow and difficult, and may even be impossible.

We use proxies here too: CSAT is an example and upsells and new customer recommendations are others. Even these are imperfect, but hopefully, it's clear that if your CSAT is high, your customers are buying more and more from you, and are recommending you to others, then that's a strong sign that you're delivering value. It's certainly a much stronger signal than how often you deploy software.

The point here is there's absolutely nothing wrong with a proxy indicator: they are almost certainly far more valuable than a leading indicator. But it's vital to remember that an indicator is a proxy and that may have implications for how to interpret the value.

A final word of warning: it's tempting to collapse indicators into a summary value, such as an average, particularly when communicating with senior people. A lot of information is contained in the shape of a distribution, and representing this shape can make sure that information is conveyed while still simplifying a lot of the details. Think for example of the difference between the following two distributions of CSAT scores with the same average:

• A single, symmetrical distribution around the average value
• A 'two humped' distribution, where end-users either have a very high CSAT or a very low one

Hopefully, it's clear the actions to improve these two situations are very different, and that is only evident from the distribution. Putting some outline distribution shape next to the average helps senior people to understand these differences.

Does this matter?

The obvious question is: this all sounds pretty academic so far, can't I just go to the gym and ignore all of this?

Well basically yes, and this is where the weight loss example breaks down. I've left it in because I think there's value in grounding this in something more realistic, but I'm well aware the analogy has long since run its course. If you're reading this, eat better and exercise and you'll be fine.

But in the more realistic business context, this does matter, particularly when it comes to the world of technology. There's a strong temptation to reduce everything to leading indicators and (maybe) proxy indicators: is our cycle time going down and our revenue going up? If so, then all good.

But you lose something in abstracting away what matters, which is delivering value to your end users. A recent(ish) example that prompted some of this thinking was the Adobe action from the FTC.

In a very brief summary, Adobe is under investigation for not making it clear what cancellation restrictions were in place for particular subscription types, or how much those cancellations would cost. I have no insider information but this feels to me like an internal team optimising the subscription process around metrics related to signing up new customers (leading indicators) and/ or revenue generated (proxy indicators) to such an extent that they forgot what a terrible customer experience it is to be tricked into buying something you don't want. Putting the termination fees somewhere hard to find drove the improved metrics, so that's what they ended up doing. If they'd focused on the ultimate goal: to provide value and a great experience to their users, then I have to believe somewhere would have had the sense to do something different.

Things that matter are often very difficult to measure. Tech people love numbers so it's incredibly easy for them to abstract away those things with proxy and lead indicators, which can be tracked on a spreadsheet much more easily. I'm not suggesting that these lead and proxy indicators aren't useful, just that it's incumbent on everyone, particularly leaders, to constantly reiterate that these are not the things that matter. They are, at best, a means to an end (leading indicators) and a directional signal (proxy indicators).

This leads me to reference Steve Jobs. He realised that there was more to technology than just "feeds and speeds" and by doing so helped transform Apple. What he delivered were products users loved, which is much harder to measure, but was actually what mattered. How fast it ran, how many pixels were on the screen, or anything else, were means to an end, but not the end themselves.