Hot Take – Measuring to Learn vs Measuring to Control

Measuring to learn vs measuring to control is a fascinating way of thinking about metrics for your organization. Janna Bastow, Co-founder and CEO of ProdPad, is on a mission to demystify the popular trends and schools of thought in her new video series Hot Takes.

In the first installment of Hot Takes, Janna talks to John Cutler, Head of Product Education and Practice Research, at Amplitude about measuring to learn vs measuring to control. John trains teams on the best practices of product management. He has written extensively on the matter over on the Amplitude blog. Watch the video to understand the difference between measuring to learn and to control. You might end up with a craving for a bagel or two…

Read the full transcript below.

Janna Bastow:
Hi everybody, I’m Janna Bastow from ProdPad and I’m here with John Cutler who is the Head of Product Education and Practice Research at Amplitude. 

And we’re going to be talking about measuring to control vs. measuring to learn, So, John, really interested to hear your take on that. 

John Cutler:
Yeah, I’m super excited to be here. I’m reminding myself of breakfast, I’ve got the bagels in the back, and you’re reminding me of like a sunny afternoon so, yeah, I’m super excited to jump into this.

Janna Bastow:
Okay, so we’re talking about measuring to learn, vs measuring to control, which I thought was a really interesting topic and I thought sort of jived with some of the stuff that I’ve talked about in the past, but I’d love to hear from you why you think this is a difficult topic that product people need to hear about. 

John Cutler:
I think that the problem is how ill defined this is. So,you’ll talk to teams about frameworks… Actually I’m just off doing a three hour workshop and I think that, up until the last five minutes, someone thought that I was trying to describe a way to measure their team and decide whether their team had done a good job or not. And that’s how hard this can be for some people to grapple with. The leader was actually on the call and they’re saying “no, this is going to be great and we’re going to use OKRs and we’re going to use this North Star thing…” and it makes sense to me and I think I got what the leader was saying, but part of their team, for a good two hours and 55 minutes, were sort of cowering in the corner of the Miro board wondering what did this mean for them and would it mean that their team would make or not make their particular goals. 

I think it’s a very, very important thing to talk about because, especially at Amplitude, in talking to teams, you see teams terrified to measure anything because they think they’ll get it wrong, and if they get it wrong they won’t be able to prove something 100%, and if they can’t prove something 100%, then somehow the metrics are dangerous and that they are going to be used to backfire on them. So it’s sort of about metrics of use. 

I think it does have to do with when you’re measuring to learn, you’re sort of exploring uncertainty and you’re hoping to just reduce it by a bit and be asking better questions tomorrow than we did today, or today vs yesterday. 

And when you’re measuring to control or goal setting, that’s when you get the things like “by the end of this quarter, we think we’re going to move this metric from here to here”, or, “our team’s particular thing is to do that”. 

It’s just two different mindsets when it comes to measurement and so I think it’s important to distinguish the two. 

Janna Bastow:
Do you think that this measuring to control thing is sort of  more of an old school mindset that you’re finding some companies are taking on?

John Cutler:
Maybe, but no, I wouldn’t say… and using the word ‘control’ probably sets it like that but I would say that, you know, let’s say that you’re a team, I mean I’ve done this when I was part of a startup, we’re like “hey, we should have some goals, it’s going to help hold each other accountable, it’s going to kind of keep us focused” and it came from the healthiest place. So I don’t think there is something inherently wrong with goals, right, or trying to add some control or alignment to what we’re doing. Now certainly in larger organizations this just becomes a stick to beat people with, it’s like no one… let’s just take something like NPS and let’s just say no one in the company believes in NPS except the person who dreamed it up, like they were going to use it three years ago. And you’re actually seeing this out there, that like teams are measured on their ability to move a metric that no one really believes in and people have a lot of problems with, and that’s that kind of like… so I would say that measuring to control ranges from “hey, we’re in this together, we’re a team, let’s set some goals. Or even, like, you know, I told Sharon upstairs who’s taking care of our baby, that I’m going to exercise for ten minutes a day, that’s my goal, and it comes the best place. So there’s a spectrum of that, from that all the way to, you know, very, very malicious use of measurement. And then the measuring to learn is it’s almost like, sort of separate idea about what we’re trying to learn, you know, are we trying to learn more about what our customers value, or are we trying to learn more whether our strategy is clicking the way we thought it was. And the whole idea there is it’s very open to challenge and open to refining, you know, and I think that’s a really important point. 

Janna Bastow: That’s a really good point, and I see that this happens to so many measurements that we create, and I say “we” in term of, like, the whole sphere, the whole industry, and we tend to create these things. Like KPIs I think were created from a good place, but then were used by management as this stick, and people just shied away from them. And then OKRs became the new thing, and now they’re starting to get a bad rap here and there because they tend to be used in the wrong way. So how do we make sure that this concept of measuring to learn doesn’t get turned into something more insidious?

Book yourself in for a demo with one of our product experts. They’ll be able to help you with your product strategy and show you how ProdPad is perfect high-flying product leaders.