Avoiding the Feature Factory: How to Measure Product Success

Why do we need to measure product success?

We may keep hearing the phrases “feature factory” and “build trap”, but what do these terms really mean and why is measuring product success now such a key part of the product management process?

For some product teams, the product roadmap is simply a list of features to build. Product managers act as gatekeepers – they make decisions on which feature the development team should work on next, provide the information necessary for the development team to pick up tickets and push features out the door. It feels really efficient when you’re doing it – after all, you can measure how many features/story points/tickets are completed and you can refine your processes to get quicker and quicker. And if that sounds like a production line, that’s because it is.

A production line is great for creating repeat copies of one thing. However, your goal when developing products isn’t to repeat previous work more efficiently, it’s to improve your product and make it more valuable. Measuring output isn’t relevant to a product’s success – you need to measure outcomes, and have confidence that you’ve solved the problem you intended to solve. 

Rather than a “build, build, build” mode, you need a “build, measure, learn” mindset, to give yourself time to go back and improve what you’ve done and get the best return on your investment.

What should product management success criteria look like?

The concept of outcome measurement isn’t easy. You can’t use the same metrics each time, as measuring the outcome requires an understanding of the benefits to be gained when the problem is solved. To truly measure outcomes, you should define success criteria before the problem is solved, by asking “how will we be able to see when the problem is solved?”. 

The way you measure success may be different depending on the granularity of the problem. Let’s consider an example like having a high amount of churn/cancellations. This is a product-level problem, and is easy to measure. There are lots of tools available to allow us easily to keep track of the number/value of customers who cancel each month. However, the ways in which we solve the problem are less defined – it could be more educational tools, in-app help, improved onboarding or a myriad other options. If we decide on educational tools, there are multiple options, such as videos, help articles, or live chat. We need to measure success at different levels. In ProdPad, we would represent this by creating a roadmap that is structured around objectives and key results, initiatives and ideas.

Objective

This is the objective (also called the OKR). We can add information over time to see whether our various efforts to address churn are working and it’s moving in the right direction. It’s the bellwether.

Initiative

This is the initiative. It is a problem definition, linked to the OKR so we can see that the work we are planning on our roadmap is helping us to achieve our company objectives. We can also see how we’re measuring success for this initiative in particular (other initiatives could be linked to the same key result).

Ideas

And finally, this is one idea for solving the problem represented in our initiative. There will be lots of ideas we could try, and each will have different success criteria, as we’re building different solutions. We might try one or many ideas in the process of solving the problem. The idea is really an experiment, something which we’re trying in as minimal a way as possible, in order to learn what works and what doesn’t.

In each case, we’re measuring the success of what we’ve defined, but the level of granularity is different. For ideas, we check to see if the change worked as expected. For the initiative, we check to see if the change solved the problem. And for the objective, we’re measuring to see if solving the problem helped us to reach our objectives.

Managing the process of measuring success

As there are different levels of granularity in the measurement of success, we also need to find different ways of recording the outcomes. Much of this will depend on how frequently changes are released. If you release small changes daily or weekly, you should check on their success on a very regular basis, and maybe keep an eye on how well you’re solving the bigger problems quite regularly too. 

It’s likely that you’ll want to check progress towards your OKRs on a monthly basis. If your release frequency is much longer, then success measurement is a less frequent process too. It’s why most companies aim to release small and often, so they can see the effectiveness of the changes they make. If you only release once a year, and can’t measure success along the way, how will you be confident that your changes will work?

There is always space at every level in ProdPad to record how effective your bets were. This starts at the idea level.

You can record the success of your initiatives…

…and whether your OKRs are on track, behind, or at risk.

Why measuring product success makes you a successful product manager

The most effective product managers learn from their successes and failures, learning from failure and iterating until they reach success. A short feedback cycle and breaking the work down into the smallest valuable piece (even if that’s an experimental prototype) allows a PM to learn quickly and offers them the best possible chance for success. It’s much better to learn how something doesn’t work at an early and inexpensive stage than when a major feature is released!

When you’re considering how you can be a successful product manager, and how you can help your business succeed too, there’s really only one answer – the build, measure, learn formula that is integral to ProdPad. Give it a try and see for yourself how it prevents product teams from falling into the “feature factory” trap.