Avoiding the Pitfalls of Vanity Metrics

In this presentation, our CEO and Co-founder, Janna Bastow, talks about why vanity metrics are a blight on your business. So, how do you avoid these and other pitfalls that will lead your company to doom?

Avoiding the Pitfalls of Vanity Metrics from Janna Bastow

When we talk about vanity metrics, we mean that pile of numbers that you and your team are reporting on. But that don’t have any real bearing on the actual health of your business.

Looking beyond vanity metrics

It’s important to look beyond just the vanity metrics themselves, which are simply a symptom of underlying problems. Look at the root of the problem. 

When you explain the concept of timeline roadmaps and why they are bad for business to individuals, the individual gets it. They are on board. But when you try to convince an organization to change its ways, all you get is resistance. Why is that? What’s going on in large organizations?

Not all organizations are the same, but you can think of them all along a continuum. Are you nimble and lean, or slow and risk-averse? Perhaps you are discovery led, or are you mostly just there to capitalize on the market you’ve captured? Are you experimentation driven or ruled by vanity metrics?

If you’re an enterprise, particularly if you’re publicly traded, you have a fiduciary duty to increase value for stakeholders. If you’re funded, you’ve got similar commitments to your investor groups. For a lot of those companies, the ideal state is a steady up-and-to-the-right curve of their value. Just as important is predictability, as having that line swing up or down unexpectedly can spook investors and shareholders, and can actually result in CEOs getting fired.

Avoid operating in siloes

The easiest way to get a predictable growth path is to break it up into smaller chunks. This seems to make total sense until you realize that it’s the exact reason that companies get broken up into siloes. 

In those siloes, you end up with teams creating vanity metrics and even competing against the heads of other silos. This, in turn, results in counterintuitive business decisions being made. One division might make a change that favors its metrics but hurts other areas of the business. These types of misaligned incentives can spell disaster for the company as a whole.

So what’s going on in these siloes? Any division in a company is usually thought of as a profit center or a cost center. Sales and marketing are profit centers: you put money in, and additional profit comes out. HR and recruitment is a cost center: you need it in order to run your business, but it costs money and doesn’t provide additional revenue. What does that make your innovation teams, like product, UX, and development? We’re a cost center, and it’s created a culture where the CTO has been forced to create vanity metrics for the division in order to measure and control costs per output.

For example, if you’re working to increase velocity, burn down your burn-down charts, and you count your work in story points, you’ve optimized for the output of features. You’re looking at the output, rather than focusing on business-level outcomes. A lot of the time, execs will look at a timeline roadmap and simply believe that in order to build all those features, they just need to throw more developers at it to get the output they want.

A bunch of features doesn’t always make the most compelling product

Everyone knows this. This way of working leaves no room for discovery, no room for experimentation, and results in stressed-out developers and product people and the accrual of excessive technical debt.

It happens in sales and marketing too! If you mis-incentivize your marketing team, they will simply load up whatever sort of leads they find, rather than finding the right sort of leads. It passes the buck to other divisions who will then have a harder time converting and retaining those customers.

It helps if you have a north star metric: A guiding, overarching goal for your company. 

It’s not uncommon for teams to accumulate stats and metrics. Remember that less is more, and cut down on these reports. Figure out what numbers actually relate to this north star metric, and ditch the rest.

The thing with that slow and steady growth curve: it’s a trap! Every product follows a product lifecycle that starts with relatively fast growth which then tapers off at maturity. Many of the largest companies are growing almost imperceptibly slowly.

While it might look like it’s constantly climbing upwards, it’s optimized to grow to local maxima, and never find the true maximum. There is always a better market, a new opportunity, and therefore a bigger true maximum. In order to find these true maximums, your organization needs to be willing to break out of the one they are optimizing for. Unfortunately, this can be uncomfortable for large, risk-averse companies that are beholden to their shareholders or investors.

Embrace change

And yet the companies who constantly innovate and reinvent themselves are taking over. They are massively outperforming indexes around the world. It works!

It’s essential for survival as well. Every company that ever did or does exist will one day go out of business. It’s just a matter of when. With so much change in the world, companies have to be adaptable to survive longer than their competitors.

It’s important to build experimentation into your processes. Make sure you build in time and space for the validation of those experiments. Change the way you talk about experiments. Don’t make commitments on how many features you’ll build as a team. Make a repeatable process around how many experiments you can run.

Ditch your feature-focused timeline roadmap, and ditch your siloed vanity metrics.

And learn to speak like an executive. They don’t care about the customer journey and the delightful user experience. They think about the risk and rewards of the business. Show them how a culture of experimentation and being adaptable is good for business. Now, this is the best use of their investment.

By doing so, you can be a Netflix in a world full of Blockbusters!