Product Management Webinar: Sunsetting a Product or Feature

Janna Bastow: [00:00:00] Hi everybody and welcome. This is our product expert fireside series that we’d run here at ProdPad.

For those of you who’ve come along before, we run these things on a monthly basis and have done for years, so it’s a series of webinars. It’s a mix. Presentations and fireside talks. Tonight’s going to be a fireside. And they’ve all been recorded. So you can head to ProdPad.

com slash webinars and see the archive of all the different conversations that we’ve had. And it’s always with a focus on bringing in amazing experts and just. Bending their ear on their insights to focus on the content and the learning and the sharing. So today is going to be recorded and you will get a chance to ask questions.

Feel free to use the chat or ideally make use of the Q& A section. That way, it’s easier to see who’s asking what and you can all chat. Up vote each other’s questions. And that makes it easier to keep a little bit more on top of them. But make use of the chat today. If anybody wants to share where they’re [00:01:00] from their LinkedIn network, their fellow product people here, then do.

So. Before we kick off, I wanna share a little bit about ProdPad. You all probably know me as one of the co-founders and the CEO of ProdPad. ProdPad is a tool that myself and my co-founder Simon built when we were product managers ourselves. We needed tools to do our own job and this sort of thing didn’t exist.

And it’s a tool that was there to help us keep track of all the different things in our backlog, all the experiments, all the feedback from our customers, and it helped to give us control and organization and help to give people in the team transparency about what was going on in the product management space.

So that’s, people didn’t get the sense that it was this place where ideas went to disappear and die. And it becomes this single source of truth for all your product decisions. Everyone knows why things are being built, what’s on the roadmap, what’s coming up, how it’s connected to the goals for the business.

And it’s now being used by thousands of teams around the world. So you can try it for free. [00:02:00] We’ve got a free trial. We’ve got a sandbox version as well that you can check out. And then that sandbox version, it’s got example data like lean Now-Next-Later roadmaps and OKRs and example information that you can use and move it all around, see how it all fits together and our team is made up of product people.

We’d love to have you give it a try and then, jump in and let us know what you think. We’d love to get your feedback on it. Speaking of stuff that we’ve been doing with it we’re constantly iterating and shipping new stuff. As you probably know for anybody who’s already using ProdPad, we have a Slack integration.

That’s really powerful. It allows you to capture feedback and ideas directly from Slack, and then allows your team to comment on those feeds, those pieces of ideas and that feedback and whatnot. And it all syncs with ProdPad automatically, which is really powerful. So just keep everybody in the loop and your team can use, voting like dislike buttons on it as well.

But the news is that we are this close to launching the teams. I don’t want to give an exa I don’t want to jinx [00:03:00] it, but we’re now literally days away from this getting out there. So if anybody’s using Slack or now Teams you can connect ProdPad to it. The other cool thing is that the Slack integration is connected with our ProdPad CoPilot.

And CoPilot is very much in the cloud. beta right now. It’s our AI tool that we’ve put as an underlayer under ProdPad. And what it’s basically allowing you to do is ask it questions or work with it as if it’s like a sidekick or a coach. So ProdPad has access to, what your big vision is, what your goals are, what it is that you’ve Wanna build, what you’ve tried to build, what’s worked, what hasn’t worked, what your customers have asked for.

So you can ask questions like, can you summarize that feedback for me over here and tell me what’s on the roadmap that might match it. You can even give it images. You can say, Hey, here’s my roadmap from last year. Can you turn it into an Now-Next-Later roadmap for, and put it into ProdPad and it’ll actually generate it and put it into your ProdPad account for you.

So some really powerful things you can do. We’re actually looking for beta testers on [00:04:00] this right now. So if anybody has any particular use cases that they can think of that they’d like to test with this, let us know what those use cases are, or let us know and get onto the beta program for it.

And you’ll be first to see what it’s capable of. So enough about us and these cool features that we have been building today, we’re going to be talking about how and Why and when to kill a product or feature. And this topic is really dear to my heart because it’s actually one of the first articles I ever wrote for Mind the Product was how to kill a feature.

And that was going back. That would have been 2011. So when I was a baby product manager, basically, um, so I’m always really passionate about refining products in this way. But here to talk to us about it today is Stephanie Massat. Now I know Stephanie because we both spoke on the how to web stage last year in Romania.

She did an amazing talk. So Stephanie is a product leader with [00:05:00] over. 12 years of experience crafting user focused strategies for top media companies, like you would have heard of CNN, Refinery29, and now Warner Bros. Discovery. She’s the director of product management. She specializes in connecting users to the stories they love through smart navigation, creative campaigns, and top tier design.

And beyond her role, she’s an educator teaching future product managers at Northwestern University’s Kellogg School of Management and through Mind the Product training. So today she’s here to share expertise on making bold strategic decisions, including how and when to let go of products and features that no longer add value.

Everybody say a big hello to Stephanie. 

Stephanie Musat: Hi, everybody. And thank you for having me. I’m happy to see you again, Janna. 

Janna Bastow: Yeah, so good to see you and thanks so much for joining. Great to get a chance to chat with you about this. And thanks for coming in and talking about this. We actually talked about this topic last year.

It was last year. It seems like ages ago. I know, 

Stephanie Musat: It feels like ages ago. 

Janna Bastow: [00:06:00] In Romania. 

Stephanie Musat: Maybe a couple of months after this and just confirm, but yeah, about a year. 

Janna Bastow: Yeah, and I remember it was just before Halloween because you were talking about killing features and something. Yeah, 

Stephanie Musat: spooky, scary.

There’s some horror theme that I did. There’s some 

Janna Bastow: horror theme behind it, yeah. 

Stephanie Musat: Something like, I don’t know, some sort of 

Janna Bastow: conference 

Stephanie Musat: talk. Yeah, we’ll work on that. 

Janna Bastow: What inspired you to focus on the topic of setting products and features? 

Stephanie Musat: When I first came up with this and to your point, I’m not the first person to talk or speak on this, but I had a couple of instances right before I did my first talk on this.

That was the realization that people need permission in order to do this. And it was initially conceived as, this isn’t as taboo as we think it is when I was initially going through my own product practice and beginning to deprecate Well, features it really felt like I was violating all of the principles of product management we build for customers.

We ship fast. We have conviction in our ideas, but the reality is our products [00:07:00] have capacity and oftentimes when we add and continue to add without evaluation without that, that more unbiased view. We end up with bloated, big projects that just don’t do much for the end consumer goal, for our business KPIs, whatever end need is.

So I was faced with this moment that had this hard truth and fairly hard choice, basically coming down to the fact that somewhere I, and we as a team made a decision that was wrong. That kind of led us to develop something that was either not in the best interest of our users or the business overall.

Or maybe that it did serve a purpose once upon a time, but we’ve evolved and matured past it. And all of that came down to both an acknowledgement of the emotional state that you’re in when you’re in this place, which is that you spend a lot of time and energy and effort building something. In fact, oftentimes you’re the one

In [00:08:00] that priority or with the evidence and its worth. And so you put a lot of energy, thought, a lot of like, your own personal identity into something like this, and then it doesn’t work. And in a moment at that, you really have to swallow your pride. and think about what is the intention of your function as a product manager, the job you’re being hired to do.

And the reality is that might end up being something like deprecating or removing that product from your larger set. 

Janna Bastow: Yeah, absolutely. And actually you speak about the emotions tied up in this. I ran into this earlier this week, so we have somebody new in our product team. And we’re talking about a feature that we trained on our product team or had a product who has joined.

And we were in a product update meeting and we were talking about a particular feature that we introduced, but hasn’t been successful. Right? It wasn’t fully baked and it hasn’t been adopted in the way that we would have hoped. And actually, we looked at it and went, let’s just take this out, and focus on things that are really going to drive home [00:09:00] value.

And the question came up of how do we take it out, right? What do we do in order to prepare for this? And I said, look, we can just. Display none, just get it out of there, right? There’s not enough people really looking at this thing. And she went, really? That’s the type of thing that would have gotten me fired at a previous job.

And there was this fear that this might happen to her. And I was like, Oh, no, that’s not going to get you fired here, right? Obviously, we need to do things with consideration, you wouldn’t just simply, Hide the roadmaps but, if it’s something that isn’t getting used and, it’s going to take more time to devise a strategy to, to, bit by bit, take it down, then just take it down and move on to the things that add value.

Stephanie Musat: Yeah, definitely. There’s unfortunately some companies that justify the existence of a product by velocity, how many times you’ve shipped something in a given month, quarter, year, whatever it is. 

Yeah. 

And that always made me a little uncomfortable, because that does cloud the decisions that you make, because you’re then [00:10:00] going for what is fast and loose, maybe, and you’re sure, maybe they’re small ideas, but the desire of the MVP in that case really gets a little twisted, because you’re then operating in a model where.

Your success is tied to volume, not impact. And if you were to sit down for a second and say of the 17 different things I just shipped, what’s working. And if the answer is zero, then all you did was really waste money, resources, because they’re paying your salary. More importantly, they’re paying the engineering salary, which often is a pretty significant cost for the company.

Janna Bastow: Yeah. 

Stephanie Musat: You build something that has a bottom line for the company. So if you say no, I’ve spent six months of engineering salary on something that didn’t work. That to me is actually riskier than saying, no, I took a minute. I thought about this. I tested my strategy. I thought I had a little bit of a bigger thought and that you’re not then continuing investing in something that is just not going to work.

So it comes both in the pre stage [00:11:00] as you’re thinking about what do I actually want to bring to market? And then the post evaluative stage where you have to be pretty ruthless in this case and honest about what is working and. Part of the issue there is the attachment to our own solutions, because we’ve come up with it, that we’re bought into it, and frankly, it’s often disguised as our own brilliance, right?

We’re so smart, we’re so great, we have this. Really clouds the decisiveness and reason to make good choices. So if you had a third party company, would anybody come in and be like, okay, well, what’s actually working, what’s not? You wouldn’t be able to justify why you keep certain things, because it doesn’t have either.

user need, the business bottom line, or a ladder up to your OKR. So it’s not a comfortable place to be, but the alternative is you’re working and investing company resources on something that just really isn’t helping. 

Janna Bastow: And you’re absolutely right. And I think some of it comes from the fact that in the past product management and development teams have been measured and incentivized based on [00:12:00] additive measures.

Burndown charts, velocity, how many features, how many story points you can ship? Can you ship this many story points in the next sprint, next quarter, try to ship more story points? And what are story points? Well, they’re basically, like features, right? Just the estimations have gone a bit squidgy along the way.

And so it’s about how many features you can get out the door, but everyone knows that more features doesn’t mean a better product. And yet that seems to be what product people are measured on. And there’s not really any reward for taking off a feature, right? You don’t really get that same sort of like, Hey, this quarter I took out five features.

I 

Stephanie Musat: don’t want to be the one going to a demo being like, so that thing that used to be there now isn’t anymore. Like it doesn’t, so being like, Oh, we get to show our work. No, it’s a little sad that way. I’ll give you that for sure. 

Janna Bastow: Yeah, this is why I like to reframe stuff that goes into the backlog.

We call them ideas in ProdPad, but actually I call them experiments when I end up talking about them, right? Because an experiment is [00:13:00] actually a broad term, right? An experiment could be, let’s add a feature. Right? By adding this feature, doing this MVP, whatever it is, we can learn this, and it can move this needle from here to here.

Cool. But it could be things like, let’s change a feature, right? You don’t need to continuously add stuff in, but it could be things like, let’s take away a feature, right? That is a perfectly viable experiment that might solve the problem at hand and increase the number from here to here, whatever it is you’re trying to do.

And a lot of people don’t think about removing features as viable experiments, just as viable as adding a feature, right? If you’re trying to improve usability or speed up onboarding or something like that, taking something out might be the answer. 

Stephanie Musat: For sure. I think you hit on something that resonates with a lot, which is that our process orients toward new development by default. There’s always this question when you’re doing quarterly planning or roadmapping of what are we going to build next without it being necessarily focused on what exactly is that problem statement. And so to your point, and I know product [00:14:00] has a lot of features around this, but this idea of being able to ladder it up to a If something is, it takes too long to do this.

The answer oftentimes is not build something new, it’s cut stuff. And so if you’re really specific on the problem statement that you’re after, and you have conviction that is going to be the thing that ends up laddering up again to those objectives you have, then the roadmap itself and all those questions you ask don’t necessarily have to be Yeah, what are we adding?

It can be what are we removing, what are we testing? All of the things that you just mentioned. But if you just go through the process today, if you were to go to, I dunno, any stakeholder or business leader, and say, actually what we’re gonna focus on now is removing things. They’re also gonna freak out, right?

Because it’s like, wait, we have all these people, what are you doing? 

Janna Bastow: Yeah. 

Stephanie Musat: But it might end up to your point, solving the problem even better. 

Janna Bastow: Yeah, absolutely. So how can you balance that emotional attachment to these features with data driven decisions? 

Stephanie Musat: It has, there [00:15:00] has to be data into this because what, and the data can be quantitative and qualitative because the actual result of the build trap that we end up in ourselves is pretty anti user.

It’s the exact opposite of what we’re trying to do. So if you lead by emotion, then you’re focusing on solutions and not problems. If you’re scared of failure, then you don’t have a realistic goal. of whatever you’re doing. And if you don’t necessarily change your process to accommodate this, then the cycle continues and perpetuates.

So it’s definitely a bunch of things. So data, for sure, is the top one. If you don’t have users using this, if you don’t actually see the KPIs that you had anticipated with this feature, if you don’t get the revenue or the growth or whatever ends up being that overarching goal, then that gives you a moment to really consider if this is helping.

I think that this becomes a really uncomfortable conversation, don’t get me wrong, but that you have to be logical, open to that discussion. I ask a lot of questions around, is it more expensive [00:16:00] to maintain than it’s worth to the user? And I don’t necessarily mean, like, actual dollar money in that case, but if the infrastructure around it is also required.

we’ve got a lot of use cases, who are requiring us to build in a certain way, to operate in a certain way, more than the usability for that user, there might be an opportunity to think about removing something. Any time you’re encouraging the wrong behavior, so if you were trying to launch a new set of features or trying to assuage customers to use something else, but they’re constantly going back to whatever exists, yeah, it might, again, be painful in the short term, but worthwhile to long term to get them to pivot.

Are you stopping you from doing something more meaningful? Do you see other metrics in your product? Bloat, so scalability any sort of ways for scope bloat to begin with or product bloat to begin with. Is it too slow? Is it uncomfortable to use? Can you defend this decision? Like, there’s many different questions you can ask yourself.

Janna Bastow: Yeah. 

Stephanie Musat: The longer you wait, the harder it gets, I think, [00:17:00] is whatever sort of the assessment. And we can go through certainly any of these questions. But the hope would be. The more that these sort of percolate in the back of your mind, why am I spending so much time on this thing if no one uses it? Why are we investing in it more when we should be doing something else?

Those are little, those make my ears perk up a little to be like, okay, wait, do we need to keep doing this? Can we talk about ways to go around it? 

Janna Bastow: And that’s actually a really good point. You touched on something there. ’cause it’s not just lack of usage. That is a reason to take a feature out.

Obviously if no one’s using it then that becomes a great, turn it off. Right. But usually it’s like there’s not enough people using it, but sometimes it’s things like, it wasn’t done properly and it needs to be, it’s easier to take it out and then rethink it and do it again properly later.

Yeah. And sometimes it’s like, you know what? We went in this direction and it’s not aligned with. Where it is that we see ourselves going now. 

Stephanie Musat: There have been a lot of times I’ve been put in positions where we get a lot of, it wasn’t built like that, or it wasn’t really meant to [00:18:00] do that. And then you start to try to see if you could build it up and then you can have shaky architecture underneath these overarching product features.

Then the mat perpetuates the problem longterm. And so the question is in that case, why don’t we deprecate the service and build it right, or build it with scalability in mind, or these kinds of things. Anytime you hear it wasn’t built for that is a pretty clear indicator. Maybe you shouldn’t keep building it.

Or maybe you need to think about a different way to use that underlying tech because again, you build on top, you build on top. It becomes the core of workflows and ways that you distribute your company or your business, but it also becomes very hard to untangle. And again, the longer you wait

to build it with area, you’re kind.[00:19:00] 

Janna Bastow: We might be getting a bit of delay on your side there. 

Stephanie Musat: Okay, I’m back. Can you hear me? I 

Janna Bastow: lost you at the longer you wait. 

Stephanie Musat: Oh, what a, 

Janna Bastow: It was a dramatic moment as well. 

Stephanie Musat: Wow, I didn’t script that, I promise. I said the longer you wait, the longer it is harder to untangle, and maybe I pontificated for a moment longer, but hopefully I’m back.

Janna Bastow: Excellent. All right, thanks. How do you handle pushback from users or internal teams as well. When sunsetting something. 

Stephanie Musat: The internal teams, I think, are harder than your external teams, frankly, or your customers. May end up remarking that they miss a feature or wish they had it, but if you’re in a B2B situation, you’re often giving them something else to then use in its place or some sort of concierge service to help them make that adjustment.

Internally is where you get again, those emotional spikes around, Oh, well, my team asked for this, or this is critical to our [00:20:00] particular Bottom line, or this is what my team needs, and that is where the investment again versus the labor, the overhead costs, the time, the onboarding, the marketing, all of these things become really interesting.

And I think that there’s a way for you to weave both the softer side of this, which is being considerate and thoughtful and understanding to the fact that, again, we’re. Beholden to business bottom lines and objectives here, 

and if 

we can easily say that, hey, this didn’t do the thing we need to, we can help solve your problem elsewhere.

It’s in some ways the same negotiation you go through once you are trying to get someone’s idea onto a road map. It’s the same sort of flip side of that where you’re like, Okay, it’s here. It’s not working. What else can we do? Let’s remove this and get you something that actually is effective. It’s not an easy conversation.

And again, people will come pretty hyped up. There’s always a lot of energy. It is uncomfortable. But you shouldn’t also accept the fact that People want things for the sake of wanting things and [00:21:00] however you end up considering that conversation is probably a little bit different depending on the team, but the goal still maintains that you have X amount of capacity in both your engineering teams and your delivery teams as well as your feature.

There was some stat many years ago that said something like 80 percent of Users only use 20 percent of features. 

It’s like, why are you operating or working on that other set of features if no one’s using them? Well, using that as a compelling way to also get people on board, identifying how many people use this, what are they using it for, that kind of thing becomes much more, a much easier pill to swallow.

Janna Bastow: Yeah, absolutely. Absolutely. And, there’s that take that if you launch a feature, no one uses it. Did you actually launch the feature? I think there’s actually there’s a, it’s a wonderful talk that was done at MTP engaged by Paul Adams of Intercom, and he went through, well, here’s what happens.

We do all this stuff to ship these features, but then, did it actually go through the steps to be marketed properly? Did it actually go through [00:22:00] sales properly? Did it actually get to the point that it was being used? Did it hit your targets? Right. And there’s so many features out there that end up just getting dropped.

And this is why you end up with that stat of so many features that just don’t end up getting used and don’t end up adding value. 

Stephanie Musat: Yeah, 

Janna Bastow: totally. And to your point of, we’re beholden to the bottom line or to the underlying business, the overarching business goals, that’s a nice throwback to the last conversation that we had on this show, which was about how to do growth.

product management, right? I think this is becoming so important in our roles, where it’s not just about us getting things out the door, but it’s about us understanding us as product people understanding what drives the business and what can we do to, help drive it forward towards that as opposed to being much more project led and just shipping a bunch of stuff out the door.

Stephanie Musat: Totally. And even on the growth side, the internal side, there’s other metrics you can think of that kind of help with this as even just a health metric. Things like, do [00:23:00] you have increased speed to market? Do you file less bugs? Like, I know that sounds crazy to think about, but if you are on legacy architecture engineering, or if you’re constantly building things that aren’t really up to snuff, you will have a lot of tech debt with that.

Can you see the benefits from consumer happiness? Or time to speed time to market? These kinds of things also help without being necessarily that hard, hey, you got growth, you got revenue, whatever it is, but all of a sudden these softer lagging indicators that what you’re doing is creating a more stable, usable product.

Yeah. I think one of the frame things that I personally work in, so I work on a pretty large scale direct to consumer streaming orgs is things around people using it more often. And if you think about the entertainment space you have your set of streaming services whether or not the Warner Brothers Discovery one, which is now called Max if you are in certain regions, it might be HBO Max if you [00:24:00] think about what you want to watch.

Now. As a product manager, there’s not much I can do to tell you, like, yeah, you want to watch this show or not. However, what I can do is make sure it doesn’t buffer every time you open the app. And that it feels pretty easy to use on the device you choose. These are things I can control because not only would you want to watch something, you also don’t want it to be painful.

And you can even notice when these things are painful. One of my old products from many years ago was an internal tool system, and it took something like 42 seconds to boot up. And I remember, One of my very first days there I was told to shadow one of the users, and I got to the office early, I sat down next to them, they opened their computer, they opened the app, and then they went to go get a coffee.

And they were like, we do this every morning because what’s more painful? Getting up and getting a coffee or watching this thing? It’s like watching paint dry. It was. Awful. 42 seconds when I say it here doesn’t maybe sound that painful, but if we were to sit in [00:25:00] silence for 42 seconds, I promise I would.

It’s uncomfortable, and that’s on us, entirely on product people, entirely on your engineering squad. Whether or not they use the product is a negotiation between product, users, marketing, all this stuff. If the thing doesn’t load, that’s something that’s a serious issue and that took us a long time to untangle and we did end up replacing it with a new service and it was one of these sort of deprecation conversations.

This thing just couldn’t keep up anymore. And in particular, this was servicing a bunch of journalists who were in the field. And I was with one of them at one point and the conversation was like, yeah, when I’m in the middle of America with lower bandwidth and hardly, spotty wifi, this thing is not going to work and it doesn’t work.

So what do they do. As they went around it, they would write out their articles in Gmail, email it to an intern, who then would upload it to the system back in [00:26:00] the headquarters. And it’s like, these are the things that, one, I appreciate creativity, so these people are going around the things that slow them down, good for them, but as The product manager on top of this project, it was horrifying to be like, Oh, you actually just don’t even use the product at all because it is so bad.

So these sort of moments of realization are the times that we have to take responsibility and really think about what we need to do next? And if the answer in this case was to get rid of it, that was absolutely the right decision. 

Janna Bastow: Yeah, absolutely. And speed is such an underrated unique selling point.

It brings to mind Spotify’s original strategy. So Spotify wasn’t the first place that you could get music online. And before then you could just torrent stuff, right? That’s what we were all doing. And there were lots of places where you could download music online, but their whole thing was that they minimize the amount of time.

When you press the button to say play, it would just play. It’s the one that’s a little bit weird, but it’s like a secret library. And I [00:27:00] think it’s really cool. That’s a magical library. It’s a magical library. It was really great. So that’s the 

Stephanie Musat: story behind it.

Janna Bastow: Yeah. So the reason why I wanted to do it is because why am I doing it? Why am 

Stephanie Musat: Am I doing it? And I’m not a journalist. I’m not a bookseller. I’m not a bookseller. The antithesis of speed is wait. Yeah. Well, you gotta strip stuff down, see what happens. That’s what I like to think about.

It’s less about what the user needs. Like, I wanna play a video, obviously. I get that for streaming. You wanna, you open my service to watch a show or a movie or something. What are the ways that product gets in the way of things like speed, buffering, can’t find what you’re looking for, don’t know where you saved it.

Those things just become irritating, and I’m in the space right now where I’m trying to make my app as enjoyable as possible, as delightful as possible, and that is in the case of sometimes making it as simple as possible. 

Janna Bastow: Yeah, absolutely. Absolutely. There’s actually a question from Dominic in the chat here.

Can you give an example of a feature that you killed and lead us to the decision [00:28:00] process of how you got there? 

Stephanie Musat: Yeah. And 

Janna Bastow: actually going to open that up for the audience as well. If anybody in the chat wants to share any sort of challenging feature or product that you had to let go of, let us know in the chat.

Stephanie Musat: I will tell you about one, though. I will also acknowledge that we’re not the only people who killed this feature. There was a time in streaming where everyone had this understanding of decision paralysis. You had too many options. There’s too many services. There’s too much content. And frankly, that’s still a pretty meaty problem space that me and my team spend a lot of time thinking in.

But in this case, we thought, okay, well, if you don’t want to make a decision here, let’s make it for you and just click a button. And it was in, Netflix had one, they called it Surprise Me. Google has it as I’m Feeling Lucky and theirs. And we had one we called Shuffle. It was called what, 

Janna Bastow: Sorry?

Stephanie Musat: Shuffle. Right, 

Janna Bastow: Okay. 

Stephanie Musat: And the idea of it was, like, you would click a button, and it would randomly pick something for you. It was [00:29:00] supposed to be really hearkening back to, frankly, linear television, where you didn’t really choose. You’d choose a channel, if anything, but whatever was on was what it was on. And the goal, again, was to reduce friction, to make it feel less committal.

We got a lot of research coming back where, if we don’t know the title or the actor in the poster or the synopsis wasn’t really clear. All these things are drivers of a decision in a streaming service. Well, if a user didn’t see someone, they recognized a title that they knew, or if they didn’t enjoy the description on the page, they wouldn’t click onto it.

So the idea behind Surprise Me on Netflix was essentially click a button, play something. Turns out people still want a little bit of control. The autonomy piece is here. It’s like that desire to have a less friction decision process is still real. But people would click on this button, it would start playing something, and it was wildly wrong.

And that doesn’t [00:30:00] help you when you still want to watch something. And there had been a lot of discussions, and there had been a lot of even tests around different algorithmic capabilities to serve a different piece of content. Do we take into account your viewership history? A lot of debate on this. And the reality still is, unless you show in advance what it is, people are still not going to click.

So this whole feature, this idea of like, what’s behind the curtain, Wizard of Oz style didn’t work for us. And so we did invest in keeping it going and we’re still thinking about that problem space. It’s still really true still to this day, but that was not the feature that solved it. So there’s no need for us to continue it.

Janna Bastow: Excellent. All right. Did you do it before Netflix did theirs, or did you learn from what your competitors were doing? 

Stephanie Musat: We ended up killing it as part of a larger migration, so it wasn’t the exact reason Netflix did, but it was around the same time. That’s the funny thing about these. Streamers are we all do the same thing around the same time.

But I think there was before ours. But we didn’t necessarily think about killing it until we thought about what exactly is [00:31:00] part of that delightful experience that really solves these media problems and we had no justification to keep it, the data wasn’t showing it. No users were really super, super passionate about it.

Yeah, we got rid of it and it did not. affect our platform metrics. Excellent. So that’s the thing. 

Janna Bastow: So was it a silent kill or was it something that you had to trumpet? Did you have to give a warning? How do you consider that? 

Stephanie Musat: Yeah, so we I’m in the space of, again, direct to consumer, large scale media we just remove things.

And now, in this case That’s not what I was 

Janna Bastow: expecting. We’re in B2B, so way fewer users. And generally, it’s like, well, you can get a little bit more leeway. 

Stephanie Musat: We wean people off of it, I guess is probably the other way. For this one, I think I’d have to go back, but I’m pretty sure we just cut it out.

Cold turkey. But in some cases, we test our way out of it. So we’ll reduce it to 20 percent of the audience for 30 percent of the audience and we’ll measure our platform level. So again, I’m not looking at feature level metrics in [00:32:00] that case. Obviously, if the thing doesn’t exist, that feature metric is going to decrease.

Like they’re the, so I’m over, solved it done, but is there a material difference to the platform to the product at large? So my service, if we remove something, does that affect things like in our world, what we measure, like just string streaming days, how many times people are coming back per week?

Minutes watched, how many videos, how much time are people spending on their service, these kinds of things. Those are the things that are our business objectives. That is what makes us relevant in the market. That’s where the money is. So if I affect those kinds of metrics, either net good or net bad those are the things that me and my team pay attention to, more so than feature individuals.

Because, again, if the feature doesn’t exist anymore, those metrics are not going to matter. So 20, you can wean it off. Some people do cold turkey, which I actually am a supporter of if you are that if you’re not risk avert because what people do in those cases, in my opinion, is find the next [00:33:00] alternative, which should be in your product in some way, shape, or form.

So if they were trying to do this thing, they can’t do it via whatever you had before. They’ll figure out a way to do it if they still need to do that function. In more of the B2B space, maybe, Janet, this is a better question for you, but I assume that you have to give some sort of advance notice, maybe, or tell some clients who might use it, or I don’t know if you have a tactic in which you’ve communicated this out.

Janna Bastow: So we have some general rules around it, right? So if it’s something that’s actually being used, then we give heads up about it, and that’s in the form of emails and in app notices and whatnot. If it’s something that’s frankly not being used, then it. Get silently removed because if it’s not being used, we just don’t have the time to really interrupt everyone’s life to tell them, Hey, by the way, this thing is going away that you didn’t even know about.

Just focus on building the next thing of value and getting it in its place. 

Stephanie Musat: Totally. I yeah I’m curious how many people, if they didn’t know it, if they didn’t know that it was removed, what is the point of making noise about it? And would they make a bigger, like, stink about it if they knew about it?

Yeah. It’s a little bit more of that psychological evaluation here, like, how much noise do we need to make? I agree with that. 

Janna Bastow: Yeah, absolutely. And, this sort of goes back to some of the tension around removing something. Are you removing it because no one knew it was there, and, therefore the feature never had a chance?

Or was it that no one actually used it because they saw it and they were like, no, don’t want to use that. And so it can be difficult to figure out whether something is remotely worthy or not. 

Stephanie Musat: In the evaluation of remove worthy, that’s a debate we’ve gotten into many times. And so one of the approaches we tried I’m curious about.

So if you’ve done the Kano model, which is like basic expectations, satisfiers, colliders, that kind of thing. We flipped it and we had our basic expectations, displeasures, and offenders as prongs on this model, where you can take your feature set that exists in your product today and plant it on this, on the map, basically, and say, okay, most of your features, the ones that are being used are probably [00:35:00] now within basic expectations of your consumer because they use them on the regular.

For the things that you don’t really want to maintain or look at anymore, do they fit into displeasures, which maybe you keep for a little bit longer or try to pivot or figure out what the fit is, or are they straight up offenders? People don’t use it, people don’t want it, like, get rid of it. And it was an interesting exercise, I would love to hear if anyone tries it and tries to flip it, but.

Instead of evaluating that in what do we need to do from the user perspective moving forward, the antithesis of that is what in our product right now is not bringing joy, is not fulfilling its need, and can we map that and have that be a spot for us to then consider adding those removal processes into our next phase of work.

Janna Bastow: Yeah, absolutely. And actually Oksana had a really interesting question in the Q&A, which was removing features. It takes time and money. When would you consider just leaving it in there, even if it isn’t used? 

Stephanie Musat: Someone used the term, like, KTLO to me [00:36:00] recently, which I had only Keep the lights on.

Okay. Which they’re like, oh, yeah, this feature is KTLO, and I was like, well, that is not an acronym that I care about. I don’t love that. And that again comes down to you and your way of working. But to me, then you just have stuff in your house. Like it’s basically an extension of like a hoarder.

Not to say that we get there. Your product itself, again, has. Arms and legs that can only fit so much stuff. And if you then create a section of your app of just keep the lights on, this stuff isn’t hurting anything, then it would take a time, you’re not wrong, an energy to remove it, but then there’s risk involved in that.

The more you jam in, then you have, again, maybe some bloating, you have scalability issues, but also if something goes wrong in one of those things, then you have to go digging to figure out what to do. what caused that problem, where the bug is. It becomes an issue with onboarding new team [00:37:00] members, maintenance overall, like all of these things end up being just your house gets cluttered.

Janna Bastow: Yeah. 

Stephanie Musat: And I am a little too Taipei to allow a cluttered house. Yeah. So if there’s truly a reason that you don’t want to invest in it anymore to me, that’s the clearest sign to get rid of it. Keeping it there is only going to cause issues later. 

Janna Bastow: 100 percent agree with that, and this is basically the argument that I made earlier this week when we were talking about removing something in ProdPad, which was basically like, remove it quick before someone actually does start making use of it, and then we have like, one person to explain to, or two people to explain to, going, by the way, I know you just started using this thing.

It’s gone now, whereas right now, it’s not really being used, just Display none. Get rid of it. Quick. Yeah, 

Stephanie Musat: totally. I see a couple people in the chat talking about tech debt and I don’t think that’s the main catalyst for this discussion, but tech debt is super real. And I’ve been painful. And painful, oh gosh really painful.

Many times I had found myself in an area where we want to do stuff, but we [00:38:00] can’t, or it’s complicated. And again, things are twisted and tied into each other and it’s knotted and it’s really hard to get that all separated. And tech debt is often also a good sort of peace offering to make with your engineering team and all this.

So if there’s a way for you to reduce that by saying the features I own are not that important, let’s get rid of some of that, make our app healthier, more scalable. Like you’re in a good spot, wasn’t me, you’re engineering folks. 

Janna Bastow: Yeah. We’ve actually reused features in ProdPad. So we’ve found that in the past, we’ve ended up over engineering bits and pieces and going, Oh, well, we created an extra piece here, like an extra database object and a thing here, and then forgotten about it.

And then years later gone, Oh, actually we need to add a new Tag object or something. Didn’t we like to hide that one year ago? We never had tech debt, but actually we can raise that back up. Bam. Now we’ve got this thing out that’s actually being used. I would not recommend that necessarily as a a product strategy.

Stephanie Musat: You and future, you clearly knew [00:39:00] something. Because it 

Janna Bastow: does require people in the team to have an intimate knowledge of what’s. Going on behind the scenes and what’s, why it was built in a particular way which probably won’t exist there forever within the product. But yeah, we’ve definitely seen tech debt be reused in various ways.

Yep, definitely. 

Stephanie Musat: I think one of the biggest lessons here, the things that I would love for everyone to think about and also comment on for sure is, so the process of when you make this decision, right, you’ve gone through your evaluation, you’ve You have reasons to believe that this is no longer needed.

You go through these so you decide if you’re going to, wean people off or shut it off cold turkey or test your way out of it. In that time there’s options for you to think a little bit about what Led you there, and that is also the space for reflection. There could be a million reasons, did the value proposition of this feature get diluted by the nature of the market, that people evolved past it, maybe the cost was too high.

Both from a consumer perspective and also [00:40:00] maintenance perspective. Maybe, one person used it or whatever it is, but there’s this moment of ownership I think is really important. And that creates the healthier culture within your organizations to continue to do this long term. To, to your earlier point, like when you were talking about your teammate, killing things is a manifestation of failure.

Like there’s this idea that I did something wrong and therefore I have to backtrack my way out of it. 

Janna Bastow: Yeah. 

Stephanie Musat: I can tell you there’s plenty of times where I worked on a product, I didn’t really think much about it. I started doing it and I couldn’t really justify it anymore. And there was even a more awkward conversation of how did it get that far.

And there were even times where I went and I built a product and I killed that product. And like two years later, I came back and I was like, you know what, I’ve got a really good idea. It was basically the same thing I bought, built two years prior. But having that moment of ownership, documentation, being super clear and broad and open about why you made these decisions makes it a little bit more palatable to continue to do [00:41:00] that.

And that might actually even be a jumping point or jumpstart for you to have a discussion with your teams. Okay, this didn’t work for this problem. What else can we do? What are the other ideas for the test and whatnot? People think, I think, when you say I’m going to deprecate something, it goes a little bit under the table.

You try to sweep it under the rug, which I can understand. But if there’s a culture, and this is the mindset shift around making this a little bit more palatable overall, is to have that discussion widely and have people buy into, yeah, we didn’t really need this anymore. And as soon as that narrative starts and your team, it becomes so much more commonplace and so much more healthy for you to then say, okay, where else can we lean in where let’s invest resources and time and energy and thinking on the good stuff and let’s get rid of everything else.

It will feel very cleansing. It will feel very positive. And all of a sudden that’s becomes the shift you want to get at instead of being like, Oh, we didn’t do this right. Let’s get rid of it and sweep it under the table. 

Janna Bastow: Yeah, that’s exactly right. Celebrating the wins that you get from removing features, from deleting code, from tidying stuff [00:42:00] up.

One of the things that I’m always recommending here that we build into our weekly processes is what we call tidy time. And tidy time is time that everybody spends, whether it’s a couple hours at the end of the week or, your Friday afternoon or whatever, where you spend tidying up and it’s not just for the product or the code.

It’s for general admin stuff as well. All that admin debt that gets stacked up. But some of it might be, For the developers to go through and just be like, what, there’s some junk in this code here. Let’s remove that. Here’s a piece that we want to review here and let’s remove that.

And actually just having regular sweeps of the product and saying this part, this can go, I think can be really healthy, just removing that frankly, that tech debt that product debt that’s been stacked up. 

Stephanie Musat: Take that, identify it, stack rank it the same way you would do new features.

You can stack rank the removal features and get it into the queue. And then again, to the point earlier, yeah, you do need engineering usually to remove something or at least some time and energy and effort. If that’s considered as part of your quarterly planning, [00:43:00] then it doesn’t become extra. It becomes part of your process.

So go through the. The process, say, for Q, gosh what quarter are we in? Q4, Q1, probably done. Q2 next year, like, let’s start removing some of this stuff. All of a sudden, it becomes, you know, effort that has been evaluated, swagged the same way as everything else. People will be on board with it.

Cause again, no one wants to work on stuff that’s not working. Like human nature in us as polite and everything that we are. We also don’t like working on things like, I was going to say loser products. I don’t mean loser products, but like things that aren’t working, right? Like these things that aren’t contributing.

No one wants to be the product owner of something that they can’t justify the value of. You want to work on things that actually contribute. So how do you do that? Get rid of the stuff that isn’t and give yourself space to do something that’s a little bit more thoughtful. 

Janna Bastow: And you’re absolutely right.

It’s about building it into the process. There should be a process by which you define, what the problems are and the outcomes for an experiment, and if that experiment is adding something or [00:44:00] changing something, then cool. Generally have those processes in place, but we often don’t have the processes or the same processes in place for removing stuff.

And I think that’s just as important, right? Saying, my bet is, I think we should remove this piece here. If we do that, we’re going to have more people using this part right next to it because they’re going to see it more, right? And we’re going to measure it by looking at this. And so you’ve got your target outcomes, you know how you’re going to measure it, you know why you’re doing it.

And then you can, as you say, rank it against other stuff. You could do this thing, or you could do this other thing over here, which. might have more impact on it, make this thing bigger or whatever it is. But also the follow up as well, right? Having clear processes, having a clear understanding as to, if you remove something, how does that propagate through your tech support, your help guides, your sales training, your other stuff like that, right?

Make sure that you’re updating your sales team on what the new demo flow looks like, or make sure people know that they need to remove it from the help [00:45:00] center. And where does it get removed from? Who’s in charge of that? That sort of thing. 

Stephanie Musat: It’s the inverse of everything new you add.

So in theory, you have some of those processes. If you’ve shipped something new, you hit up your customer sale or your customer service team or whoever, and say, okay, this new thing is available. And that is actually both in, D to C and B to B. Like here’s something else that exists in the same process that you use with product marketing or whoever else is part of that.

The inverse is the same too, which is you should have the same communication flows, the same documentation, the same stakeholder management or roadshows or whatever you end up doing. 

I 

don’t see this as any different in that. And I think that’s the critical piece here again, as you begin to deprecate and remove things.

It should be understood because you also don’t want sales to start selling against something that doesn’t exist anymore or have people ask about this feature and have your CX team say, Oh yeah, of course we still have it. And like, there’s an education piece. Absolutely. And the same way you educate people on what’s new, you educate them [00:46:00] on what has been removed.

Janna Bastow: Absolutely. So with experiments, you were talking about taking it away bit by bit. Is this something that you would roll back if people scream? 

Stephanie Musat: People screaming is interesting to me. 

Janna Bastow: Maybe not outright screaming, but that would be an interesting reaction. 

Stephanie Musat: The physical act of it, no I might be a minority, but like, that is an input of a larger decision.

So yes, people screaming or perhaps just contacting their team, more likely, and being like, what happened, where is this thing, is one arm of this. Again, if you look at platform level metrics here, and you don’t see a decrease in any of the key things that you track, a couple people being irritated about this does not equate a reason to bring it back.

Thank you. We had been brought in on a project once when I was consulting for a company and they’re like, we’re building this because 3 percent of people have been really vocal about it. And I [00:47:00] was like, that is crazy to me. But then they showed me exactly how vocal those people are. And it was such a small basic pivot on a feature that I bought into it.

I still don’t know if I agree with it entirely, to be honest, but like when you look at the larger ecosystem that you’re operating in. A couple people complain in the short term, well, do they stop paying for the service long term, or do you have a contract that got cut because of it? Like, you have to think of what this means overall.

If they’re complaining for the sake of complaining, some people just like to complain. And I’m not quick to necessarily move on that unless I see that platform measure actually move. 

Janna Bastow: And also you’ve got to think about what 3%, right? If it’s 3 percent across the board then, you’ve got to consider that.

But if it’s 3 percent of your highest paying customers, right? If it’s your biggest enterprise by customers who are all saying, no, we use this tool. If you take this away, this is not going to work for us. That plays a different bearing. And that probably gives a reason as to why that thing was built in [00:48:00] the first place.

If it’s 3 percent of people 

Stephanie Musat: don’t, that’s the thing that would get me. Yeah. 

Janna Bastow: Yeah. 

Stephanie Musat: Cause no one’s Dang it, I have the data that says no one’s using that. So that’s where that tension comes in. 

Janna Bastow: Yeah. But also identifying your ICP and segmenting your data down can actually help clarify who’s using and who’s not using, cause you might discover, actually, you know what, we’ve got 15 percent of our users using this thing, we’re trying to build for this segment here and actually have those people using it, they’re not the ones who are going to help us become the X of Y, wherever it’s we’re going.

So if we take this away, Right. They might yell at us, but those aren’t the people that we’re trying to build for and towards. You’ve got to understand who’s using it and how. 

Stephanie Musat: Totally. And then, to your point, you also have to quantify the impact of those people. Yeah. One of the things I think What’s scary about deprecation is people often use data to sugarcoat the reality because they made a case off of it.

So they’d be like, Oh yeah, but there are people using it. Look, and then show you data. And it’s like, for me, there’s a very specific example of [00:49:00] like, yeah, this is successful. I was like, where? They’re like mobile users in Brazil. And I was like, that is so specific. And like, I’m not going to keep a global feature because of a mobile user in Brazil cohort.

That’s really not really interesting. 

Janna Bastow: So Huda has a question in the chat. They’ve asked how do you balance between giving the feature or product enough time to succeed versus cutting your losses early? When do you know when to kill it? 

Stephanie Musat: That’s a great question. And unfortunately, I can’t necessarily define that for you.

I can tell you for me, my often horizon has to be more than three months because three months to us is a retention metric. So if someone does something for three months or longer, that equals a healthier lifetime value of our type of subscriber. So I would hope, or perhaps you and your teams can work on what that health metric looks like, or like a long term, long term value metric looks like, and you would run it for at least that.

But during that period of time, within three months, six months, whatever it is, there should still be a value of metrics that you’re looking at. You could still take the AB test within that. You could still [00:50:00] do a time where you launch it, let it run for a little, and then remove it again, and play around with the variables before stripping it out entirely.

Though to an earlier point, like I do think Data in the first instance will be very telling. You don’t want people to buy into something, even if it’s small, and then you have to take it away from them. So the nature of your business will be very helpful in determining that time horizon. But I can tell you just straight up 3 months minimum for where I am.

I’m currently working because that’s our retention metric. 

Janna Bastow: Yeah, and I can speak to what we look at here at ProdPad. Mostly we’re looking at things like, weekly active users, right? How many people are actually using functionality and coming back every week. And so if it’s something that, Most of the stuff in ProdPad is designed to be used on a weekly basis.

Some of the things you’re not going to be doing every week. But generally if it’s not being used within the first few weeks or so, we get a good sense as to whether it’s going to make the cut or not. So what we generally do is whenever we [00:51:00] put something out there, we have our target outcomes written saying we expect this level of usage by this point.

And if it’s not met, that’s when it starts Getting onto the chopping block.

Excellent. I think that’s all the time that we have for questions unless anybody else throws any last ones in there. But in the meantime, I want to say a huge thank you to Stephanie for taking time to share all these really insightful takes on how to kill features.

So thank you, Stephanie. I really appreciate that. 

Stephanie Musat: Thank you for having me. I appreciate the conversation. 

Janna Bastow: Excellent. And just some final bits to wrap up. So everybody, you know that we do these things on a monthly basis. Today has been recorded. So you’ll find that at ProdPad. com slash webinars.

And at the same place, you’ll be able to sign up for our next one. We’re going to have Ben LaMoure joining us in January. So same time, same place. We’re going to get a date there very shortly. So watch this space. And if anybody is interested in taking a look at ProdPad, we run personalized [00:52:00] one to one demos where we’ll show you through how to do a roadmap and how to manage your backlog and how to connect it with your JIRA and your ADO and whatever it is else that you’re using.

So on that note, I want to say a huge thank you to everybody for joining today. Thanks for all your questions. And once again, Stephanie, thank you so much for joining. This has been 

Stephanie Musat: Awesome. Thanks again. Thank you for having me. It was great to chat. 

Janna Bastow: All right. Wonderful. All right. Take care. Bye for now.