10 Years After the Lean Product Playbook: PM in the Age of AI

Is there a risk that people jump too quickly into the solution space and don't adequately investigate the problem and solidify that they're solving the right problem with these tools? Before

vibe coding, you know, solutionbased thinking is very prevalent. People start

talking about we should build this feature, we should build this product.

Sales teams are asking for features, stakeholders are asking for features, clients are asking for features. All

that discussion is actually in the solution space. A lot of PMs are

solution space. A lot of PMs are spending their time just managing the scrum process and they don't have time to do discovery and now it's easier than ever to create a solution which is kind of interesting if it's like hey if

designing it and coding it is no longer the bottleneck then it's like well what text you put in is the only thing that makes any difference what specifically has AI changed this feels pretty timeless yeah it is pretty timeless I

mean you know at the end of the day you still have to understand your customers and AI is not going to tell you you know about your customers Dan Olsen I have been following your work for over 15 years. What has changed in the past 10

years. What has changed in the past 10 years? I think really two main things

years? I think really two main things have changed. One is the other big

have changed. One is the other big thing, how has AI changed product management? AI has pretty much changed

management? AI has pretty much changed every step in the product management process. It can help you explore new

process. It can help you explore new opportunities that are out there in the market. It can help you identify

market. It can help you identify segments. It can help you brainstorm

segments. It can help you brainstorm feature ideas. It can help you do

feature ideas. It can help you do competitive analysis. But probably the

competitive analysis. But probably the most fundamental area it's changed the most is you're a legend in the product management community for your work on the lean product playbook. What was the

thesis? Yeah, the thesis was basically

thesis? Yeah, the thesis was basically there was this term really quickly. I think a crazy stat is

really quickly. I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on

not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's

episode.

Dan Olsen, I have been following your work for over 15 years. You are a legend in the product management community for your work on the lean product playbook.

For those of us who have forgotten in the last 10 years what that was about, what was the thesis?

Yeah, the thesis was basically that there there was this term product market fit, you know, defined by Mark Andre back in 2007, but there was no like rigorous framework or systematic process for how to do it. And through my

consulting and speaking, I realized that I could create that process. And that's

the lean product process, which is a simple six-step process that, you know, thousands of companies have followed at this point to try to achieve product market fit. Yeah. So, this is the

market fit. Yeah. So, this is the six-step process. Uh basically you start

six-step process. Uh basically you start with defining your target customer. You

figure out what are their underserved needs. Those two layers form the market

needs. Those two layers form the market part of product market fit. And then

from there you define your product's value proposition. You figure out what

value proposition. You figure out what your feature set should be. That's where

MVP thinking comes in so you don't over scope it. You figure out your UX. You

scope it. You figure out your UX. You

design the UX. You prototype it. And

then you go and test it with customers to see how you're doing with product market fit. And you iterate through a

market fit. And you iterate through a learning loop to try to achieve product market fit or decide if you have to pivot or worst case, you know, you pull the plug. But that's basically the

the plug. But that's basically the process at a high level um for how does she product market fit. What has changed in the past 10 years? I think really two main things have changed. One is when

the book came out product management wasn't as well understood as it is now or valued and you know um in the last 10 years it's really exploded. People are

way more product manager jobs. People

appreciate what product managers do and so there's more appreciation for the techniques and frameworks in the book and more and more companies are actually applying it. The other big thing which

applying it. The other big thing which we're going to go deep in today is AI.

Obviously AI we've we've been fortunate to live through I've been fortunate to live through a few different disruptive waves of the internet, you know, mobile uh and now we've got AI and so it definitely is impacting how teams are creating products and I'm excited to get

into that with you. What specifically

has AI changed? This feels pretty timeless. Yeah, it is pretty timeless. I

timeless. Yeah, it is pretty timeless. I

mean, you know, at the end of the day, you still have to understand your customers. And AI is not going to tell

customers. And AI is not going to tell you, you know, about your customers. Uh,

you know, um, you still have to figure out how to segment your market. You have

to figure out your persona. It can help you brainstorm and come up with, you know, potential ideas and segmentation attributes, but at the end of the day, you still have to get out of the building and talk to people and uh to do your discovery research, basically. So,

it can help you synthesize discovery research, but you've got to come up with it. Um, and then the next thing is you

it. Um, and then the next thing is you need to once you've identified the customer problems, you've got to prioritize which ones are the biggest opportunities. That's where my

opportunities. That's where my importance versus satisfaction framework comes in. And an AI is not going to be

comes in. And an AI is not going to be able to do that for you. You're going to have to use judgment. And um, you know, I played around with Genai. It's really

good for brainstorming and creating convergent like divergent ideas, but it's not good as good when it comes time to eval kind of converge and evaluate and prioritize ideas. And so that's where you still need to help. You still

need to do that. And then your value prop. You know, I've tried a couple

prop. You know, I've tried a couple times to get uh Cha GBT to create a product strategy. They kind of sound

product strategy. They kind of sound good on paper, but when you scratch the surface, the substance isn't really there as much. So, I feel like that's the case. And then defining your feature

the case. And then defining your feature set, that's another place it can help.

It can help you brainstorm features, right? You say, "Hey, I want to solve

right? You say, "Hey, I want to solve these problems for customers." It can come up with some really cool feature ideas. And then the biggest place though

ideas. And then the biggest place though is in that creating the UX, creating the prototype. And we'll get into the

prototype. And we'll get into the details of that. But with AI today, you can do it a lot quicker. And if you don't have a designer, you can actually do it now where before you might not have been able to even do it at all.

I think that really is the unlock, right? Is with AI prototyping tools and

right? Is with AI prototyping tools and with vibe coding tools, what used to be PMs sitting in a dock, writing things

up, and then begging their designer for some resources or some time to do some explorations off of a napkin sketch. PMs

are now more empowered than ever to go ahead and come up with some explorations in the solution space. Yes, it's true.

PMs really couldn't do a whole lot without designers. So, let me walk you

without designers. So, let me walk you through the old flow. Um, I learned early on in my product career how important UX design is. So, in my book, I have a whole chapter on UX design. And

I like to describe the different artifacts that people could use and like kind of preferred order. So, I like to categorize them by interactivity and fidelity. As you can see here, we

fidelity. As you can see here, we usually start with a product brief or a PRD, right? some textual document. Um,

PRD, right? some textual document. Um,

and then we start doing hand sketches with our teams. We iterate those until we're happy with them. And then we can move on to clickable wireframes if we want. Um, low fidelity. They're low

want. Um, low fidelity. They're low

fidelity. They're usually black and white. A tool like Bamic, you can test

white. A tool like Bamic, you can test with those. But really, the best place

with those. But really, the best place in the old workflow to test was when you had clickable or tapable mockups. Yeah.

Right. These days, you use a tool like Figma to do that. In the old days, you might use Invision with whatever assets your designer gave you. And I you know for a lot of my clients I this is where the rubber met the road. We would do

waves and waves of user test at this stage and then once we worked things through and iterated to the point where we had confidence in product market fit then we would go code the product and of course we test the line product. But

basically in order to do that flow you needed to have someone who could prototype on your team. And that brings up something I've been talking about for a long long time which is the design gap on many teams. If we just take the highle activities that a product team

needs to do to create a product and we say it's basically defining the product which is mainly PM's job designing the product which typically falls in the designer and the develop then basically there's different levels of UX maturity

within a team and different org structures that you see the most common one and level one is basically hey we've got developers and that's it. There are

a lot of startups that are like that.

Actually a lot of AI startups these days are like that too because at the end of the day if you don't have coders you're not going to build anything. So that's

like the least UXsavvy level. The next

level up is, hey, in addition to engineering, we have PM, but neither one of them is really a UX expert, right?

That's the idea. You're gonna see, you can see when you see products from these kind of companies, you see the lack of UX design come through in their product.

Now, maybe, you know, one of these two can lean in and help try to cover some or most of that gap. Maybe the product manager, you know, has picked up some front-end skills, some design skills, or maybe you have a front-end coder who's

picked up design, right? They might be able to get by. And maybe between the two of them, they can kind of cover it.

Frankly, in the early days of in it, where I started my product management career, that's how they got got by before design was a thing. That's how

they got by. The PMs and the engineers thought enough about UI to get it right that our product was easy easy to use.

But the real thing you want to have obviously is the triad, the trio that we talk about. You've got product

talk about. You've got product management, engineering, and UX who can do that. So, you know, unless you really

do that. So, you know, unless you really had level five, it was really hard to like rapidly create good prototypes. The

cool thing is with vibe coding, it's completely changed that one. If you

don't have if you're if you're living with a design gap, you as a PM now can actually create those prototypes and you can create them quickly. And even if you had a designer, we know that designers are busy. Um they get on multiple

are busy. Um they get on multiple projects, so your project might not be a priority for them. You can now take it on yourself and get it done. And so you can get to a prototype more quickly even if you have a designer. And so the new

workflow is, you know, pretty is pretty interesting is you can just, you know, you're starting with text again. You're

starting with text, you know, whether it's a product brief or a PRD, that's going to be the input into your Vive coding tool. And now you're going to get

coding tool. And now you're going to get a live prototype. You know, before Vive coding, people would create what we call HTML prototypes. It would be full-on

HTML prototypes. It would be full-on HTML, CSS, maybe a little JavaScript, but no backend if they wanted to really kind of have it be really high fidelity.

Nowadays with vibe coding tools, you can totally have a front end and you can even have a back-end tool and so you can get quickly to that live prototype to test your idea. And that's one of the key things about the lean product

process of is you want to get through all those bottom layers and get to the prototype as quickly as you can because it's by putting that prototype in front of customers and getting feedback where the real learning and iteration towards

product market fit happens.

Yeah, I think that the biggest unlock for people was just testing those clickable prototypes that we were building, whether they were in Balsamic

or Figma. But now what we're seeing with

or Figma. But now what we're seeing with AI prototyping tools like Vzero, we just had the CPO on the podcast. Lovable

Bolt, we had the CEO on the podcast.

With those tools is that any of the designer, the PM, the engineer, heck, I've been hearing stories about salespeople and marketers, right, can

create live prototypes.

Totally. Yeah, it's true. It empowers

everyone to do it. um you know we can talk about you know basically especially if there's a different use cases too right if you're creating a brand new concept then it's great for that because you're starting from scratch you have no existing design

system you have no existing code base so you can create it so it's great for those kind of conceptual new concept ideas um you know obviously if you have a designer the design is probably going to be better if a designer's involved versus not these things will generate

kind of plain vanilla UI and they are getting better and better you know I I've used several of these I use lovable and bolt I was just you know Yesterday, two days ago, I was at this brainstorming session with a client and the team had come up with this idea and

I just I wasn't didn't even say anything to them. I just pulled out my laptop and

to them. I just pulled out my laptop and pulled up Lovable and next thing I was like, "Here's a prototype." And they're all like, "Whoa." You know, so it's a great way to get your ideas out quickly.

Um, which is super exciting. Now, there

is a separate use case where it's like, okay, I've got an existing product. We

have an, you know, so one, if you have design systems you need to live with, um, then you can, you know, you need to figure that out. Although I was just playing around with some of these tools and one of them actually let you kind of import your existing design system if

you wanted to which was pretty cool. Um

you know so it's it's very interesting.

So it's interesting because the the unit of work before like at a high level it was like product managers created text right designers created figas and then

developers created code from the figas.

So the biggest design work product was a Figma board right well now you that doesn't have to be the design product anymore. You can go straight from text

anymore. You can go straight from text to a live prototype. You can still go to Figma. The funny thing is to see the see

Figma. The funny thing is to see the see the workflows. You can still go to Figma

the workflows. You can still go to Figma or back from Figma if you want to. You

know, it's kind of interesting that everyone's still plugging and playing with Figma for the most part. Um, and

then at the back end when it's time to integrate with your code, if it's if you have existing code, then that's that's a whole other thing. Um, but just to illustrate and get to a prototype and test a product idea, it's really fast

and it's great.

This episode is brought to you by work OS. If you're building a SAS app, at

OS. If you're building a SAS app, at some point your customers will start asking for enterprise features like SAML authentication and skim provisioning.

That's where work OS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand so that you can ship quickly and get back to building other

features. Today, hundreds of companies

features. Today, hundreds of companies are already powered by work OS, including ones you probably know like Verscell, Web Flow, and Loom. Work OS

also recently acquired Warrant, the fine grained authentication service.

Warrant's product is based on a groundbreaking authorization system called Zanzibar, which was originally designed for Google to power Google Docs and YouTube. This enables fast

and YouTube. This enables fast authorization checks at enormous scale while maintaining a flexible model that can be adapted to even the most complex use cases. If you're currently looking

use cases. If you're currently looking to build role-based access control or other enterprise features like single sign on, skim or user management, you

should consider work OS. It's a drop-in replacement for Ozero and supports up to 1 million monthly active users for free.

Check it out at workos.com/lenny

to learn more. That's workos.com/lenny.

Today's episode is brought to you by Jira product discovery. If you're like most product managers, you're probably in Jira tracking tickets and managing the backlog. But what about everything

the backlog. But what about everything that happens before delivery? Jira

product discovery helps you move your discovery, prioritization, and even road mapping work out of spreadsheets and into a purpose-built tool designed for

product teams. Capture insights.

Prioritize what matters and create road maps you can easily tailor for any audience. And because it's built to work

audience. And because it's built to work with Jira, everything stays connected from idea to delivery. Used by product teams at Canva, Deliveroo, and even The Economist. Check out why and try it for

Economist. Check out why and try it for free today at atlassian.com/roduct-discovery.

atlassian.com/roduct-discovery.

That's atlasia.com/product-discovery.

Jurro product discovery build the right thing. Is there a risk that people jump

thing. Is there a risk that people jump too quickly into the solution space and don't adequately investigate the problem and solidify that they're solving the

right problem with these tools. I I I'm so glad you mentioned problem space. My

uh my heart is very happy right now.

That's one of the things I'm very passionate about. And yeah, you're

passionate about. And yeah, you're right. I mean before vibe coding, you

right. I mean before vibe coding, you know, solutionbased thinking is very prevalent. Um you know people start

prevalent. Um you know people start talking about we should build this feature we should build this product sales teams are asking for features you know stakeholders are asking for features clients are asking for features all that discussion is actually in the

solution space and so if we go back to that high level define design discover or develop a lot of PMs are spending their time just managing the scrum process and they don't have time to do

discovery and so they end up just kind of building what people asked for and then it's kind of like ready fire aim you never validated it was a true customer problem so I do think one is our solution based thinking was already

prevalent and now it's easier than ever to create a solution which is kind of interesting if it's like hey if designing it and coding it is no longer the bottleneck

um then it's like what text you put in is the only thing that makes any difference right and how well thought out is who the customer is and what their problems are and what features we need in order to support that um so I do think it will lead towards solution

based thinking the other thing I thought a lot about too is I think it's going to lead to um a higher challenge for differentiation, right? If everybody's

differentiation, right? If everybody's using the same tools, you know, it's kind of like raising the bar. You're not

going to see as many bad UX websites if everyone's using this because it kind of follows general principles. You're not

going to get some amazing UI. You're

going to get some plain vanilla UI. You

can ask it to copy the UI of another website. So, you can copy people, you

website. So, you can copy people, you can create plain vanilla stuff, but if you really want to innovate in UX, which we can talk about, go a little deeper in next if we want, you're still going to have to design. But basically if anybody

can create any idea then what's the differentiation at that point right and the only real differentiations are you're solving a problem a real problem that's an important problem you're solving it in a way that's better um you

have a data advantage right that's the other thing people are talking about as they realize AI kind of these tools are kind of becoming commoditized and the proprietary data that you have becomes more important as well so so I think

you're going to see uh increasing you know challenge in differentiating one product from the other when anybody can create a product pretty easily. Yeah.

The way I like to think about it is that AI is really good at pulling people up to the average. You can even see this in a platform like LinkedIn. Like a lot of the bad content that used to be out there, those people are plugging their

content into ChatGpt or Claude and they're bringing it up to average. Yep.

Y but you don't see those people winning, right? The people who are

winning, right? The people who are winning are the people who are, you know, crafting it on their own. It

doesn't sound like the what I like to I think of JGBT text as like brown, you know? It's like the average of

know? It's like the average of everybody's writing style, right?

There's there's no clear like red or blue or pink or different style there that you're getting that pointed voice, right? That's why everybody loves Marty

right? That's why everybody loves Marty Kikin's articles because it's his voice.

He's not writing with AI. And I think that with design that's the new bar that we're setting with these AI prototyping tools is that you need to create that

differentiated design. So what is the

differentiated design. So what is the right sequencing? You know when should a

right sequencing? You know when should a PM be experimenting with prototyping tools and when should they bring the designer into the process? How do they make sure that that solution phase which

is now totally transformed is done in a way that both PMs and designers will be happy? Yeah. Well, I'd love to pick up

happy? Yeah. Well, I'd love to pick up on what you said because the way I think about how how is AI helping people? What

is it doing? I think of a bell curve like you said, right? And basically what it's doing is people that were at the bottom of the bell curve. The floor of the bell curve is now a lot higher. The

floor is rising basically, right? So the

floor is lava. The floor is rising as Genai gets better and better, right? And

I love the quotes are always like, "Hey, this is the worst GPT you're ever going to use in your life. It just keeps getting better." So the floor, you're

getting better." So the floor, you're right. So the people that were like the

right. So the people that were like the bottom decile now they're performing in the second decile or you know third decile or something right? So it's like there's you know it's it's getting rid of that bad low end of the curve which is related also to the whole speculation

about hey what's it going to do for jobs? I remember about nine months ago

jobs? I remember about nine months ago people were like or a year ago oh it's going to kill PM jobs and in my head I'm like I don't think it is. If I think about the jobs that are most at risk, I think it's the developers. It's the

junior front-end developers because they're just at the end of the day code is text. And if they're in the bottom

is text. And if they're in the bottom decile, right, uh, of junior developer coders and this stuff can crank out third decile, fourth decile stuff, then, you know, it's going to replace that.

And, um, you know, so that's that's it.

And I think to answer your question about when to bring in a designer, let's get into the, you know, I think this is where it's important to double click on UX design. So, let me show you my

UX design. So, let me show you my framework. Like I mentioned when I got

framework. Like I mentioned when I got into PM um I I realized that you know UX design is super important. I mean

it's just really really important and uh and so I made it a point to learn all about it. I basically I could write to

about it. I basically I could write to put it in the parlance reason here I could write the best text ever. I could

write the best requirements document the best user stories but if we fall down on the UX I think about it like an iceberg because like an iceberg there's a part that's above the water that's visible.

That's what most people see and react to and that's the visual design. And that's

the part that's above the water. You

know, we're all looking at that's why people love highfidelity prototypes or live coding things because it's real pixels. It's high fidelity. It's got the

pixels. It's high fidelity. It's got the colors. It's got the fonts. It's got the

colors. It's got the fonts. It's got the images. But when you use a product

images. But when you use a product that's got product market fit and it's got a great user experience and is easy to use, it's because the team has like taken into account and done a really good job with these more foundational

layers below the waterline that aren't as obvious. Starting at the bottom, it's

as obvious. Starting at the bottom, it's conceptual design. like what's the

conceptual design. like what's the conceptual design overall design for this for this product right a lot of e-commerce things the conceptual design is just a catalog plus a shopping cart it's a very common conceptual design

Uber's conceptual design wasn't necessarily that it was just map ccentric it was like you know the thing about a conceptual design you can design it you could you could describe it verbally it's like hey let's show the user in the middle of the map let's show

the cars on the map where they're driving around and when the user hails a car then let's show the car going to them right there's a thousand different ways to implement that visual design but That's the conceptual design. The next

layer up is information architecture.

And so information architecture, interaction design, these are the things that these cool coding tools are probably not going to do as well as, you know, they're not going to do as well as a top-notch designer for sure, right?

Over time, they're going to get better like we're saying, but information architecture is, you know, is the way you're structuring the information and organizing it make sense? Like when you go to look for something, is it where you expect it to be? Does the navigation

make sense? Do the labels make sense?

make sense? Do the labels make sense?

And then interaction design is basically anything you can click or tap and the flows, right? And so that's the one of

flows, right? And so that's the one of the things that comes to mind to me when I vibe code. Like if you don't tell it the flow, then it it's probably not going to get the flow right. If you've got a

multi-creen, multi-page, or if you're designing one screen, there's not really much flow. But if you've got a

much flow. But if you've got a multi-step workflow or multiple feature product where they've got to navigate between it, you got to think about this.

And that brings up something that's really interesting is like I always feel like yes, you should start in the problem space. When you get to the

problem space. When you get to the solution space, it's so funny. You see

this with all Gen AI. You make a request in your prompt and it gives you something you didn't expect like oh I forgot to say make the grass green or whatever. And then you then you edit it,

whatever. And then you then you edit it, right? And you're like, "Oh, I forgot to

right? And you're like, "Oh, I forgot to say make the sky blue." And so it's funny by going back and forth by going to the solution space, it helps you go, "Oh, I forgot to I forgot this one problem space requirement." So it can

really be helpful to actually help you explore jointly the solution and problem space to get better requirements and better specs if you will. So but to answer your point you know if you're building some straightforward pattern

you know like some e-commerce shopping flow there are best practices out there you might try to innovate. If you want to innovate then you should get a designer but if it's pretty pretty straightforward or you're following an existing design system that's where you can kind of stay in the guard rails and

just kind of replicate what you already have in your product. But if you're really trying to create a better user experience, a better information architecture, better interaction design, or even a novel visual design, then

that's when you want to get the human touch of like a 10x designer involved in that. So I think that you know

that. So I think that you know theoretically timing wise a PM could go prototype the general functionality and flow with vibe coding understanding that

hey this is not this is not anywhere near the the final UI per se. It's just

getting us the flow and the functionality down perhaps.

So what I'm hearing is that there might be a reduced need to bring a product designer into every feature that's shipped. I agree with you. I mean right

shipped. I agree with you. I mean right now the you know the the good news is again like just like product management has risen. The appreciation that to

has risen. The appreciation that to develop a a great product you need a developer a designer and a product manager and ideally they're all very skilled in their domains and they are

ideally collaborating very well together. That's a high expectation.

together. That's a high expectation.

What a function of that now is most teams do have a designer and to your point the designer is involved with any UI creation, right? And therefore they are potentially a bottleneck for any UI creation. And so I think kind of

creation. And so I think kind of combining what we're talking about perhaps to your point, the more plain, you know, trivial stuff can be done without them and we can reserve their

time for the stuff that really does require more creative thinking like creating a new flow or creating, you know, some some cool new feature.

I also think that there will probably need to be a bit of an evolution within the product design field where I think that it has been a little bit like you said a bottleneck for what a team can

create and I think that this next iteration of product design we're going to see more and more that product designers have to be able to work with a simple feature and maybe build that out using prompt. So, you know, we see

using prompt. So, you know, we see things like Figma make, right, where it takes your existing design system. They

just prompt it out so that they can do that really quick and then when they actually need to more deeply apply the double design, double diamond design process, they can do that for bigger

features. Yeah. Yeah. Well, it's

features. Yeah. Yeah. Well, it's

interesting. Yeah. So, Figma launched Figma make, right? So, you can go straight from your Figma. The funny

thing is everybody else there's a lot of people Figma is kind of the was the dominant player in the old workflow, right? And there's that's not going to

right? And there's that's not going to change overnight. And so a lot of these

change overnight. And so a lot of these tools plug and play very well with Figma. And Figma themselves are trying

Figma. And Figma themselves are trying to innovate and and make it, you know, make it be the case that like the last thing they would want to see probably is designers having reasons to not be in Figma, right? That would be bad for

Figma, right? That would be bad for their business. And so I think and they

their business. And so I think and they have a a robust plugins and extensions, you know, platform and things like that.

So to the extent they can support these like I'm saying like I'm you know when you double click and get deep into it it's like okay we need to build a new feature it's relatively straightforward but we need it to comply with our design

system you know and certain companies have really you know robust wellthoughtout nice design systems well if a if a new PM comes in and says I'm just going to create a vibe coding prototype and it looks nothing like it

that's going to be a problem right so but like I said you're already seeing people that people are recognizing this the industry is recognizing that And this is a broader case. Take a step back. You know, everyone's excited about

back. You know, everyone's excited about Genaii. And when you play around with

Genaii. And when you play around with Genaii, like I did the whole action figure thing that everybody was doing, and I I decided to do a Star Wars figure because I'm a Star Wars fan. And what I found was it was good, but when you

wanted to change one thing, it would like change the rest of it somewhat. And

I I view it like rolling dice. I call it a roll. Like every time you make an

a roll. Like every time you make an edit, it would like roll the whole thing. And you it might fix the one

thing. And you it might fix the one thing, but it's like whackable. It would

mess something else out, right? And so

kind of a general philosophical concept that I'm pushing these days is now that we have Gen AI that's generating new stuff, we need a good edit UI. Edit AI.

We need a good edit AI, right? That's

what we need because and that's the thing is, hey, cool, honor my design system, but just generate this stuff in it. So you're seeing tools like being

it. So you're seeing tools like being able to support that use case recognizing that yeah vibe coding is cool when you have starting from scratch if you're a startup or a brand new product but for these companies that have thousands of lines of existing code

robust design systems how do we use those tools there I think the support is getting there and I think that's where the designers hopefully design tooling ecosystem will also support that which I think it will so what are the specific

tools you'd recommend for vibe prototyping yeah I mean I think you know it's interesting uh the space is changing very quickly. Um, but as of right now,

very quickly. Um, but as of right now, there are kind of like the more hardcore like if you are a coder, know how to code, there are tools, you know, like cursor that are in that kind of camp.

And there are other ones like if you're more like, well, I don't really know how to code. I want to care more about

to code. I want to care more about prototyping, you know, there's things like lovable and bolt that kind of live in that camp more, right? And uh, and then there's still and there's another category and those are great. Like I

said, if you want to build a new product from scratch, you can put in a prompt, it'll create a new product from scratch.

Everything's cool. The other thing is like imagine you're a PM, you don't have a designer and you need to modify or enhance an existing product or feature.

You know, in the old days, you might take a screenshot of it and then do what I call Frankenstein it like hack it up, you know, like put in some weird you try to make it like look as good as you can, but you're adding button janky buttons

and drop downs like it gets the job done. But the cool thing now is to

done. But the cool thing now is to support that specific use case where instead of starting with text, you can start with a screen grab of an existing product and then it creates basically a

Figma-like artboard from that, right?

And so then you go, all right, I'm going to change this text. I'm going to add a button here. I'm going to do this. And

button here. I'm going to do this. And

then you can just like with Figma, you can create a interactive prototype. So

when somebody taps here, it goes here and goes there. So the tools in that specific category are visally wizard which got acquired by Miro, Magic Patterns and like UX pilot. Those those

like they're kind of you know um and you know like magic patterns will actually generate code for you. The other ones will generate the artboard for you. You

can then export them to Figma. So again

there's multiple workflows depending on where you want to take what they create.

But they're focusing more on that use case of hey I want to you want to create an editable you know you don't have the original Figma. If you had the original

original Figma. If you had the original Figma file, you'd go and that would mean you probably had a designer, you'd go and edit that. So, if you kind of live in a Figmacentric world, your workflow is probably still going to rely on Figma Centric when you're trying to extend or

enhance existing functionality. But if

you're a PM that doesn't have a designer and needs to do that, you can use these tools. So, there's kind of those three

tools. So, there's kind of those three categories of tools. The hardcore vibe coding tools where you're really going to get in the code. There's the lighter weight ones when you're just trying to prototype. And then there's these other

prototype. And then there's these other ones that focus on this what I call reverse prototyping use case. You

basically suck in a screenshot and then you can mod make it really easy to modify and edit it and create uh a clickable prototype from that. So if I had to summarize what we talked about in terms of product management in the age

of AI, Jennai is changing most of the steps but the step that's most consequential is the change in prototyping and PMs need to learn how to use these tools but they also need to

learn within their organization how to collaborate with designers in a way that it doesn't feel like a stepping on their turf. Is that a fair summary? I think

turf. Is that a fair summary? I think

so. Yeah, I think so. And I think that that I'm glad you brought up the turf thing. I mean, um, you know, at

thing. I mean, um, you know, at different companies sometimes PMs and designers do have minor, you know, some degree of clash or v, you know, kind of, um, uncertainty about responsibility or overlap and ownership and things like

that. So, you're right, when you've got

that. So, you're right, when you've got tools that can suddenly generate designs, then you want to be careful about that. And you know, I think good

about that. And you know, I think good PMs when you do have a designer you're working with and in the interest of time because they're busy or you just want to get your ideas out there, you would create a wireframe or a mockup or

nowadays a vibe coding prototype. It's

always with the intent that if it needs to go through a design process, this is just kind of, you know, directional and illustrative. And of course, we would

illustrative. And of course, we would love to get your design expertise to make this even better and add some, you know, some truly unique components to it. So I think that's that's kind of the

it. So I think that's that's kind of the the spirit in which PMs could be approaching it if they're working with designers.

I think yeah, when you're working with designers, a lot of it is also just about being clear and upfront about your intentions, right? I'm not trying to say

intentions, right? I'm not trying to say this is pixel for pixel what we should build. Yeah. But like Dan was saying

build. Yeah. But like Dan was saying earlier, to explore the problem space effectively, I wanted to also explore the solution space that was giving me

good signals. And so I put together, you

good signals. And so I put together, you know, an AI prototype, but depending on where we see, you know, the impact of this feature and your resource availability, we could go through more

design cycles on this or we could just do a minimal, you know, making sure it conforms to our design system. Yeah, I

think Yeah, exactly. That's the intent.

That's the intent of doing it. And it's

funny because the way I think about it, too, going back to lean product process, like of course you should start off with discovery research. You should talk to

discovery research. You should talk to customers to understand their needs and preferences and what they're using today and things like that. There's kind of like that gets you to a certain level, right? There's diminishing returns.

right? There's diminishing returns.

After you talk to your 20th customer, you're probably done seeing new patterns. So, there's diminishing

patterns. So, there's diminishing returns. That's as far as you can go

returns. That's as far as you can go with customer interviews. The next way to kind of really derisk and increase your confidence is with a clickable interactive prototype. And so, that's

interactive prototype. And so, that's why they're so valuable once you get to clickable valuable prototype. And the

more the higher fidelity, more realistic it is, which by definition, these vibe coding tools are creating, you know, stuff that's actually in a browser or stuff that's actually in a mobile app, then it's that's that's when you like

the customer goes, "Oh, yeah, I know I asked for X in discovery, but now that I'm going through the workflow, I forgot that you also need to be able to export this to Excel, right?" So they can. So

like it really unlocks the next level of customer validation that you can do. And

to your point, I think there's value in doing that. uh if the PM can drive a lot

doing that. uh if the PM can drive a lot of that and derisk it and iterate it without taking design's time that's great and but then again to your point then it's like hey we validated these problems with the customers we validated

this highle flow and functionality set with the customers now we'd love to work with you to make it a great UX and a great UI you know AI evals are one of the most important skills for PMs and I

know you know they matter the question is are you doing them right most teams are winging it with basic metrics and hoping for the best. Meanwhile, the

teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought

evaluation. Today's episode is brought to you by the AIE evals for engineers and PM's course by HL Hussein and Shrea Shunker. This live Maven course will

Shunker. This live Maven course will teach you the battle tested frameworks from HML and Shrea who are the engineers behind GitHub copilot's evaluation system and 25 plus production AI

implementations. Four weeks live

implementations. Four weeks live instruction. Next cohort starts July

instruction. Next cohort starts July 21st. Start shipping AI that actually

21st. Start shipping AI that actually works. Enroll at maven.com with my code

works. Enroll at maven.com with my code AG-RO-Growth

for over $800 off. That's ag-p oct- w. Today's episode is brought to you by

w. Today's episode is brought to you by the AIPM certification on Maven run by McDad Jaffer who is a product leader at OpenAI. This is not your typical course.

OpenAI. This is not your typical course.

It's eight weeks of live cohort-based learning with the leader at one of the top companies in tech. OpenAI just

doesn't stop shipping, and this is your chance to learn how. Run along with product faculty and Mo Ali. The course

has a 4.9 rating with 133 reviews.

Former students come from companies like OpenAI Shopify Stripe Google and Meta. The best part, your company can

Meta. The best part, your company can probably cover the cost. So, if you want to get $500 off, use my code a a kas25

and head to maven.com-faculty.

That's mavn.com/pr

o du ct-f a cu ty. And ideally, you're bringing design in early in the process where they can help feedback whether they need to be involved in putting those AI prototypes in front of

customers and if you have a UX research team, whether they can decide. So I

think it's almost like before the PM goes off and does all these things, making sure that they have the right checkpoint to say, I'm going to be going doing these things. Do you want to be involved or even lead this process

potentially? Yeah, that's a great point.

potentially? Yeah, that's a great point.

I mean, we're kind of we're kind of painting a picture where there's like limited design bandwidth and that's why we're kind of going down this path. If

there if that's not a good assumption, if there's adequate design bandwidth, then yeah, it' be great to partner with UX on this. Um, you know, I think what you'll see is the forward thinking designers are going to lean into all

these tools and processes. So, it might flip back where it's like we're going to go back to, yeah, you know, who's better at creating these prototypes than a PM?

A designer who's really good at VIP coding. So, you know, like if they

coding. So, you know, like if they embrace it, right? Some people are sticking their head in the sand. Other

people if they embrace it, you know, they could theoretically and I I I you know, now that we talk about it like this, I am confident in the next year, definitely two years, there will be like

power AI design tools for designers, you know what I mean? That are going to have like power features and stuff that PMs are going to be like, well, I don't even want to deal with that. What do you mean the padding and margin? I don't want to deal with all that. you know I don't you

know so uh so I think that that you might see a bounce back uh with with forward thinking designers where the tools have emerged enough um where people you know people can can do that

the funny thing is you know we're kind of skipping over the other thing another trend to talk about is just like I've always been a big fan and believer in wireframes because for me that was the quickest way to kind of get your ideas

into a solution UI like a low fidelity one for understanding with your team and stakeholders, but two, so you could have initial high level discussion with customers with tools like Sketch and

Figma because that's the tool and most designers, they go straight to high fidelity. And so we're kind of missing a

fidelity. And so we're kind of missing a lot of times you miss that high level discussion about the flow and things like that. The funny thing is some of

like that. The funny thing is some of these these design tools I was talking about, not the lovables and things like that, but the other category of tools like the visily and the wizard, some of them actually offer a wireframing mode

where they actually like deliberately stay low fidelity so that you can kind of work through those issues if you want. When should people stay

want. When should people stay deliberately low fidelity?

I think that you know it's it's interesting now because with vibe coding it changes the calculus that it used to be before it's like hey the presumption was it would take a lot of effort and the cool thing is when you do a vibe

coding it's usually a pretty plain vanilla UI it's not getting crazy with some crazy fonts and colors and stuff that are going to distract right so in a sense they're kind of low fidelity from

a visual design standpoint but they have to be high fidelity kind of they have if you're going to put a flow in there they have to be a flow but what I tell people usually is like if you're The number one

situation is one of the top situations is this happens a lot. Say your team is trying to figure out the scope of your MVP and there's one feature that's on the bubble. You know, half your team

the bubble. You know, half your team thinks it needs to be in there for the MVP. This is going to be a horrible MVP

MVP. This is going to be a horrible MVP without it. The other half of your team

without it. The other half of your team thinks no, we can get by without it. And

usually it's not the team. Usually the

team's cool without it. It's some senior stakeholder pounding on the table saying I can't believe you're not going to have this feature or sales, I can't believe you're going to ship without, you know, it's going to be horrible. So in those situations, that's when I like to in the

interest of time quickly create a wireframe without the feature and then say to the stakeholders who's pounding on the table, hey, I you're so convinced this is a critical feature. I bet you that if we talk to 10 customers, you're

gonna you're going to expect that like eight or more are going to complain about where the heck's this feature, right? And they'll usually say, "Yeah,

right? And they'll usually say, "Yeah, you're right." I'm like, "Okay, cool.

you're right." I'm like, "Okay, cool.

Tell you what, we'll do a quick wireframe. Well, go talk to 10 customers

wireframe. Well, go talk to 10 customers and if if like say five or more complain, we'll be the first ones to say you're right and we'll put it in. But

can you agree that if like you know uh you know not like only one or two complain that we'll be okay without it.

And that's how you set up the rules to try to get progress without taking the hit on that. So wireframing in the past was a great solution to do that. And

then you prototype it without the feature and you see how many people go, "Hey, where's this feature?" You know, you don't ask them if they want it. You

see if they complain if it's not there.

So that's the idea. Theoretically with

vibe coding, you could kind of get to that same point as well. The other use case is if you have like a very complicated flow like a multi-step workflow with many branches and

conditionalities and things like that, um you might want to kind of do some UI exercises with wireframes to kind of validate the flow and things like that, right? Um the interesting thing as I

right? Um the interesting thing as I said that out loud is if you have a lot of conditionality a live prototype then might be able to handle that much better than you know a clickable prototype because a clickable prototype you've got to kind you can't put like an if

statement in there right you've got you click on this it's going to go here you're going to click on this it's going to go there but so so I think that's typical uses of wireframe is and also just to get the team on board so I do

think that live live vibe coding prototypes can um replace some of that stuff um for teams but I do think it's important to kind of at least think through and that's, you know, we didn't

get into when you're inputting a vibe coding tool. If you want to get a good

coding tool. If you want to get a good UI or UX out of it, you need to specify those things, right? You need to be like, hey, this is going to be a five-page flow. Page one should have

five-page flow. Page one should have these things. Page two, three, four,

these things. Page two, three, four, five, right? You need to tell it the

five, right? You need to tell it the flow. If it's going to guess the flow,

flow. If it's going to guess the flow, it's probably not going to be great. Um,

yeah. So, I think that low starting low fidelity with your thinking is still important. Still thinking about, okay,

important. Still thinking about, okay, well, these are the the screens that we want. These are the conditional if

want. These are the conditional if statements that we want and potentially using a vibe prototyping tool to exercise those low fidelity. You know,

before we had to draw those boxes on napkins or draw them in balsamic. Vibe

coding can get that to you pretty quick.

Yeah. So, it sounds like don't skip the low fidelity step just because we can go high fidelity. Um, and then potentially

high fidelity. Um, and then potentially what I think people miss with the vibe coding tools is always input a screenshot of your product. always give

it your design system so it looks like your product, but you don't need to jump to that right away. Right. Right. Yeah.

I mean, the thing is it's tough. So,

back to that iceberg. If you tell it to vibe code something like I did when I did it in like five minutes for that thing two days ago with that team, it of course put in some like, you know, alert

icons, notification icons that were red and this and that, you know. So it it it kind of can't help itself from doing some kind of visual design and high fidelity and then people can fix on it

fixate on that and say oh that's a cool feature and so you know another general principle it's it's not so much low fidelity is when you're designing something like if your page has different components on it or widgets on

it like what is the goal or point of each of those widgets right that's the key right so if you're creating some like dashboard that's going to have four widgets on it if you were just to kind of leave that like a Slack variable for

the vibe coding say hey I want a dashboard with four graphs it's going to pick four random graphs right but if you say hey I want a dashboard with four graphs the graph one should really show user engagement over time graph two

should you know whatever you know so it understands the intent this happens a lot where people just design things and that can happen when there's not a good handoff between PM and design where the designer is just like kind of making

stuff up for the design um so it's really important when you do UI design that you understand what problem is this solving or what's the goal of each UI component what's the goal of this page, what's the goal of this component on this page? You know, things like that.

this page? You know, things like that.

The other thing it can really help with is creating realistic copy. So, like

that's where it can really save you time as well. Like, you know, back in the old

as well. Like, you know, back in the old days, designers would put like Lauram, Ipsum, Delore, like just placeholder text. We got that that ship sailed a

text. We got that that ship sailed a long time ago. And but, you know, if you're a designer, you got to put in some data. The the you know, those five

some data. The the you know, those five coding tools are really good. Like I had it create a dashboard to track a VC startup portfolio. It created like one

startup portfolio. It created like one AI startup with a cool name. It created

a health tech startup. It created, you know, created like multiple startups with good names in different spaces and and just kind of autogenerated some data that I didn't even tell it to do, which was pretty cool. Yeah. So, you can use

it for better placeholders as you're going higher fidelity over time. So, one

of the things we've we've basically narrowed in on here is that there are going to be a lot of PMs or even engineers at startups that are going to be getting more into the design and user

research process. And so, I want to

research process. And so, I want to educate them. Talk to us a little bit

educate them. Talk to us a little bit about these timeless principles. How

should people be doing user testing?

Definitely. And that's what I'm another trend I'm excited about, Akos, is I think in the last, you know, six, seven years, people have realized the importance of user research. You're

seeing more UX research jobs. There's

research conference a lot more research conferences. The value of it, you know,

conferences. The value of it, you know, companies like Dovetail, user testing is obviously huge. But it's really exciting

obviously huge. But it's really exciting and for me, I was I was very lucky. My

first PM job at into it in it valued market research so much that we actually had a PhD in market research on our Quickn team. And so I learned from her

Quickn team. And so I learned from her how to do best practices in qualitative research and survey design and all this jazz. And so it's cool to see a lot of

jazz. And so it's cool to see a lot of people want to do it, but to your point, they may not know how to do it. The cool

thing I will say at the beginning is it's one of the easiest things to do.

It's just talking to other humans. Um,

you know, and I'll give you some tips on how to do it in a sec. But why are we doing it? So you can see here, this is

doing it? So you can see here, this is my kind of learning or iteration loop that I like to use. Um, a lot of people may be familiar with build, measure, learn from lean startup. That was cool because it got people realizing, hey, we

need to iterate. We're going to build, we're going to learn, and we're going to iterate. Um, couple things I don't like

iterate. Um, couple things I don't like about it. starts with build and at least

about it. starts with build and at least until vibe coding came along that was a very expensive step right and if you got if you got to go into production code it's still an expensive step so I like to start with hypothesize that's what we

do we form hypothesis we design ways to test those hypotheses in the product world they're often prototypes and then we go out we test it with customers from the customer test we gain learning and

then from that learning we revise our hypothesis and then we go through the loop again we revise our prototypes you know we go around and I've seen this work so Well, um, if you do this methodically, you can really like

iterate your product and and achieve higher and higher product market fit.

And so part of the key component is once you do have the prototype, however you created it with your designer, with one of these Vibe coding tools that once you have at the top of the process, it's time to go and talk close the loop with customers and go talk to customers. And

so, you know, when you have your prototype and you want to test with people, there are fundamentally three highlevel different ways of testing it.

The first is in-person moderated where you actually in person with the person next to them you know with the prototype on their device the browser or their uh

you know their mobile mobile phone. um

that used to be very common obviously with COVID and Zoom and things like that and the convenience it's it's much more common to do the second one which is remote moderated and the this is remote

moderator is totally fine right back in the old days when the screens weren't as good the tools weren't as good the bandwidth wasn't as good the technology could get in the way sometimes but these days with Zoom you know and a and an online prototype it's pretty easy to

have the person share their screen and and conduct the interview and then finally is remote unmodderated what does this mean this means that there's not an actual moderator there. So, this is like a recorded session like user

testing.com. And the way this works is

testing.com. And the way this works is you get your prototype set up and then you ask people like to complete a series of steps or tasks and you might try to ask them for feedback after each step or

things like that. But you kind of really need to put a lot of thought into that script because people may get stuck, they may get off script, they may you know they may may follow it, they may not. So the advantage of remotear is you

not. So the advantage of remotear is you can do it quickly and I think that's another interesting area. Kash seeing AI may come in to try to like do that more

and try to like automate insight gathering you know like if if hey go take this web code prototype take this script go you know here's a recruited

set of users go do unmodderated with them have them schedule it have them run it syn do text to you know do you know speech to text do analysis on that and give me the data that we are not quite

there on that but that's a future thing I could see happening But I will say that that is the advantage of that is kind of getting a lot of data quickly.

The disadvantage is you can't ask why, you can't clarify. And so, you know, the moderate at the end of the day for me is much more valuable than unmodderated.

But unmodderated can be good, especially for usability issues. If you know you've got some usability issue on a certain page, but you're not sure why, you can point a bunch of people at that flow and try to figure out why. I like to talk to

people in waves of five to eight. You

know, there's a lot of old school usability research about that kind of a number, you know. um even if you talk to 10 or 20, it's still not going to be like statistically significant. But

after five or eight, you usually start to get diminishing returns. You've

identified the main things that you're going to learn there. So that's how I like to do it. And then basically you take your prototype and you test it with people in ways of five to eight. So when

should people be thinking about what's the flowchart for if this then inerson moderated, if this then remote moderated? How do you think about the

moderated? How do you think about the pros and cons of these different options? Yeah, I think that the earlier

options? Yeah, I think that the earlier in your process and the earlier you are in like creating this new product, the more you should lean on moderated because you don't even know what questions to ask. You don't even know how to write the script. Right? This is

a mistake I see people make in user research. It's this allure of, hey,

research. It's this allure of, hey, instead of talking and doing a one-hour interview with 10 users, that'll take me 10 hours. Why don't I do a survey of a

10 hours. Why don't I do a survey of a 100 users? Sounds great, right? The math

100 users? Sounds great, right? The math

sounds great, but the problem is you're not going to get the same quality of data. You can't ask follow-up questions.

data. You can't ask follow-up questions.

you can't see what's going on, right?

So, um I think in general when you're trying to achieve product market fit, moderated is much more valuable in the specific use cases of okay, we know we

have a usability, we have a product that's live and we have a usability issue. Let's go do remote unmodderated

issue. Let's go do remote unmodderated to figure out what how where people are getting tripped up. That's one specific use case. Um,

use case. Um, you know, I guess you could theoretically if you were late in the process, say you like work through a lot of stuff and for some reason you just wanted to get a lot more customer

feedback quickly, then you could do remote unmodderated on like a mature, you know, mature live prototype just to kind of get some see if there's other feedback, right? If you worried maybe if

feedback, right? If you worried maybe if you're worried about browser compatibility, you wanted to test on a range of browsers or a range of devices, that might be something like that. So

the general theme there is kind of like yeah testing different conditions to understand what's up. So in general I'm a big fan of moderated and uh I would only use unmodderated in those kind of

specific use cases. Um

what do you lose in a remote unmodderated interview versus an inerson? Well you you know that's a

inerson? Well you you know that's a great question and they used to you used to lose more because like I said before like the tools weren't as good. So

sometimes like this that the prototype was bigger than the screen. So they had to scroll up and down. They couldn't see the button. You give you an inter

the button. You give you an inter technical issues like the connection.

All those things are gone nowadays. You

know, you just have a zoom and you're good to go. So the last thing you really you lose is when I am sitting next to you, it's very subtle, but when you get good at user research, when I'm sitting

next to you, above and beyond whatever I see on Zoom, I can pick up on little subtle things like you might sigh or you might tap your hand or might be some other non-verbal body language that's

letting me know, oh, you really like this or oh, there's a problem or you're concerned or you're confused. I can kind of see where your eyeballs are going, too. It's very subtle, but you know,

too. It's very subtle, but you know, it's like that extra little 10 15% of information fidelity. Um, it's not

information fidelity. Um, it's not critical. It's not critical by any

critical. It's not critical by any means, but if you're a talented researcher, that helps give you an edge and, you know, you can pick up on things a little bit. Yeah. I always go back to

some of the best research I've done was when I was working on Fortnite and we always brought people in to our campus to actually watch them play because you

learn so much more especially when it comes to a game like are they throwing the controller? Are they rage quitting,

the controller? Are they rage quitting, right? We don't want rage quitting or

right? We don't want rage quitting or are they having so much fun? They're at

the bridge of their seat, right? You

might be able to understand that over zoom, but the amount of texture you get is like I would say like 50%. Like you

get a lot less texture there. I Yeah,

texture is a good term for it. Two other

thoughts I have. One is I forgot to say like I you know I started my PM career into it which was like a pioneer in usability testing. We had like this you

usability testing. We had like this you know back in the day a million dollar usability lab. It had like a one-way

usability lab. It had like a one-way mirror. It had like a camera on the

mirror. It had like a camera on the person's face. It had a camera on the

person's face. It had a camera on the screen you know back in the day. So you

like you know but it's funny because that's a little bit like a lab rat thing. So they're coming into your

thing. So they're coming into your thing. The other thing it made me think

thing. The other thing it made me think about Akos which we also did which is also important is are you watching them use the product in the natural environment like in situ right yeah and so the big thing that in it did was they

would do follow me homes like back in the old days it was like a box shrink wrap software you'd buy at the store so in it employees would like camp out at fries and best buy and they like you know when someone checked out be like hey I know this is kind of weird but can

I follow you home and watch you install quicken and see and then you realize you know like oh their disc drives over here or the printer's over here and they have internet issues like you know what I mean? So so that's the other thing you

mean? So so that's the other thing you lose if you're there's it's funny I guess with an inerson moderate there's inerson moderate in the lab and there's in process in person moderate in the field. Yeah. And so I do think for

field. Yeah. And so I do think for certain product types in the field observation is is very relevant. I could

totally see for Fortnite that getting closer to what it's like in the real conditions would make sense. So really

there's a hierarchy that we've evaluated. The best type of user

evaluated. The best type of user research you can do is you know truly ethnographic. It's in person in

ethnographic. It's in person in situation in the field. Then there's

inperson moderated in your lab. Then

there's remote moderated and then there's remote unmodderated. And

probably people are also talking about synthetic AI feedback at the very end.

Right. So there's all of this feedback that you can get. And it sounds like the earlier your product is the more you want to go up. So probably the earlier your product feature is, the earlier you

are in the discovery process, the more you want to be higher up in that cycle.

and probably also the more important the feature right if it's a big bet then you want to get higher quality research I would qualif the way I qualified is the

degree of uncertainty or like the uh the degree of confidence you have you know if you if your product if you're the market leader in a segment and you know the market super well and you know the feature set super well and you're just

launching some new incremental enhancement and everybody on the team feels super confident about it cool you probably don't need much research but if it's like no we're doing a new product for a new market that we don't really

know, then that's where you want to do more discovery, you know, and more in person. So, a lot of PMs, they'll be new

person. So, a lot of PMs, they'll be new to this, like UXRs and designers have been doing this. They haven't been doing this. So, can you walk us through what

this. So, can you walk us through what is the timeline of a user testing session? Yeah, I mean I and that's and

session? Yeah, I mean I and that's and that's the thing. My book's called a playbook because it's intended to give people like follow these steps.

Obviously the details are different depending on what your product or market area is, but I tried to put a lot of practical tips that I, you know, learned along the way. And I mentioned I started, you know, doing user testing at into it where it was like this

million-dollar lab. I went straight from

million-dollar lab. I went straight from into it to like a early stage startup.

They didn't have a lab or anything. And

so I coined the term ramen user testing because you don't need all the fancy stuff. It's basically like you, the

stuff. It's basically like you, the user, and their laptop or their device and then the prototype and that's basically it, right? And so just to kind of give people a starter skeleton of

what the timeline, how how to run this thing. Um this is this is what I

thing. Um this is this is what I recommend. I you know I don't like to

recommend. I you know I don't like to just jump right in the prototype for a few reasons. One is um you know talking

few reasons. One is um you know talking to someone to share your opinions about some new product you may or may not have used, especially if it's a new product concept. It's not an interaction people

concept. It's not an interaction people have every day. It's kind of weird and especially people that don't like to talk to strangers and stuff. You're

like, "Hey, can you you know?" So there is value in in just getting warming up the person, letting them know that you're a fellow human who's not going to be, you know, rude to them and you're nice and you're ask them about their day. You're talking to them about what's

day. You're talking to them about what's going on just to build some rapport. The

more rapport you can build, you're kind of investing in the gas tank for later, you're going to get more returns out of it, right? And then also, this is where

it, right? And then also, this is where I like to just start asking, "Hey, how do you get this need met today? What are

you using? What have you tried before?

Why do you use the one you use? What do

you like about what you don't?" I get hired by clients all the time to do to apply my lean product process. I almost

always learn about some competitor they didn't know about in this discovery research at the beginning like oh we never heard of those people right or you also might find out that it's not like some other software competitor but like oh yeah I just do that with Excel or CSV

export or duct tape them you know it's like whatever the MVP substitute is. So

then once we've done that um then I like to just spend you know whatever the appropriate amount of time is you usually at least 30 minutes maybe more if it's a big robust prototype where I show them the mockup or product right

and then the thing is uh this is a mistake I see you know some some PMs do that aren't as familiar with this is they try to guide the user

through the thing. It's like, okay, first I want you to register, then they register. Okay, now I want you to go try

register. Okay, now I want you to go try to book a hotel. Now, you know what I mean? Like, they're kind of like

mean? Like, they're kind of like directing the user and the thing that occurs to me like that is not the real world. The real world is the person's

world. The real world is the person's going to be sitting there with your product in the browser on their phone and they have to figure it out. Yeah.

So, I like to be as real. I just shut up. I basically like, hey, here's our

up. I basically like, hey, here's our new product. I'll pull up the prototype

new product. I'll pull up the prototype on the laptop. I'm like, you know, I'd love to, you know, I'd love to see you, you know, interact with it. And then

what I do is I give guidance. I'm like,

hey, you know, like most people um you know, it's I call it think aloud protocol. It's on the next slide, but

protocol. It's on the next slide, but basically most people are not used to sharing their thoughts out loud with a stranger when they're looking at some weird new product concept, right? So,

you have to be like, "Hey, share your ideas." And so, I'll be sitting there

ideas." And so, I'll be sitting there quietly and sometimes it's very it's like this awkward silence and they're like, "So, do you want me to register?"

And then I just give my patline. I'm

like, "Do whatever you would do if you're home with your with by yourself with this product on your computer." And

they go, Okay. So, so I'll register then. I'm like, do whatever you do. And

then. I'm like, do whatever you do. And

they realize, okay, it's not going to be a test or I'm going to guide them through the rat maze. They got to figure it out. And so, they'll keep going. So,

it out. And so, they'll keep going. So,

you may be wondering, well, shoot, I need to sit there quiet the whole test.

No, what I do is I wait to observe something noteworthy to pounce on. And

that's where being in person, that texture you said, that's where you might pick up some sigh or some little thing where you can tell they're like struggling. Obviously, you can tell if

struggling. Obviously, you can tell if their mouse is looking around. They're

looking around. They can't find where to go. So, I let them, you know, I let them

go. So, I let them, you know, I let them flail for a little bit. And then I'll be like, "Hey, can you tell me what's going on?" And they'll be like, "Well, you

on?" And they'll be like, "Well, you know, I'm trying to find where I can sort by blah blah blah." And then I go, "Oh, okay." You know, can you tell me

"Oh, okay." You know, can you tell me about that? So, I jump in and we talk.

about that? So, I jump in and we talk.

So, I don't know. I'll priori ahead of time what we're going to talk about, what's going to come up in the user session. And I'll we'll share in a

session. And I'll we'll share in a minute if we have time here, a framework for how to capture this information systematically. But when it comes up, we

systematically. But when it comes up, we go deep dive on those things. And then

we kind of go now if at the end. And so

then I don't break character. I want to be it like like I'm not even there, right? And I ask questions. Now, at the

right? And I ask questions. Now, at the end, the very end, the wrap-up. It's

important to do a wrap-up. You know,

they might even be like, "So, I was trying to click here. It doesn't work.

Is this broken?" You know, I'll be like, "Sorry, I can't answer that." You

hopefully the engineers are watching it.

If it's a live product, you want the engineers there, too. So, they feel the pain. But at the end, I'll be like,

pain. But at the end, I'll be like, "Hey, you know, when you're trying to do that thing, yeah, that's our bad. That's

a bug. Sorry about that." And then if for some reason, like there was a key screen or page they didn't get to. I'll

be like, "Hey, I want to just let me orient you to this page. Here you go.

Can you please give me your feedback?"

Right? just to make sure you get that.

And then at the end, I found it really important to basically ask people, you know, how likely would you be to use this product? Because I learned the hard

this product? Because I learned the hard way that like usability feedback, usability, poor usability can prevent you from having product market fit, but just because you have good usability does not mean you have product market

fit. They are orthogonal at the end of

fit. They are orthogonal at the end of the day. One can block the other, but

the day. One can block the other, but they're orthogonal because I was working on a product and you know, I tried to do my best on UX design. We did the V1, we launched it, ran some user tests, there

were some UX issues, there were some bugs, there were some like we need more instructions, we made examples, some copy needed to be fixed. So after and we were really fast startup. So after about

the 15th test, we had fixed all the questions and complaints that had been brought up. So starting about the 15th

brought up. So starting about the 15th user test, people were just getting through the new user flow. They were

getting to the main page, no issues, no complaints. So I was feeling pretty

complaints. So I was feeling pretty good. I'm like, "All right, cool. So how

good. I'm like, "All right, cool. So how

likely would you be to use this?" And to my surprise, like about 20% of people said, "I would never use this product."

Even though they had not complained, issued a single complaint, right? And

what it turns out is there were different mental models for how to solve this problem that I didn't know about.

And so once I discovered that, then now in my discovery research going forward, I ask people, hey, can you tell me how you like to get your news? And I

realized, hey, there's three different ways people like to get their news. And

we had kind of like subconsciously designed it for the way me and my co-founder like to do the news, right?

and then uh we realized it's not going to be the case. So, so it's a good reminder, you know, that you need to the usability if you know what I didn't say about the pyramid is any one of those layers if it's off really off is going

to prevent you from achieving product market fit. They don't have to be

market fit. They don't have to be perfect, but if you have a bad UX, it's going to do that. So, that's the thing.

What are the dos and don'ts of these interviews? Some dos and don'ts. You

interviews? Some dos and don'ts. You

know, you know, the other thing is most people remember what their mom told them. If you can't say something nice,

them. If you can't say something nice, don't say anything at all. like they're

very nice and and so they're very positive about your prototype like yo this is great oh this is great great great right so you what you need to do is like let them know and I can do this as a consultant like hey the whole

reason we're getting feedback is we want to know figure out how can we make this product better for you and other customers so if we go through this whole test and you haven't told us anything we can make better that actually doesn't help us out so you're helping us by

telling us how we can make this better for you and by the way I don't even work at the company I'm a consultant so you're not gonna hurt my feelings like please don't worry about hurting our feelings By telling us constructive criticism, you're actually helping us.

And then I mentioned the think aloud protocol. So most people I like to prime

protocol. So most people I like to prime them at the beginning and say, "Hey, I know this is not typical, but as you're going through the prototype, could you just try to verbalize the thoughts that are going through your mind?" And you usually have to remind somebody because

they start just quietly clicking and doing stuff like, "Hey, can you let me know what's going through your mind as you go?" So that's the idea. Um, and

you go?" So that's the idea. Um, and

then it's really important to take notes. um because there's so much if you

notes. um because there's so much if you were taking notes and paying attention there's so much information you can extract even from one user interview and different people um are going to see different things and that's why it's

also important to do like a review right away like you might line up like I like to like line up a power day of like three to five user interviews in a day we go through them all and then we have a team meeting at the end and we debrief

and and different people see different things now if you're uh not a super experienced moderator it might be very overwhelming to try to moderate the test and take your own notes. So, it's

perfectly fine to have a separate notetaker if that's kind of the situation they're in. So, those are the dos. What are the don'ts? What are the

dos. What are the don'ts? What are the big mistakes people are making in user interviews? One is that they try to help

interviews? One is that they try to help the user get through the user test and have a good experience, right? And I

understand if you're the PM and this is your baby, you're kind of secretly rooting for the user. Um, but you know, again, your product needs to stand on its own. So, you want to be a scientist,

its own. So, you want to be a scientist, an objective realist, and try to detach yourself. Um, I've seen some PMs that go

yourself. Um, I've seen some PMs that go as far as to like put their hand on the person's hand on the mouse, move the mouse, and click the button for them.

Like, it doesn't count. Like, you know, if you're doing it for them, it doesn't count. Um, you obviously don't want to

count. Um, you obviously don't want to get defensive or blame the user. That

usually doesn't happen, but every once in a while it does. And then the biggest thing is actually how you ask your questions. And so, like, you know, at

questions. And so, like, you know, at the end of the day, if you're if you're a PM or a developer, designer, and you're like, you know, I really want to do X design, but I'm just feeling nervous about it or not confident, I encourage you to try. It's one of the

easiest things to do in the product world. It really involves active

world. It really involves active listening, right? You just let the

listening, right? You just let the person talk. You can reflect back what

person talk. You can reflect back what you heard to probe more deeply. Ask why,

you know, why is that? Things like that.

But the biggest thing is how you ask your questions. And the difference

your questions. And the difference between leading or closed it in questions. So if I say, hey, that was

questions. So if I say, hey, that was easy to use, wasn't it? That's an

example of a leading question. It sounds

like a question, but it's not really a question. I'm putting the I'm really

question. I'm putting the I'm really pressuring the person to give a desired answer, which is yes. So, you definitely want to avoid leading questions. The

other thing is not as bad is a closedended question. If instead I said,

closedended question. If instead I said, "Did you like that feature?" What are they going to say? They're probably

going to say yes or no. I'm secretly

hoping they please say yes. Please say

yes. You know, because I'm rooting for the thing. But say they say yes. What

the thing. But say they say yes. What

did I actually learn? I didn't learn why they liked it. Or if they said no, I didn't know why they liked it. Right

now, in normal humor conversation, human conversation, yes or no questions are great, but you want to avoid those kind of questions. And basically, it all

of questions. And basically, it all comes down to how you start your questions. If you start your questions

questions. If you start your questions with do you, did you, does that, you're going to get a yes or no answer. Are

you, was that, right? So, it's just a little Jedi mind trick to catch yourself and start using more open-ended things like how, what, why, things like that.

You'll get a lot more richer data from your customer interviews. Brilliant. And

how do you capture all these notes? How

do you capture all this feedback in a systematic way? And that's I that's

systematic way? And that's I that's something that people struggle with. And

so what I'll share here is how I is literally how I do it. When you do these tests, I like to categorize them in three highle buckets. You're going to get feedback on your features that are functionality like if they're missing

some functionality or this functionality doesn't work the way they wanted to or it's missing some key aspect. You're

going to get feedback on the UX design that that iceberg that we talked about.

And you're also going to get feedback on messaging. Right? If words don't make

messaging. Right? If words don't make sense to people or it means one thing to you but something else to them, you're going to get feedback. And so I I start out with a blank slate like this. And I

don't know what in the rows, each row is going to be kind of a material observation that I have. I don't know ahead of time what it's going to be. And

then at the end, I like at the end of the interviews, I like to just ask, hey, on a scale of 1 to 10, how valuable do you find this product? And on a scale of 1 to 10, how easy to use was it? And you

know that this is what I call semiquant or quant on qual. We don't have a huge sample size, but we can kind of get some indication and we can see if there are trends over time. And so I show I run my

first user test with my first user, right? Okay. And basically there uh this

right? Okay. And basically there uh this is what I learned. User one, they said, "Hey, feature X that was in your prototype, they thought that was valuable. Awesome." But they complained

valuable. Awesome." But they complained that we were missing key feature Y. So

we note those things down. And by the way, I use a little plus or minus for positive bullets and negative bullets.

They we noticed that they had trouble registering. So we note that as a UX

registering. So we note that as a UX deficiency, they happen to mention they like the hero image on our homepage. So

we put that in the messaging. And we

asked them to score value and ease of use. We got seven and five. Right. We go

use. We got seven and five. Right. We go

to the next user. Uh we move on to the next user and they bring up the same two feature set things that the first user did. So we're getting signal there. They

did. So we're getting signal there. They

didn't say anything. They didn't have as much problem with registration, but they didn't see the signup link and they happened to comment that they thought our design looked professional. So

that's a positive. In the messaging, they said they didn't understand our tagline. We got a seven and seven on

tagline. We got a seven and seven on value and ease of use. Right? We go

through that. We keep going through in this wave. I'll keep it at five to keep

this wave. I'll keep it at five to keep it simple. And as we talk to more users,

it simple. And as we talk to more users, we don't get new items coming up. we

just get additional you know people mentioning or not mentioning the problems that we've seen to date and then when we get to the end of the wave we can do this thing I'm talking about semiquants and we can say okay what

percentage of users brought up each issue and now we can use this to take a step back and say okay how do we need to iterate our feature set our problem

space our UX prototype to address this so in this case obviously we're going to have to add feature Y right maybe that was the debate the team had like hey do we need feature Y or not okay looks like we We need it after all. We need to fix

regge. If 60% of people having trouble

regge. If 60% of people having trouble with it, we need to fix that. Signup

link isn't visible, you know, and the tagline, you know, you can prioritize based on this, right? And so that's what you do. So that's so we can collapse

you do. So that's so we can collapse those five user tests into the feedback from a wave. And then we go back and we basically say, okay, let's go fix all those things. And with Vibe Coding,

those things. And with Vibe Coding, it'll be even quicker to have a new prototype that you can now go back. And

then now I'm just going to show you wave by wave. So in each of these waves,

by wave. So in each of these waves, we're talking to five day users. We're

capturing the issues and what percentage of people have it. So this is how this is the summary of the problems from wave 1. We'd love to see all those go to 0%

1. We'd love to see all those go to 0% in wave two. Let's see how we did. We

won. We run wave two.

And cool feature Y. Now we added it. No

one's complaining. That's missing.

That's great. Makes sense. Means we we nailed that. Check off. Good job team.

nailed that. Check off. Good job team.

However, now we're getting new feedback that hey, X and Y aren't working together the way they should or we didn't think about that. We just added Y. We didn't think about how it works

Y. We didn't think about how it works with X. And so that's a new issue that

with X. And so that's a new issue that we need to address. Looks like we fixed the usability issues with registration.

That's awesome. Signup link. You know,

we went from 60 to 40. Maybe that's some little progress. Maybe it's just noise.

little progress. Maybe it's just noise.

We still haven't really solved that. And

now, yes, we did add feature wide, but now we're getting new feedback that it's actually hard to use. It's too hard to use. And it looks like we fixed our

use. And it looks like we fixed our tagline issue. We got a slight increase

tagline issue. We got a slight increase in value and ease of use. 7 to 8 and 5 to six. You know, we go to the next

to six. You know, we go to the next wave. And then cool, we've we made X and

wave. And then cool, we've we made X and Y work together. Now we made progress there. We finally fixed the signup link.

there. We finally fixed the signup link.

Looks like we might have made some progress on feature Y usability, but not enough. And you know, now our value

enough. And you know, now our value needs to view scores up 97. And then we do one last round. And now we've kind of solved all those issues. That's kind of what that iteration through that loop looks like and how you kind of can

systematically capture feedback and in a way that your team can say, "Cool, let's address these items. Let's see how we did going to the next round." as you're iterating round to round and if you get like all greens like on wave four and you're getting high value easy v scores

you should be feeling really good that hey we've really validated product market fit and now we can either launch this thing or take it to production code

this is the work that PMS designers and user researchers really need to be doing right this is how you ensure that your features aren't just bloating up your

product with more unnecessary stuff that people don't need this is how you ensure that you're building a minimum lovable product. It is. It is. And you know

product. It is. It is. And you know what's interesting is a lot of places they value prototypes and they value getting user feedback, but the team struggles with

like what do we do with this? We we got we've got pages and pages of notes.

We've got tons and tons of videos of these sessions. How do we turn the

these sessions. How do we turn the corner? How do we synthesize it and make

corner? How do we synthesize it and make actionable? What how do we iterate from

actionable? What how do we iterate from here? And that's why I kind of like this

here? And that's why I kind of like this simple high level framework for teams to kind of be able to do that. So, we've

zoomed in on all of these changes that AI has created to product management. I

want to reflect on product management a little bit. Have PMS become glorified

little bit. Have PMS become glorified jury jockeyies? Has the role gone off

jury jockeyies? Has the role gone off track?

This is a complicated question. The

short answer I'll say is mostly no.

There are definitely some customer companies where that is the case. There

are definitely some companies where that is the case and PM is basically a waiter taking orders from stakeholders and customers and just turning around and feeding them to dev and Jira. So that's

they're definitely Jira jockey places like that. Um but that is you know not

like that. Um but that is you know not what you're seeing in the best companies out there. The other thing you can get

out there. The other thing you can get into is like you spend a lot of time in scrum, right? you're spending a lot of

scrum, right? you're spending a lot of time writing the stories, refining the stories, um you know, talking to developers about the stories in all the scrum ceremonies, right? And you might be on more than one team. So, you're

going to two sets of ceremonies or three sets of ceremonies, right? Uh you're

working with design. You're probably

doing some QA. QA is not sexy. So, who

does it? PMs, right? Status updates,

updating stakeholders, all this jazz, making slides, right? So you can get so sucked into that scrum apparatus that you you know you're wholly consumed by develop and you don't have any time for

discovery or defining and actually doing your job. So and I I hate to say in the

your job. So and I I hate to say in the best org though that doesn't happen right. So again if we look at the bell

right. So again if we look at the bell curve right the top quarter are not operating that way. The bottom quarter operating that way and the main factor I I always like to kind of syn kind of

figure out why diagnose why and it's not always just this but it's usually the staffing ratio. And if you just want to

staffing ratio. And if you just want to look at one metric is like how many on average, how many devs do you have? How

many PMs do you have? Divide the two. Or

even on a team, this PM has 10. I was

just at a company where a PM had like 12 14 12 or 14 devs. I'm like that poor PM.

How they're not going to have time to do any discovery or do anything but be a jury jockey, a scrum jockey because that's all they have time to do, right?

Yes. I think we have been overburdening PMs where product leaders almost take it as a point of pride. I've kept my PM or lean. We have a 14 to1 developer ratio.

lean. We have a 14 to1 developer ratio.

But we've made developers so freaking productive with cursor and windsurf. Now

they're turning out stuff so fast. If a

PM is supporting 15 developers, they almost inevitably become a jury jockey.

Yeah. And that's why the ratios are kind of messed up. And this actually this predates my book. You know, I've been speaking um to product managers since 2006. And so this is my original

2006. And so this is my original framework. I call it the four Ds.

framework. I call it the four Ds.

discover define design develop right? And I I won't go through the

right? And I I won't go through the questions. You can see the questions

questions. You can see the questions that the key questions you're trying to answer and derisk in each of those phases. But what we're talking about

phases. But what we're talking about here is a situation where because of staffing ratios and other things, company culture, not understanding the role of PM, they are spending like over 90% of their time in that develop that

final column, maybe a little bit working with design, but they as a result they are starving themselves and not spending time on discover and define which at the end of the day is where product managers

add the most value. Right? Developers

are going to develop, designers are going to design. If you're not out there understanding the customers and problem, then no one else on the team is going to be. And so it's a problem when PMs don't

be. And so it's a problem when PMs don't have time to do that. That's how you end up with a lot of products that are just feature factories that don't solve real customer problem. What's a product trend

customer problem. What's a product trend right now that's total BS?

Um, I think this happens anytimes there's some hot new hot new technology.

And so right now I call it like, hey, sprinkle some AI on your product. I call

it that, right? It's like people and I think that you know uh a year ago everyone was like just trying to jam AI in their product adding co-pilots all over the place just without any rhyme or reason. I think it's starting to settle.

reason. I think it's starting to settle.

There's still some of that going on.

slowing down a bit but the at the end of the day going back to our discussion about problem space solution space AI is a solution right just like you know crypto a solution blockchain's a

solution these are all you know um virtual reality augmented reality those are solutions and so the problem that can happen especially when there's a new big sexy one is it's like a hammer looking for a nail like how can we AI

this thing and uh and and and one a lot of product managers can't figure out how to AI it because you're trying to force a square peg in a round hole The other thing is you're not starting with a customer problem. So even if you do

customer problem. So even if you do manage to sprinkle some dust on there or integrate it somehow, it's not going to go. And I think we're seeing a lot of

go. And I think we're seeing a lot of failure of those early attempts at trying to just bolt on AI to existing products. Um whereas now you're starting

products. Um whereas now you're starting to see people starting from what problems can it really solve? Like a lot of these vibe coding tools are solving real problems or VI prototyping tools.

They're solving real problems. So, a lot of people will look at you and a lot of the creators tend to have the highest retention on these podcast and

they'll be wondering, well, Dan, he's a really successful book author, but I'm not sure, is that really the business of Dan? When you break down the pie chart

Dan? When you break down the pie chart of your revenue and how you're making money these days, what does that look like? Well, it's kind of funny when

like? Well, it's kind of funny when people think that a book makes you tons of money. So, if you publish with a

of money. So, if you publish with a publisher, it's kind of like signing with Hollywood or being Taylor Swift with your records or something like that. Uh, you make some money, but that

that. Uh, you make some money, but that is not the main uh pie slice of my of my uh thing at all. I mainly um spend time doing a few things. One is training

workshops, right? So, as PM has exploded

workshops, right? So, as PM has exploded exploded in the last eight years, um companies have said, "Okay, we we understand the importance of PM. We

hired a bunch of PMs. We want to train them now." And there aren't a lot of

them now." And there aren't a lot of great resource. There are more these

great resource. There are more these days than there used to be, but a lot of times they will hire someone like me to come in and I'll tailor the training for the team. Sometimes it's just product

the team. Sometimes it's just product management. One of my favorites is when

management. One of my favorites is when it's actually product plus design plus dev and we go through all these slides and more that I went through with you so they really understand because it's kind of funny. I like to use analogy that

of funny. I like to use analogy that product development is a team sport.

It's like, you know, the Golden State Warriors. I'm in the Bay Area. They

Warriors. I'm in the Bay Area. They

don't just show up at the court and go, "Hey, let's play some basketball. What

position are you going to you know, they know their positions, they know their plays, they practice them. But when it comes to product development, we just hire a bunch of random people, throw them in and building or Zoom and go, "Hey, go make product, right? We don't

talk about positions. We don't talk about plays. We don't practice." So

about plays. We don't practice." So

those workshops are very um experiential, interactive, a lot of exercises, things like that. I also love speaking at product conferences. And

before COVID, you know, I speak at a lot of conferences. I used to write an

of conferences. I used to write an annual list and I'd keep track of like how how are they growing? How many do we have? Where are the new countries that

have? Where are the new countries that are having them and then COVID hit and I stopped. So this year I was super

stopped. So this year I was super psyched on my Substack Dan Olson.substack.com.

Olson.substack.com.

I wrote my 2025 conference guide and I was excited to see there's more this year than there were before COVID and there's a lot of new conferences. So I

just was in San Diego for a first-time product conference. I'm heading to

product conference. I'm heading to Calgary for a first-time product conference. So a lot of first-time

conference. So a lot of first-time conferences. So workshops and speaking.

conferences. So workshops and speaking.

I also speak at private events if you know the other cool thing is as products emerge more and more companies are actually getting like their product team together for a day or a half day or two days for an off-site to talk about the

craft of product management like that's the thing we're so busy that when do you have the time to actually sharpen your axe and sharp add to your toolkit there's really kind of so so a lot of companies recognize that so I'll go and

speak or give workshops at those events too um but then the other thing that I do that I love and how this all a lot of this content developed is hands-on consulting Right. I used to be an

consulting Right. I used to be an interim VP of product for post seriesa a startup. So I did that for box in 2007.

startup. So I did that for box in 2007.

I basically was their head of product. I

did it for one medical group in 2009. I

did it for India. I've done it for a lot of companies and that's I kind of stumbled into being that interim VP of product back in the day. I did that in 2005 and a lot of people want to be fractional CPOS that day. Like I did it

back then. And that's accelerated my

back then. And that's accelerated my learning because instead of just being at one company, I saw a lot of other companies. So I still do that today.

companies. So I still do that today.

these days it's more of an advisory or consultant capacity, but I still like working with early stage ops. In fact,

if you're watching this, I'm super excited about AI. So, if there is an AI starter out there that doesn't have a header product, that would be a great fit to kind of play around and help out and help them achieve apply some of

these concepts. Um, and then those are

these concepts. Um, and then those are the main ways. And then I also I'm just a big believer in best practices, promoting best practices in community.

So for over 11 years now I've been running a monthly meetup here in Silicon Valley where I live um called Lean Product Meetup and I bring in top speakers. Marty Kagan's been there nine

speakers. Marty Kagan's been there nine times. You know I speak there a lot of

times. You know I speak there a lot of other people spoken of Jeffrey Moore Crossing the chasm. A lot of all the people that you know that are probably on your podcast and blogging and have

books have been on there. Um and so during COVID we had to take it virtual and when we came back from COVID we built up this great audience around the world. So I we do hybrid events now. So,

world. So I we do hybrid events now. So,

but that's just like a little side project. And there's one other side

project. And there's one other side project I do, which is I run an annual product conference with three other folks, co-organize it with them called the product leader summit. We've done it nine times. It's a small 120 person

nine times. It's a small 120 person invite only um product conference for product leaders because there aren't a lot of um events for product leaders.

So, yeah. And then I try to write on Substack. I try to do stuff on LinkedIn,

Substack. I try to do stuff on LinkedIn, you know, try to play around with five coding tools, stay current and and write thoughts about all that stuff. So it's

it's a pie chart with a lot of slices, but mainly workshops, speaking and consulting and advising are the main main components. How lucrative can that

main components. How lucrative can that be?

Uh I mean it just depends. It can it can be lucrative. It depends on how much you

be lucrative. It depends on how much you want to work, right? That's the thing about being a a solo person. I've been

doing it for 20 years on my own. You can

decide to dial up your time, dial down your time, you know, have flexibility.

So it really I don't think there necessarily has to be a limit on that.

And you know the other thing is you can decide do you want to like scale up and have a team to do some stuff do you want to stay solo things like that those are some of the decisions that some people get into and some people that have like

hired a team they go I don't want to do that I want to be solo so you know there's different it's interesting to see you know in the product world different people taking you know different approaches on how to you know create their own kind of product

business. If people want to learn more

business. If people want to learn more where should they go? Yeah, I have my main website is danheifyenolsen.com osn. Um I'm on substack dan

osn. Um I'm on substack dan olson.substack.com where I'm actually

olson.substack.com where I'm actually posting a series about product strategy if people are interested in that. And uh

and then I have a YouTube channel just uh YouTube.comdan Olson and on LinkedIn

uh YouTube.comdan Olson and on LinkedIn happy to I I interact mainly with people on LinkedIn. So hopefully also I'll see

on LinkedIn. So hopefully also I'll see you at one of these conferences coming up. Awesome. Well, thank you so much for

up. Awesome. Well, thank you so much for being here Dan. Thanks a lot Kosh.

Appreciate it. I really hope you guys enjoyed that episode. It would mean a ton to me and the team if you could please subscribe on YouTube, follow on Apple and Spotify podcasts, and leave a

rating and review. Those ratings and reviews really help grow the show and help other people discover the show. And

they help fund the production so that we can do bigger and better productions.

Can't wait to share the next episode with you. Until then, see you later.

with you. Until then, see you later.