Teresa Torres on continuous discovery in B2B, AI prototypes, and synthetic personas

50 to 80% of the time we're wrong.

We can't trust that feeling, right? Even

if it feels like this is the obvious right thing to do, I would say like in an organization big or small, ideas are like ice cream and discovery is like is like broccoli. It's an unfair fight. The

like broccoli. It's an unfair fight. The

vast majority of people are not going to pick broccoli over ice cream. But just

like with health, we see like we get better outcomes if we eat broccoli than if we eat ice cream. I'll tell you the biggest mistake people make is they try to change their organization.

It's true.

If you're an individual contributor on a product team, that's the wrong level to try to enact change. The level where you should try to enact change is with you individually. And then as we start to

individually. And then as we start to adopt these habits, other people in our organization start to get curious about how we're working. And that's when we

have the opportunity to influence. For a

product team in a B2B context to be set up to be successful, a few things need to happen.

Welcome to Siden podcast. Today is a special one. We're going to talk about

special one. We're going to talk about my beloved topic discovery. And I'm

honored to greet one of the most influential voices in modern product management and person who is demystifiing discovery process Theresa

Torres. For those who for some reason

Torres. For those who for some reason are not familiar with she's author of essential reading for product teams continuous discovery habits. She spent

years in trenches as product leader and now coaches product teams. So, if you ever struggle with building features that customers don't use or felt like

you're guessing on what to bit next, today's conversation is going to be incredibly valuable. So, I'm very

incredibly valuable. So, I'm very thrilled to have you today, Theresa.

Welcome to the show.

Thanks for having me. I'm excited to do this.

Yeah. To set the stage, let's maybe define the glossery for the beginning.

Uh we'll talk about discovery today. And

uh discovery helps us to find question what to build and risk the implementation. But there are different

implementation. But there are different types of discoveries like being uh working on design agency we mostly focus on initial uh product discovery and I've talked to my peers working on products

asking like what discoveries they run and it's usually also the finding the problem uh finding the pain points of different personas then planning the implementation scoping and working

through implementation and your continuous discovery is a bit different so can you explain what how it stands apart and what are the key uh elements that make it stand out.

Yeah. So, my continuous discovery framework was designed for outcome focused teams. So, they're not starting with a solution. They're starting with um they're not even starting with a customer problem. They're actually

customer problem. They're actually starting with an outcome. So, an outcome could be a business metric they're trying to improve. It could be a big strategic objective the company's trying

to reach. It's really just as a team,

to reach. It's really just as a team, what's being asked of us? What does

success look like? And then from there we're looking at so that's sort of business focused and then from there we're looking at what do our customers need and then and customer needs are infinite right we can spend our whole

lives trying to address customer needs.

So we're going to filter those needs based on our outcome. So that's why we start with the outcome because it tells us which needs we should pay attention to. Um so a lot of it is um starting

to. Um so a lot of it is um starting with an outcome, discovering the needs, pain points and desires of our customers that can help us reach that outcome and then discovering the right solutions.

Yeah. And what are the main criterias for this continuous discovery to work?

So first one as I understand that product leadership should delegate the authority and ownership for a product team to actually impact the product and what else? So the key concept with even

what else? So the key concept with even that first one you threw out is this distinction between empowered teams and autonomous teams. So some people hear like, "Oh, I'm being tasked to deliver an outcome. I should be able to do it

an outcome. I should be able to do it however I want." That's not quite true, right? We work in a business context. We

right? We work in a business context. We

work with other teams there. We need

alignment across our teams. We need alignment with our business strategy.

And so it's not the case that we just give a team an outcome and they can do whatever they want as long as they reach that outcome. Our leaders need to set

that outcome. Our leaders need to set not they need to not just tell us what outcomes are important but they also need to set the strategic context. So

mission, vision, strategic objectives.

Think about it as like the way that we work here, the types of things that we would do, the types of things we wouldn't do. Our revenue model is even

wouldn't do. Our revenue model is even part of that. How we sell our product that that gives the team guard rails. So

it's your job is to drive this outcome within the guard rails of that broader strategic context. And what's hard about

strategic context. And what's hard about that is in a lot of organizations that strategic context isn't very well defined. And then it means that leaders

defined. And then it means that leaders get frustrated because teams suggest things that they wouldn't consider doing. They violate one of those guard

doing. They violate one of those guard rails, but the guardrails are implicit instead of being explicit. And so it's not just that teams leaders have to delegate to teams. It's that they have

to set clear guard rails so that teams are empowered to make decisions that will fit in the broader business context. And that's often a very common

context. And that's often a very common gap.

Great insight. That's something that I lack right now in my project.

Yeah, it's very common and know a lot of it is common because our stakeholders have it in their heads like it's it's implicit but they forget how much isn't being communicated to the teams. And

then I think the other things that need to be play in place is the teams need access to customers. So in a lot of organizations sales or customer success often acts as gatekeepers to customers

whereas we we want the people that are building the product to have direct access to customers. Ideally we have good instrumentation of our products and we have good analytics. I know close to

like half of product teams across the world have zero analytics. That doesn't

mean they can't do discovery. They

certainly can. But definitely as you get more advanced in your discovery efforts having good analytics can help. We

definitely will dive deeper into getting access to customers because it's not only about users but actually like who make decision. But while we are on this

make decision. But while we are on this kind of like initial level, I wanted to understand what are the main obstacles for teams to implement that this

framework because first we can kind of look through the lens of startups or smaller teams that actually are like in position to have an ownership around the

product but still they do not follow the continuous uh discovery in terms of like finding the opportunities. What's the

main kind of hurdle or obstacle to implement that? Is it cost, timing,

implement that? Is it cost, timing, skills, experience gaps?

I think startups are actually the toughest environment to do this well, which is a little bit surprising. It is

very possible to do it in startups. But

I think what makes it uniquely hard in a startup is two things. One is most startup founders don't start with an outcome or a customer problem to solve.

They start with a solution. And they

have a very strong vision. And

especially if they're a first-time founder, they don't have the experience to know that often times our first idea is very flawed. And it takes a lot of iteration to get from a flawed idea to a

good idea. And so a lot of first-time

good idea. And so a lot of first-time founders have to go through this learning curve of realizing that like it's a little bit like the don't tell me my baby's ugly, right? Like there's a little bit of this discomfort they have

to grow through before discovery is really going to be effective. And even

if they hire people on their team or they hire a design agency, even if they're like doing all the right activities to understand is their product idea going to work, it doesn't mean they're receptive to that feedback.

And in my experience, most founders have to get knocked down quite a few times before they're receptive to that.

There's a reason for this. Like to start a company requires that you be a little bit crazy, right? And that craziness is what leads to startup success, but it also is what can get in the way of us

seeing the flaws in our products. And so

I think a lot of second time founders, definitely third-time founders start to learn this. Uh it's a little bit easier

learn this. Uh it's a little bit easier when you have an experienced founder. I

think the second reason why it can be harder in a startup is because you're racing to get to revenue before you run out of money. It makes it even harder to say to a customer who may not be your

ideal customer, um, no, we're not going to do that for you. And so what we see startups do is they get pulled in a lot of different directions unless they have a very experienced founder who's keeping them focused.

Yeah. And for those teams that are part of larger organization, we can assume that there is like there's delegation of authority and ownership or there are so

those feature teams that um kind of just work with customer requests. Uh do they have anything like in power to change the status quo? Yeah. So both in

startups and in mature companies like I would say across the board an overwhelming majority of companies are very solution focused. The organization

talks about ideas. They prioritize

ideas. They put ideas on their road map.

Like literally everything in the organization is geared around solutions.

And I think what's hard about that is um a lot of them are guesses, right?

They're not um they are not incorporating customer feedback. There's

not a clear understanding of customer needs. Sometimes there's not even a

needs. Sometimes there's not even a clear understanding of business needs, right? And a lot of this is just due to

right? And a lot of this is just due to human nature. Like in human nature, it's

human nature. Like in human nature, it's really easy to jump to solutions. We all

do it. It actually takes some discipline and some rigor to force ourselves to say, wait, why are we considering this idea? What problem are we trying to

idea? What problem are we trying to solve? What value does that create for

solve? What value does that create for the customer? What value does that

the customer? What value does that create for the business? So, this is a little bit of a discipline. It's like

telling people to eat their vegetables.

Like we all know we should do it, right?

But like we prefer ice cream. And so I would say like in an organization big or small ideas are like ice cream and discovery is like is like broccoli. And

it's it's a it's an unfair fight, right?

The vast majority of people are not going to pick broccoli over ice cream.

But just like with health, we see like we get better outcomes if we eat broccoli than if we eat ice cream.

Yeah. I don't know this analogy.

Yeah. And like the good news is I think we're going through a time period where a lot more organizations are starting to recognize that like we can't always just eat ice cream. We need to eat some

broccoli too. And some teams take it too

broccoli too. And some teams take it too far. Like they say we can only eat

far. Like they say we can only eat broccoli. And that's not a great life

broccoli. And that's not a great life either. Like you probably want to mix in

either. Like you probably want to mix in some ice cream. So like it's okay every once in a while to just build a stakeholder idea. Like that's not going

stakeholder idea. Like that's not going to break the world. But your question of like if I'm a working on a product team either in a startup or a big organization and like my company doesn't work this way, what should I do? I'll

tell you the biggest mistake people make is they try to change their organization. If you're an individual

organization. If you're an individual contributor on a product team, that's the wrong level to try to enact change.

The level where you should try to enact change is with you individually. So what

are the things that you're doing daytoday that could get you a little bit closer to a continuous discovery mindset? And this is uncomfortable for

mindset? And this is uncomfortable for people because it's really easy to think like I can't do this because those other people need to change. It's like we're giving up our agency. We're giving up

our sense of responsibility. But if we flip it and we say, well, what can I do?

It might not be easy. In fact, if your organization doesn't work this way at all, it's not going to be easy. But if

you flip it and say, well, what what's like the first teenytiny step that I can take? A mantra I use a lot is something

take? A mantra I use a lot is something is better than nothing. Is it actually is empowering. You're taking back your

is empowering. You're taking back your agency and you're saying I do have some control over how I work and all of us have some control over how we work. Like

I don't know a single human whose boss is standing over them all eight hours a day dictating what they do, right? So we

all have some control. And the reason why that works is that like first of all we do we have to work within our sphere of control. We do control a lot of our

of control. We do control a lot of our day especially as knowledge workers. And

then as we start to adopt these habits, other people in our organization start to get curious about how we're working.

And that's when we have the opportunity to influence. Whereas if we don't change

to influence. Whereas if we don't change our own behavior first and we just tell people you're doing it wrong, all we do is we get people on the defensive, they dig their heels and it turns into this ideological war. And it literally nobody

ideological war. And it literally nobody wins an ideological war.

For sure.

I really like the idea like start with yourself like lead by example. I'm also

curious on the human side. Is discovery

something that you can actually learn?

There's a lot of habits and skills related to discovery. So in my book, it's called continuous discovery habits and I outlined I think 13 different habits. Each of those habits requires a

habits. Each of those habits requires a lot of different skills. So when you're defining outcomes, it's a very analytical quantitative process. when

you're interviewing customers, it's a very qualitative empathetic like um EQ like humanto human process. Um when

you're experience mapping, there's like a visual element as part of it and requires some drawing skills. Um when

you're moving into identifying the assumptions your ideas depend upon, it's a very black hat or devil's advocate or disisconfirming questions kind of exercise. Um so there's a wide variety

exercise. Um so there's a wide variety of skills involved. I don't think there's very many people that are going to be equally good at all of those skills. And this is why we talk about

skills. And this is why we talk about discovery as a cross function functional team sport, right? Because your

engineers are probably better at asking disisconfirming questions than anybody else on the team. That's a very common pattern. And you want them helping to

pattern. And you want them helping to identify assumptions. Your designer

identify assumptions. Your designer probably, but not necessarily, is the best drawer on your team, and they're probably going to be drawing those experience maps for you. It's just one of those things where you want to look

at like where are your strengths and where are the strengths of your teammates and how do you as a cross functional team take advantage of each other's strengths and collaborate. So

not just farm it out and let one person take the lead but really use each other's strengths to come up with a better team perspective.

Now I would like to shift more into more practical sphere. The building block of

practical sphere. The building block of continuous discovery is repetitive process of customer interviews to collect stories. It really resonates

collect stories. It really resonates with me because my beloved colleague she repeatedly reminded us that we are not users and we need to collect the context about like to build empathy and I think

it really resonates with the designers but uh still I think collecting stories is a bit different how people do interviews right now and you mentioned

in some either product talks or in the book that our research is right now tiered toward validating rather than exploring. So can you kind of explain

exploring. So can you kind of explain what's the difference and how this type of interviews is different from what currently teams are doing?

Yeah. So I would say there's two key building blocks. One is interviewing

building blocks. One is interviewing customers and collecting stories. But

the second is assumption testing. And I

actually think these are both equally important. What we get from customer

important. What we get from customer interviews is we're uncovering unmet customer needs, pain points, and desires. We're learning about our

desires. We're learning about our customers. We're not exploring our

customers. We're not exploring our solutions. We're not getting feedback on

solutions. We're not getting feedback on our solutions. We're really focused on

our solutions. We're really focused on how do I learn about who I'm designing for, who I'm designing with. Ideally, um

where does our product fit in their daily lives? It's generative research.

daily lives? It's generative research.

It's helping us see and learn and understand goals, context, needs, etc. Assumption testing is helping us to evaluate our solutions. So once we understand the problem we're trying to solve and we're starting to design

solutions, we're running assumption tests to understand will this solution actually address this need? Now, you

mentioned one of the things that I do teach is for interviewing in particular, I teach a story-based format of interviewing. So, what that means is

interviewing. So, what that means is instead of going in and asking your customers, what do you like to eat for lunch? Um, we're going to ask them, tell

lunch? Um, we're going to ask them, tell me about the last time you had lunch.

And it sounds like a very subtle difference, but the first question invokes a very speculative response.

Your brain is not and not you specifically like all human brains are not very good at um summarizing our behavior or reflecting on our behavior.

A lot of this is like grounded in our biology. Like our brain's job is to keep

biology. Like our brain's job is to keep us alive. It's to keep our lungs

us alive. It's to keep our lungs breathing and our heart beating. It's

not to do all this mental work that we think it it is for, right? And so when we're asked a question like that, um our brain generates a fast answer, but that fast answer doesn't necessarily reflect

reality. So for listeners that are

reality. So for listeners that are familiar with um the book Thinking Fast and Slow, that's Daniel Conorman's book.

Daniel Conorman wrote about um multiple decades of his work with a uh Amos Tverki where they basically invented the field of behavioral economics. And

really what they uncovered was this whole suite of cognitive biases. So

mental errors that our brains make and in that book he introduces this metaphor of system one versus system two. System

one is our fast brain. It's that it's your brain is trying to spend as little energy on mental processes as possible so it can focus on keeping your biological systems working well. The

problem with that is that it's errorprone. It can be errorprone. So if

errorprone. It can be errorprone. So if

I ask you what do you like to eat for lunch? There's a number of cognitive

lunch? There's a number of cognitive biases that are going to interfere with your response. It's going to be things

your response. It's going to be things like um recency bias. So like what did you have for lunch yesterday or um your favorite lunch, right? but maybe you

don't have it very often or it might be colored by you recently had lunch with someone you really like and so that stands out in your memory. Right? So

there's a number of cognitive bias or it could it could be influenced by like what you hope you would eat for lunch every day. Right? So like your healthy

every day. Right? So like your healthy aspirational self as opposed to your real self. And so when we're

real self. And so when we're interviewing our customers, we have to be aware of this psychology and we have to learn how to ask questions that invokes memory instead of these like

fast system one responses. Now most

people on product teams are not experienced researchers. We don't have

experienced researchers. We don't have deep roots in qualitative research. And

so what I found is that teaching people to collect stories is a really simple tactic. It's still a skill. It takes

tactic. It's still a skill. It takes

practice. We do have to get good at this skill, but it's a simple practice that everybody can understand that helps to get much more reliable feedback. Now, a

lot of people don't do it because I think people don't respect interviewing as a skill. And even if they like read my blog and or read my book and say, "Oh, I got to collect a story." They'll

they'll make a mistake. They'll say,

"Well, what do you typically eat for lunch?" And they think they're

lunch?" And they think they're collecting a story, but as soon as I add in, "What do you typically eat for lunch?" We're now back in that

lunch?" We're now back in that speculative realm. So like this really

speculative realm. So like this really is you have to keep the participant grounded in a single instance. So what

did you eat for lunch today? What did

you eat for lunch yesterday? Right? And

this what this does is memory requires system two using Conorman's metaphor. We

have to stop and think and remember and as the interviewer there's a lot of techniques we can learn to help them remember, right? And then we want to

remember, right? And then we want to walk them through all the detail of the actual story. And this is what's going

actual story. And this is what's going to make sure we're learn we learn about people's real behavior and not like their aspirational behavior or what they speculate their behavior is. Like I used

to teach a workshop where I would ask someone to volunteer and like there was one example I think I included this in the book. I asked a woman tell me um

the book. I asked a woman tell me um like what is your criteria when buying a new pair of jeans? And she didn't even hesitate. She said number one my number

hesitate. She said number one my number one criteria is fit. And then I said okay what else? And she said well price and style. I'm like okay. And then I

and style. I'm like okay. And then I asked her, "Tell me about the last time you bought a pair of jeans." And she said, 'I bought them on Amazon.' And the whole room chuckles, right? Because how

do you evaluate fit? Her number one criteria on Amazon. And so I asked her that. I go, "Well, how did you know if

that. I go, "Well, how did you know if they were going to fit?" And she said, "Well, I bought the same brand I've had in the past." I said, "Okay, that makes sense." And then I said, "Have you ever

sense." And then I said, "Have you ever bought a pair of jeans that were the exact same brand and they fit differently?" And she said, "Yes." And I

differently?" And she said, "Yes." And I said, "Okay, so if fit," and any woman who's bought jeans has had that experience, right? if fit is her number

experience, right? if fit is her number one criteria. I was like, "Well, why did

one criteria. I was like, "Well, why did you buy them on Amazon?" Well, it turns out they were on sale. So, her primary criteria was getting a deal and her secondary criteria was the convenience of online shopping and her third

criteria was fit. But that's not what I learned when I asked her what's her criteria for buying a pair of jeans. And

this isn't unique to her. I've done this in literally every in-person workshop I've ever done. We see it almost every single day. We our brains are very

single day. We our brains are very errorprone when talking about our own behavior, but they're less errorprone when we talk about specific memories.

People are really bad at answering direct questions.

Yeah, they are.

Mhm. And uh I also like while on on on the topic about like deep uh understanding on the context uh since we were working with B2B a lot I always had

struggles with connecting and um like user uh feedback or kind of interviews from users connecting it to customers their need because they're decision

makers and how it translate to product outcome because like if we collect stories from users how does it then get m gets mapped to the customer

expectation on the value because they make the decision and they kind of switch and buy and so on and they we need to satisfy their expectation on the

value. So I think like this uh not only

value. So I think like this uh not only testing and that the risking value risk is the the hardest one and how to work it in the context of B2B. So for a

product team in a B2B context to be set up to be successful a few things need to happen. The first thing is their outcome

happen. The first thing is their outcome sets the scope of their discovery. Like

a lot of B2B is helping us do our work, right? So it's all about like how do I

right? So it's all about like how do I help you do your work better and that's a lot of our B2B teams are focusing on enduser work in the product. We often

also have buyer teams, right? We have

product teams that are focused on the buying journey and on the buyer specifically. So if you're on a buying

specifically. So if you're on a buying if you're on a buyer team, your customer is the buyer. If you're on an enduser team, your customer is the end user. And

the reason why I say this comes down to your strategic context is your outcome on the enduser team needs to be framed in a way where it's directly tied to a buyer outcome. We're not like in B2B,

buyer outcome. We're not like in B2B, we're not just trying to satisfy end users because it's fun. We're trying to satisfy end users because it's going to drive a renewal. And so part of that strategic context is there has to be a

connection between our product outcomes and our business outcomes. And so if for example you're on a B2B team and you're focused on like optimizing some process

that people do in their daily job there is a belief in the organization that optimizing that process is going to drive retention or optimizing that process is going to bring in new

customers who are struggling with that.

Right? So part of the strategic context is making sure that our outcomes even our product outcomes our enduser focused outcomes are supporting the overall business.

Mhm. And it's it's built on assumptions like you need to test those assumptions like this uh user outcome will drive business outcome.

Absolutely. And often often times the way that we test those assumptions is we d we try to impact that product outcome and if it doesn't in turn drive the business outcome then we need to look at

what's the next product outcome to try that might influence that. There are I will say there are some teams that span the boundaries, right? So there are some parts of a B2B product that influence

the buyer and impact the end user. And

those teams we usually put our most senior teams on those problems because they actually have a very complex ecosystem and they need to be interviewing both buyers and end users.

But if I had a B2B company that had like 40 product teams, I probably have three or four of those teams and everybody else is either end user only or buyer only. I wasn't in a position when our

only. I wasn't in a position when our product leader created AI powered prototype with like back end and stuff.

He demoed it to the customer. Uh he got positive feedback and uh to me it sounds like confirmation bias. He made an assumption. Uh he validated it and now

assumption. Uh he validated it and now it's impacted our road map. But I just struggle understanding if that's the right thing to do for product teams to

kind of validate with AI prototypes those hard decisions if it actually will convert the client on on the B2B context. So my first question would be

context. So my first question would be okay one customer liked it. What about

the rest of the customers right? Yeah.

So that's the first problem. We don't

build our products for one customer. We

build our products for the market. And a

big question product teams need to be asking like the way that I frame product work. What's the smallest amount of

work. What's the smallest amount of product we can build to serve it to service as much of the market as we can?

There's a reason for that, right? One,

we obviously want to serve as much of the market as we possibly can, but we don't want a big sprawling product because it's expensive to maintain. It

leads to a terrible user interface, right? It's hard for our customers to

right? It's hard for our customers to learn.

So really, it's balancing smallest thing we can build to serve the most people.

So already what I heard is we're reacting to one customer conversation.

The second question I would have is like, okay, we showed this customer a solution. Did they try it out? Did they

solution. Did they try it out? Did they

use it? Or were they just shown a demo?

So, we know from books like The Mom Test um and literally everything in the universe of like getting feedback from customers if we just show our shiny object, it's really easy for somebody to say, "Oh, that looks great." And that

could be they're just being nice. It

could be it really does look great, but until they try it and get their hands dirty, they don't see the warts. It

could be like it really does look amazing. Just like I really am going to

amazing. Just like I really am going to go to the gym four times next week, right up until I start to get busy, right? Like it could be that it's

right? Like it could be that it's tapping into that aspirational self and like they they wish they would use something like that but it doesn't mean they would. So there's a lot of

they would. So there's a lot of challenges with this and this is what AI prototyping enabled like this what you just experienced has always happened.

The way it used to happen is your boss would talk to that customer or whoever it was in the organization would talk to that customer and they just riff on ideas and they would talk about that idea and it would come to you as a vague

idea that you then had to turn into a product. What AI prototyping does is now

product. What AI prototyping does is now anybody in our organization can build something that looks like a product. The

good news is the more a customer sees something that looks like a product, the more likely they're going to see things that don't work for them. So odds are you're getting a little better feedback than you would have if they just stayed

in vague words. But I still don't think it's the right way to go. Now, here's

what I'm going to tell you. There will

be times where it makes sense to start with AI prototyping. Sometimes we're

going to build stakeholder ideas to build relationships. Like sometimes

build relationships. Like sometimes there are ideas that are obvious enough or simple enough or fast enough or earn a big account that it does make sense to

just move forward with the solution. The

challenge is it always feels like that's the case. And we know from data like 50

the case. And we know from data like 50 to 80% of the time we're wrong. Right?

So depending on whose numbers you believe, some people say 80% of their ideas don't work. Some people say 50%. I

personally believe it's closer to 80%, AB testing tools tell us it's a lot closer to 80 to 90%. Um, so

we can't trust that feeling, right? Even

if it feels like this is the obvious right thing to do. So that's where I'm starting to look at like, okay, well, how many customers are going to be impacted? How much work is involved?

impacted? How much work is involved?

What's the risk? Am I willing to just gamble and build it anyway and take on that risk, or do I feel like I need to mitigate some of that risk and do a little bit more discovery? I will share I love AI prototyping tools. In fact, I

just built a little utility on lovable last night in 20 minutes and it it's no matter how much I use these tools, it still feels like magic. What I like them for is when we're assumption testing.

One of the things that we teach in our courses is you want to prototype to simulate a specific moment. So, we're

starting with a very specific assumption. That assumption lives in a

assumption. That assumption lives in a specific moment in a story map. Our

prototype is just of that moment. Well,

we now can create that prototype in like three seconds. Vibe coding is amazing

three seconds. Vibe coding is amazing until it's not, right? For anybody who's played with these tools, like they're really powerful. You can create real

really powerful. You can create real stuff until the real stuff gets a little too complex and it breaks. But if you're using AI prototyping to create a very simple interactive mockup or prototype

of a single moment to test an assumption, they're great for that. And

so what used to take design work now takes like a sentence.

So, I love AI pro I love AI prototyping for two purposes. One, they're really great at creating little personal utilities. So, if like there's something

utilities. So, if like there's something in your workflow that you just want to automate, they're really great at that.

Like this idea of software for one they're amazing at. And then I think in the discovery process, they're really great for um assumption testing. Well,

there are two paths I would love to uh like first to dive deeper into situation when uh stakeholders make decision and just like tell us what to do next and another one AI. So maybe let's dive into

AI uh because you mentioned like using AI prototyping to drive assumption testing and there are other ways how AI impacts the discovery phase like there

is a post that I just like saw today about trends in AI and how it impacts the discovery phase and I will quote them and maybe you can react to that. So

first is learning by delivery over discovery.

Pwell Hearn's post. Yeah. So maybe you saw this post as well.

uh PM uh doing AI abouts and product trio blends. So uh can you dive deeper

trio blends. So uh can you dive deeper into those changes how they impact the discovery using AI tools?

Yeah, I wouldn't frame it as discovery happens in delivery. So first of all, discovery always happens in delivery.

It's not where I would start. Even if

you can just press a button and your thing is delivered, it's not where I would start. There's a lot of reasons

would start. There's a lot of reasons for this. Already with continuous

for this. Already with continuous deployment, we're overwhelming our customers. Things are changing all the

customers. Things are changing all the time. They're not always changing for

time. They're not always changing for the better. Even if delivery is free,

the better. Even if delivery is free, it's a very expensive way to learn.

You're rewriting product documents.

You're getting your support team up and running on the new change. Plus, it's

expensive for your customers to learn a new interface or learn a new feature.

Like, we don't want to incur all of that expense before we've done any discovery at all. So I do not no matter how even

at all. So I do not no matter how even if it's zero dollars and zero time to ship a feature I don't think we should learn in delivery. I mean we do learn and delivery but I don't think we should start by learning and delivery. So I

disagree with that. His second one can you remind me what the second one was?

Metrics emerge from data.

Yeah. So Pablo and I both took the same class on AI evals. The class was called AI evals for engineers and technical product managers. It's taught by HL

product managers. It's taught by HL Hussein and Shrea Shanker. It's

phenomenal. I strongly recommend it. And

this idea of metrics emerge from data comes from that. They teach um for AI products to look at your traces and to do error analysis and then that helps you identify metrics that you write

evals around. This is a really big topic

evals around. This is a really big topic and I know a lot of people are not familiar with this space. I will say to some degree our metrics have always emerged from data, but it depends on what metrics we're talking about. If

we're talking about our business outcomes, they need to be derived from how our company makes money. that is

what our leaders care about. They they

should be derived from your revenue model, not from an user behavioral analytics. Um, so I saw that and I

analytics. Um, so I saw that and I didn't I mean I kind of get it because I took this class with him and there is one context where that is true. Um, but

I didn't like it as a broad stroke. One

thing I wish Powell did a better job on and I'm he has actually gotten much better at this so I will give him credit for that is I wish that he attributed his sources. So, the product trio

his sources. So, the product trio blending, there was a speaker in our class. His name is Aman. I'm gonna look

class. His name is Aman. I'm gonna look up his last name really quickly. Uh, I

can't really I'll I'll maybe I can give it to you for the show notes. He gave a talk where he talked about product trios and um PM like what do PMs do in the trio, what do engineers do, what do

designers do, and how AI is forcing that to blend even more. And a lot of this is because eval is a is a um part of a LLM app development process that requires

both the product manager and the engineer. And it a lot in the early days

engineer. And it a lot in the early days which where we are right now a lot of PMs are learning the engineering work and a lot of engineers are learning the PM work because sometimes it's easier to go faster for one person to just do the

whole process. I did this myself like I

whole process. I did this myself like I took the class I learned the engineering work through the evals and I am now doing both the product work and the engineering work on my own AI product.

Um I have for two decades now argued that the roles in the trio should blur way more than they do. Um I don't think this is a new trend at all. Like I think

the the more your roles blur in a trio the better the better you're going to work together. Um so I don't think

work together. Um so I don't think that's a change. I agree with the sentiment. I don't think that's a

sentiment. I don't think that's a change. Um, and I would love it if you

change. Um, and I would love it if you would include a link to to Aman's work in your um, show notes. And then the the one about PM should be doing eval. Uh, I

100% agree with that and I'm actually going to be doing a lot more in this space to help educate product teams on the role of evals and how to do them well and how to do them cross functionally and also it resonates with

the access to customers and meetings. So

sometimes you cannot sit on the table with like high netw worth people who make decisions and sometimes if they record this transcript you can get it or like video recording it'll be great but sometimes it's just like a transcript

and it's implied that you can understand the context. So that's one thing where I

the context. So that's one thing where I think that although you can use AI to make the transcript or analyze it, it it's not great. And even like sitting

and doing it in old ways, just read through and debrief. That's way better than just like throw it into AI and like what the team was working on or what people were discussing. So that's one

thing. Another is using synthetic data.

thing. Another is using synthetic data.

I've heard stories that sometimes teams can create kind of proto persona using AI and even run some tests with this

synthetic persona or uh create synthetic data to not eliminate but like filling the gaps if they don't have enough

understanding of the context or uh like business understanding so you can go and run your research and uh like filling the gaps uh that you miss. So is it the

like can AI help this to fill out those those gaps or what are not todos uh for uh teams in using AI for discovery?

So you mentioned sort of transcripts synthetic data and uh kind of testing with synthetic personas actually a real not a startup maybe even a startup uh I don't remember the

context but I've heard the story about that.

Yeah. Okay. Okay. So, let's tackle the three that I want to tackle is transcripts, synthetic personas, and synthetic data. Because I actually think

synthetic data. Because I actually think those last two are two different things.

Mhm.

So, with transcripts, here's what I'll say. I've said this um I wrote a blog

say. I've said this um I wrote a blog post not too long after ChatGpt came out that said something like, "Don't use generative AI to replace discovery with real humans." I still believe that. Now,

real humans." I still believe that. Now,

the key is to replace discovery with real humans. That doesn't mean we can't

real humans. That doesn't mean we can't use AI to add to what we're doing with discovery with real humans. So my point of view is this. This will probably never change. If you are building a

never change. If you are building a product used by humans, you probably need to be talking to those humans and to be testing your solutions with those humans. Full stop. I don't care how good

humans. Full stop. I don't care how good AI gets. That is probably never going to

AI gets. That is probably never going to change. That doesn't mean we can't use

change. That doesn't mean we can't use AI. Um, in addition to that, like I'm

AI. Um, in addition to that, like I'm going to give a real good like a real common example that I think most people are going to be familiar with. A lot of product teams have access to Gong sales

videos and sales transcripts and sales trends. That's great. Am I going to tell

trends. That's great. Am I going to tell you you shouldn't use Gong data? No. Of

course you should use Gong data. For

people that aren't familiar with Gong, it just records all of your sales conversations. It lets everybody on the

conversations. It lets everybody on the team access them. Does a little bit of AI synthesis on top. It's a great product. Does that replace my own

product. Does that replace my own interviews with customers? Absolutely

not. Right. So, what I like about Gong is I can now do my weekly customer interviews and build my own mental model of how my customers go about their lives. And on top of that, I can get

lives. And on top of that, I can get this gold mine of data from my sales team. I can see where they align. I can

team. I can see where they align. I can

see where they misalign. I can see where my sales team might need some feedback on overcoming objectives. So, I can help with sales enablement. I can see where I might be missing something in my interviews because customers prospects

keep bringing up the same stuff in inter sales conversations and I might need to broaden my scope in my interviews right so it's additive it's not it's not replacing it's not a substitute um I

think for most of these things I feel the same way when synthetic users first came out so synthetic users is a startup where they build these like personas based on your target customer market and

they pitch it as like on their first homepage. It was like recruiting is

homepage. It was like recruiting is hard, talking to customers hard, getting permission is hard, just talk to our synthetic users instead. That's built as a replacement. I do not like it as a

a replacement. I do not like it as a replacement. Now, does that mean there's

replacement. Now, does that mean there's no use of synthetic users ever for any purpose? I'm not sure. In fact, John

purpose? I'm not sure. In fact, John Whan um is a researcher who's like dove deep on the use of AI and research and he's actually a big fan of synthetic users. Does he use it to replace talking

users. Does he use it to replace talking to humans? No, he uses as an additive,

to humans? No, he uses as an additive, right? So like if you look at the amount

right? So like if you look at the amount of work you do in your product day, you don't talk to users about everything.

That's impossible. So maybe for the things that like don't bubble up to be important enough to your interviews, you get some feedback from synthetic users.

Do you treat it as gold absolute truth?

No. You treat it as input from an AI, but it's probably better than nothing, right? Um but the key is don't let it

right? Um but the key is don't let it replace your your discovery with real real humans. And then the third piece,

real humans. And then the third piece, synthetic data. So this synthetic data

synthetic data. So this synthetic data thing comes from LLM apps. So when we're building LLM apps, we need to be able to test that LLM apps AI features are

non-deterministic. So most features we

non-deterministic. So most features we write code um there's n dimensions of things it could do. We can write unit tests for all those n dimensions and we

know the code does exactly what we wanted it to do. When we're building an LLM app, it's nondeterministic. There's

an infinite number of things that it could do. it doesn't always do the same

could do. it doesn't always do the same thing every time and so we need a way to test it and so the way that we test it is with eval are a way of how we're evaluating our LLM product. The problem

is when we're new, when we have a new feature, we don't have a wide variety, a production data set of how customers interact with it. And so we don't know

how to evaluate how good our app is. And

that's where people are using synthetic data to generate a range of post potential customer inputs to then test the app before they put it into production. I think that is a perfectly

production. I think that is a perfectly valid use of synthetic data, but it raises questions about how do we generate that synthetic data and how do we generate it in a way to make sure it matches what we might see in production

or that it matches our best guess at what we might see in production. And

that's where I think a lot of our discovery can help. If we have a really rich understanding of the opportunity space, we can use our opportunity space to help um structure the way that we generate synthetic data. Now, if we've

never talked to a customer and we're just guessing what customers might ask and we're using guesses to generate synthetic data, it's probably garbage, right?

Mhm. Yeah. And we also kind of skip the the topic about opportunity solution tree. And this is something that I

tree. And this is something that I really would like to tap a little bit at least. Uh so opportunity solution tree

least. Uh so opportunity solution tree is a way to map business outcomes or like not business like but just product outcomes to opportunities to assumptions

and uh and you also mentioned that sometimes uh it be could be a situations in which I don't know like there are memes when product managers come up to

their team and saying like hey I was just on a stakeholder meeting or leadership meeting and now we need to pivot or kind of create something that we haven't expected. So sometimes as you

mentioned like uh it's okay to follow the whim of stakeholders but if this is a constant uh like change or pivots and they don't uh not like kind of follow

the outcomes or opportunities like the team is working hard uh they can invest time in discovering opportunities but there are some external factors like potential customer or even this like new

technology wave with AI and I'm sure that teams uh found theel elves in a situation when they were kind of not demanded but stakeholders told that we

need AI and it probably wasn't on the road map and it wasn't on the kind of on the opportunity space at that moment. So

doesn't mean that they need to create new opportunity tree or fit it into the existing tree. So how does it

existing tree. So how does it practically work?

Yeah, let's talk about this in the context of AI because I think you're right. Everybody's going through this.

right. Everybody's going through this.

So first of all, let's talk about an opportunity solution tree. It's just a visual to help your team stay aligned as you try to reach an outcome. So, if

you're familiar with decision trees, there's your outcome at the top of the tree. It branches where you're mapping

tree. It branches where you're mapping out the opportunity space. So,

opportunities are unmet customer needs, pain points, or desires. So, as you interview customers, you're collecting needs, pain points. You're mapping them out. You're giving structure to the

out. You're giving structure to the opportunity space. And then you're

opportunity space. And then you're choosing a target opportunity to work on. And as you choose that target

on. And as you choose that target opportunity, you're exploring multiple solutions. and you're evaluating them

solutions. and you're evaluating them with assumption tests and that's just as you work your way down the decision tree there's those different components.

Okay, so Chad GBGD comes out in what November 2022. Uh took about a year.

November 2022. Uh took about a year.

Companies started realizing this is a real thing and suddenly everybody's saying what AI features are we going to build. Most companies because again what

build. Most companies because again what we talked about earlier, humans shiny object syndrome. We're all talking about

object syndrome. We're all talking about ideas. We're all living in the solution

ideas. We're all living in the solution space. Executives probably came to

space. Executives probably came to product teams and said we're going to build A, B, and C. Now imagine that you were on a team being told to build A.

Doesn't matter what A is. you're just

being told to build a and you had been doing continuous discovery. Does your

stakeholder care about A or do they care about having an AI feature? They

probably care more about having an AI feature. So if you're being asked to

feature. So if you're being asked to deliver A, but you have a already have a good understanding of the opportunity space, you can ask a better question.

You can look at your opportunity space and say which of these opportunities is now better solved by AI. And now you can start to negotiate with your stakeholders. You can say, "Look, I can

stakeholders. You can say, "Look, I can build a I'm not really sure how to fit that in my opportunity space. It's not

clear to me what need it solves. Maybe

you can help me fill in the gaps."

That's a part of the conversation.

Another part of the conversation is here's two or three other needs where I could see o AI having a big impact. What

if I explore those instead or in addition to now we're having a much better conversation, right? But if I have no understanding of the customer, no understanding of customer needs, and

I just have a different idea for I want to build feature B instead of feature A, now we're just in an ideological war.

It's an opinion battle about is A better than B. But if I have a rich

than B. But if I have a rich understanding of my customer in the opportunity space, it's not an opinion battle anymore. I'm bringing new

battle anymore. I'm bringing new information to the conversation. I'm

showing that I know my opportunity space, I know my customer, I understand my outcome, I understand what matters to the business, and I'm in a much better position to negotiate and influence.

Mhm. Yeah. If we understand what pain points our customers have, then it's like we understand how AI can solve that. I really love the story from

that. I really love the story from notion. They wanted to launch AI

notion. They wanted to launch AI assistant first, but after discoveries and iterations, they just like created a button that magically sorted out the

inbox. So that was for their male

inbox. So that was for their male client. Um, so yeah, that was kind of an

client. Um, so yeah, that was kind of an interesting thing like eventually you come to that decision but if you understand the clients and their needs, pain points and the opportunity space in advance then you can come up with a

better solutions.

Uh I'm being time mindful of the time.

So maybe we can move to closing and we usually have a section with rapid questions. Uh it's not necessarily yes

questions. Uh it's not necessarily yes or no, but just like uh like what's on the top of your mind. So uh what's the most surprising thing you've learned from coaching teams? Something you

haven't expected.

That's a really good question.

Uh, I think one thing that surprised me is at large companies how much how little work is happening. Like teams

spend all day every day in meetings.

I'll give an I'll give a really specific example. I know we're supposed to be

example. I know we're supposed to be quick. I had a team that like they their

quick. I had a team that like they their um customers were clinicians, doctors and nurses. They were having a really

and nurses. They were having a really hard time finding doctors and nurses to interview. I asked them I said, "Don't

interview. I asked them I said, "Don't doesn't your company have a clinician team? Like can you just go interview a

team? Like can you just go interview a colleague?" It took six weeks to get on

colleague?" It took six weeks to get on a clinician's calendar even though they were co-workers. And it's because

were co-workers. And it's because they're literally booked in meetings.

Some of them are double and triple booked in meetings all day long. And I

know that meeting culture is terrible and that this is a problem for everybody. I had no idea it was that

everybody. I had no idea it was that bad.

What's the red flag that tells you the team isn't ready for continuous discovery.

They're still stuck in like territorial battles, right? I do this, you do that.

battles, right? I do this, you do that.

We don't talk to each other. We just

hand things over the wall.

Yeah. Cross functional team is the best.

uh one tool that every PM should use product manager I mean KBT or cloud easy and uh best question to ask a customer when you're completely stuck

story based question always tell me about a time when yeah and as we wrap up uh I would like to hear your thoughts about the trends

uh like about software development life cycle in general because everything is collapsing and how what are your predictions uh will there be a time when AI will do

discovery as well? I mean like not we need to talk about customers and uh with people but uh in terms of like uh pinpointing potential opportunities or

like flagging something into your face when there is uh I don't know like some trend in data uh automatically. So what

are your predictions about impact of in general?

I agree with what Marty wrote in his essay about this. I think the easier it becomes to build things, the more important discovery is going to become.

And I think there's a really great analogy for this. For the people that have seen the Simpsons episode where Homer Simpson designs his own car, it's like a Frankenstein vehicle with like

amazing features and no coherence. Like,

it's just silly. And I think that's what we're going to run the risk of is when anybody in the organization can decide to add something to the product and it's effortless, we're going to end up with really unusable products. And I think

we're already seeing that. We're already

seeing a ton of companies tack on AI features without thinking about customer problems, without thinking about quality of those AI features. In fact, I posted over the weekend asking on LinkedIn,

what's your favorite AI feature that was added to a nonAI product? And very few people have replied because I think very few people have a good example of this.

We have, I bet if I asked the opposite, what's your least favorite? I would get hundreds of replies because we're in this like day one garbage. like

everybody's seeing it as an arms race and people aren't thinking enough about quality. Um so I actually I just think

quality. Um so I actually I just think we're I don't think the basics are changing. In fact, I think they might

changing. In fact, I think they might becoming more important.

Yeah. And also like speed of uh delivery is so fast like you can have like multiple changes and users are already frustrated like I remember that this button used to be here and it just will

amplify. All right. I would love to end

amplify. All right. I would love to end this conversation with your quote which I love very much about uh most of work of uh in discovery is not following the

process it's managing the cycles. So uh

it just like have a continuous improvement mindset start with some something small iterate improve and just like uh it's not there is no silver

bullet for discovery.

Yeah I think the key is to not think about the book as a recipe but just think about it as guard rails. It's

habits. I mean, I put habits in the title for a reason, right? So, it's just how do we build good habits so that we're always learning. It's a it's all about fast feedback loops with customers.

Yeah, Teresa, it was very incredibly valuable. Uh, thank you so much for

valuable. Uh, thank you so much for sharing so much so many insights and it was a great pleasure hosting you today.

Yeah, thanks for having me. It was fun.

And to everyone who's listening, if you found this episode helpful, please subscribe and leave a comment what resonated with you the most. I would

love to hear about your discovery processes and how you develop your discovery skills. Woohoo! Yay!

discovery skills. Woohoo! Yay!