Google Product Manager Metrics Interview: GPT-5 Launch

Product success metric interviews are now a part of more than 50% of product management interview processes. So if

you want to land a product management job this year, you have to master this product interview question. In today's

video, we're going to walk through an A+ example from a former Google product manager so that you can see the structure, the steps, and ultimately how to deliver a great product success

metrics interview. So without any

metrics interview. So without any further ado, let's get straight into it.

Mark, how would you measure the success of GPT5?

>> Okay, GPT5. Um, super interesting question. Uh, actually a couple quick

question. Uh, actually a couple quick clarifying questions. Um, am I

clarifying questions. Um, am I responsible for all of GPT5? I mean,

that's a pretty big product.

>> Let's assume that, you know, the OpenAI product team, it's actually surprisingly lean. Like 90% of the company is

lean. Like 90% of the company is researchers and engineers. So they do have, you know, CPO Kevin while you're reporting up to him. He has a pretty

flat or of like 20 PMs and yeah, you're responsible for the GPT5 roll out because they're releasing a model almost every 3 months. They have a PM on each one.

>> Okay. All right. Cool. So, uh let's go there. We'll go pretty pretty

there. We'll go pretty pretty broad-based. So, the way I'm going to

broad-based. So, the way I'm going to approach this problem, um classic analytics problem. The way I'm going to

analytics problem. The way I'm going to talk about this problem is one, um talk about OpenAI, what their mission is, and how GPT5 might roll up into this mission. Uh the next thing I'm going to

mission. Uh the next thing I'm going to do is I'm going to create some product goals. You know, what are we going to

goals. You know, what are we going to try and do with CPD5 and how is it related to the mission? Give us some context. Um then what I'm going to do is

context. Um then what I'm going to do is um take a look at the product. I'm going

to really break it down in terms of you know who the stakeholders are, what are the actions they take, what are the key metrics that they take. Uh we'll spend a little bit of time here and then finally what I'll what I'll end up with is I'll

end up with a north star which is what's really really important to target to and then we'll also talk about some things like some countermetrics and downstream metrics as well and we can also do some trade-off questions at the end if we want to. Okay, sound good so far?

want to. Okay, sound good so far?

>> Love it.

>> Okay, awesome. Okay, so let's start with uh let's start with actually the mission of of OpenAI and what's important. So um

the mission of open AI is to ensure that artificial intelligence benefits all of humanity. This is a really really really

humanity. This is a really really really big mission. And so when we think about

big mission. And so when we think about chat GPT right this is really the um this is really the face of the company.

This is how people interact with the company. Uh this is their exposure to

company. Uh this is their exposure to the company. And they've had a number of

the company. And they've had a number of models in the past. I think they started with one, two, I think when they got to three is when it got interesting and then three and 3.5 and four and 4.5 and five is their latest hallmark. So

obviously this is a big uh phase of the company and it's and it's super super important. Um also I want to say um the

important. Um also I want to say um the use cases that is going on um is um well we'll get into use cases but we'll just say it's used by consumers, professional

users, companies, governments probably now at this point, right? So a lot of people are using chat TPD5. I think it's actually the most downloaded app on the app store in general.

>> Yep. So, um let's as we kind of drill into the importance we can see why this is important to open AAI, let's create a little bit of a mission for chatbt5 and I think it's going to be an extension of

the mission of products before it. But I

think it's to make um AI accessible um and useful and uh aligned to everyday problem solving. Of course, it means

problem solving. Of course, it means very different things to different people, but it helps with creativity and productivity. So, again, we want to make

productivity. So, again, we want to make AI accessible and useful. Sounds a

little like Google in a way, but but anyways, let's go on. Okay, so now we've got this we've got this broad goal. Um,

I want to break that down a little bit.

Uh, because when I think about OpenAI, um, I think there's going to be four kind of sub goals underneath this. Um so

uh on the sub goals one is that as a brand we're trying to acquire more and more users to try GPT to use GPT. So I

think goal number one is or sub goal number one is acquisition. Um sub goal number two is for all these new users new users using chat tpt and all of our existing users who are on three four and

five we want them to upgrade or three and four to upgrade to five. We want

them to do uh we want them to use the product. We want people to use this on a

product. We want people to use this on a regular habitual basis and we want to demonstrate a really strong engagement and then there's a couple other layers here that I think are important to J GPT specifically or open AI specifically. So

one is around quality, right? So this is about AI, right? This is about making AI useful and accessible. And so it's got to work. The first versions of it worked

to work. The first versions of it worked pretty well, but I think they inspired people, but you know, people had problems with hallucinations and things like that. Uh and so we're going to

like that. Uh and so we're going to continue to invest in uh in quality and our road map to to get to uh to get to AGI. And the last piece we want to take

AGI. And the last piece we want to take a look at is probably some layer of monetization. What you should notice

monetization. What you should notice here is how quickly Mark came to a framework that he could use to guide the rest of the conversation. This is really important in case interviews.

>> It's a very different model than a Google. Um Google's model is spend more

Google. Um Google's model is spend more time on site, click on blue links, make money. Uh chat TPT is more of a

money. Uh chat TPT is more of a subscription model. So, it's more about

subscription model. So, it's more about jobs to be done and it's really about helping you do your task uh and getting people to upgrade to their models and pay them subscription revenue. And I

think I believe actually subscription revenue is the number one driver of revenue at OpenAI right now. So I think this is a a really important layer.

>> One question for you. How are you going to think about the consumer versus the enterprise business?

>> Um good question. Um in fact let me I'll take a step back a little bit. So when I think about users of Chat GBT actually I think there's a number of users and there's a number of surfaces. So the

users I I'd put in consumers. I think

it'd put in companies um actually I think it'd also put governments right and and even chat tpt openai is kind of a platform itself. So I think there's both uh consumer users and corporate

users. Um and I think corporate users

users. Um and I think corporate users are are basically like larger consumer users which are and I see lots of companies doing doing this all the time which is they are subscribing to chat

GPT as a platform so that their employees can be more productive right.

Um this is also an interesting use case for how GPT is used in surfaces right so obviously there's a web- based app which everyone is familiar with and there's a

mobile app which everyone's familiar with there's also have a desk desktop app which less people are familiar with uh and there's also APIs right so they have an API layer and this is where um

companies are using it as well too to use CH GPT uh as a service to make intelligent offerings and we see things in everything from customer support to chat bots and things like that. So it's

powering it's powering all sorts of use cases uh just uh around the circumference. So if I think about chat

circumference. So if I think about chat GBT, I think about all of that usage. In

fact, this leads to what I'll call a primary goal, which is around engagement, right? So we talked about

engagement, right? So we talked about four goals here. Acquisition,

engagement, um, monetization, quality. I

would actually put engagement kind of the top most important thing. So let's

actually break this down a little bit further and we'll talk about actions and metrics at this point.

So, in the actions game, what I want to take a look at is what are the primary ways that people interact with this and and then uh and then we'll break it down into a number of metrics to to really

kind of geek out this product from a metrics perspective. So, from an actions

metrics perspective. So, from an actions perspective, uh and this applies to, I think, consumers and businesses alike, right? They're going to visit the

right? They're going to visit the homepage. Uh they're going to visit the

homepage. Uh they're going to visit the app. They're going to download it. Step

app. They're going to download it. Step

one, start two, they're going to they're going to chat with it, right? They're

going to start engaging with it. Um,

step three, they're going to, you know, have this chat experiment experience.

They're going to provide feedback, right? And from that, they might, you

right? And from that, they might, you know, upgrade to the paid plan. They

might say, "Oh, this is really great. I

might start using the API." Um, and then, uh, based on those outputs, they might also maybe share their output with friends or colleagues or share through email or things like that. Uh and then

the last two actions I'm going to look at are really going back to our goal but I want to look on more broadly are on uh key metrics around quality and monetization and we'll talk about those

in a little bit more detail. Okay. So

let's talk about monetization metrics.

So um again since uh monetization is a really really really big part of this product. So I think the things we're

product. So I think the things we're going to look at are standard metrics like Arpoo and I think um this product has a $20 subscription rate. I think it also has advanced rates for users as well too. And then corporate users are

well too. And then corporate users are in a completely different bucket because they start using the API and the API is kind of a per token type model. So we

want to have a number of models where we're doing you know per user and also per token to basically see uh which is really kind of almost a per query model to see how uh we are monetizing. And

then we want to take a look at conversion rates which are you know converting free to paid and then paid to corporate and then paid to API and corporate to API as well too. So there's

a number of kind of mini conversion funnels getting people to use more and more. So uh monetization metrics um

more. So uh monetization metrics um really important part of the equation.

Um also as part of that we might also have people who are churning out people are like paying $20 a month and going I don't want to do it or maybe they signed up for the API and then they're not really utilizing it. Uh so we want to

take a look at those users part of our overall monetization ecosystem.

Okay. So we talked about a lot of metrics. Big question now is what's our

metrics. Big question now is what's our north star? What's the most important

north star? What's the most important thing? So when I think about the

thing? So when I think about the stakeholders here, you know, I think about consumers. I think about

about consumers. I think about companies. I think governments are also

companies. I think governments are also involved and interested in this. I think

uh open AI certainly interested in this.

What's everyone interested in? So I

think consumers are interested in I want amazing AI responses which are you know fast and inexpensive and useful. If I

think about companies I think they want the same thing. I want AI responses which are fast and and useful and uh you know and costefficient right. If I look at uh if I look at governments and

stakeholders like that I want to make sure that um um responses are safe and that uh safe things are happening right now. I think safety is a a layer that

now. I think safety is a a layer that that probably uh borders the whole thing. And if I'm open AI, I also want

thing. And if I'm open AI, I also want to, you know, pave this world towards a safe, responsible arc towards AGI, right? That people are using it in a in

right? That people are using it in a in a way that's uh uh that's useful and it benefits all of humanity and it benefits all of humanity is also interesting to governments, which I think they they have a vested interest in that. Not sure

about consumers or or companies, but I think uh certainly open AI and uh governments do. So based on all of those

governments do. So based on all of those things, I think the one northstar that would kind of bring all these together is we talked about this concept of jobs completed. I think our best northstar

completed. I think our best northstar here would be the number of jobs completed. And uh let me define that

completed. And uh let me define that just arbitrarily as um when we have a job, someone has an intent to use AI and say for today I have a specific intent.

If I'm able to do that job, it's done.

It might take me one query or two queries or three queries, right? but I

really want to get that job done. Um, so

it's really about accomplishing jobs.

And if I can accomplish that job more efficiently, better for the consumer, probably less expensive for OpenAI because I don't want to spend them all day quering. It's

just going to be very expensive, right?

So, I want to be able to help people get jobs uh done efficiently. And I know that if I can help people get jobs done efficiently, they're going to come back and do it again and again and again. And

um and I I think we want to use uh guard rails uh like things like safety and quality as a ways to make sure that we're doing the right jobs for the right people. So our northstar number of jobs

people. So our northstar number of jobs completed. Uh and the last thing I want

completed. Uh and the last thing I want to talk about uh are a couple of countermetrics and uh maybe some downstream metrics.

>> Let's take a quick break to notice how Mark answered the question, but he was unfazed and quickly kept moving on. You

need to keep a forward pace in your for this north star. I think it's interesting in that >> it's definitely getting towards what we want out of users. I'm just worried how

are we going to operationalize this? How

are we going to measure this reliably and effectively in a way that you know when the product team goes and talks to a researcher about some specific model,

they're not going to get overly into the minutia about how this was defined.

Um that's a good question. So I think um one there's a lot of work from data science here. Um I I think it's

science here. Um I I think it's important for Northstar to be fairly understandable in terms of number of jobs. Now how we actually define jobs we

jobs. Now how we actually define jobs we can probably break that down because you can also have a job which has specific intents. Let's say like I want to build

intents. Let's say like I want to build a website, right? That's a big job, right? Which is going to have lots of

right? Which is going to have lots of little lots of little subtasks. So I

think it's about defining right what's a level of subtask per session. And I

think I'd use work with data science to kind of figure out what's the right right granularity of a job. I guess you could say the right granularity of a of

a task uh to be done. Um but I want to break it down so that it's it's like the um it's the right task size. It's not so big like I want to build a website which

has got thousands of things, but it's also not so small so that every query counts against it, right? Because really

what I want to develop is an efficient uh artificial intelligence system that when I ask it a question, it can give me a reasonably good answer that I think

like wow this is a great answer and it may invite me to be like I've got everything I need or it may invite me to explore more based upon this information. So I think in terms of you

information. So I think in terms of you know what's the right fidelity I think this is something that we really need to work on with um with uh the data scientist team to figure out what that

right fidelity is but I think the highle job of getting um the the number of jobs done I I think it's the right I think it's the right high level metric to focus on I think the company can get around that

>> okay so uh marching forward a little bit here on countermetrics um so you know if our if our you know if our key metric is jobs to be done. I'd say actually a key

countermetric is job's not done. Right?

Imagine people coming and going, "Ah, I couldn't get it to do the thing I wanted to do it. I give up." Right? This is not on the path to AGI. Clear countermetric

for us. Um, also, I think some other good countermetrics overall for us are things like latency or API error rates.

Right? We've all had scenarios where we're using this and typing questions along and it's just like, I'm full. I

can't answer. I'm busy. Right? These

things are all countermetrics on our on our arc towards ATI. And then also I think um thumbs down rates, hallucinization rate hallucinization yeah hallucination rates are um I I

think those are also great metrics to use counter metrics to understand the overall quality of the product. So we've

got good guard rails to make sure we're getting the jobs done. We're doing it fast. We're doing it high quality. Uh

fast. We're doing it high quality. Uh

we're doing it in all these ways. And

then the last thing I'll take a look at here is um downstream metrics which is you know what happens right if people are using this product downstream what's the what's the consequence of this so I

think uh big consequences of using this product are obviously the the funnels that we talked a little bit about right so you know going from a user to a subscriber to a corporate user to using

API for downstream companies right those are all great downstream metrics to look at um also I think um extension ions into the ecosystem, right? So

third-party apps and API calls, things like that and then really kind of uh enterprise adoption across the industries in general, right? Using

this, I think those are all, you know, going from AI today to much broader uh applications with third party developers and companies. I I think there's a lot

and companies. I I think there's a lot of ways we can we can measure this on a on a downstream basis. Okay. So uh so what have we done today? Um just that quick summary of where we've been. Um we

said hey how would we measure uh chat GPT5 and what we did was we came up with a mission statement for what we want chat GPT to do. We created a set of goals across acquisition engagement uh

quality and monetization. We walked

through, you know, kind of the user journey from querying to responding to answering to sharing and all these kinds of things and a bunch of metrics on that. And we ended up with a um a really

that. And we ended up with a um a really solid northstar goal on uh jobs done uh which I think would be a great way to measure chat TPD5 as well as some counter uh downstream and

countermetrics. Okay. Um let's wrap it

countermetrics. Okay. Um let's wrap it up there. Um what kind of trade-off

up there. Um what kind of trade-off questions do we have?

>> Notice again how Mark just answered the question and moved on. he didn't over fixate a little bit more on the north star. Let's say

star. Let's say >> um one of the things that's happened in OpenAI actually is that we've really gotten a preference for more I would say harder metrics and we specifically want

to look at kind of like CFO level metrics. How would you define kind of a

metrics. How would you define kind of a CFO level northstar metric for GPT5's launch?

So, a CFO, if so, if I'm a CFO metric, um, putting on my CFO hat, which I don't play one on TV. Um, um, putting on my

CFO hat is I want to build a financially sustainable company, right? So, on a CFO hat, I'm looking at um, what are my costs of of all this infrastructure,

right? I've got servers and and usage

right? I've got servers and and usage and energy utilization and things like that. So, I've got, let's say, I've got

that. So, I've got, let's say, I've got a billion dollars in cost, right? And

then I've got the revenue side of the equation, which is what are all the revenues coming in? That's from API calls. It's from subscriptions, things

calls. It's from subscriptions, things like that. And obviously, I want to make

like that. And obviously, I want to make sure my revenue is greater than my cost.

Otherwise, as a CFO, I am making a um a uh a long-term investment in a short-term company. So, I want to make

short-term company. So, I want to make sure from a CFO perspective that my, you know, that my revenues, my gross margins are financially sustainable. I would

probably break that down on a per query basis, which is across the services. You

take that billion dollars. If I've got a billion queries, that means it's a dollar per query. And it's definitely not that, right? Um, and then I look at um from a cost perspective. And I look

at my revenue perspective. I'm going to look at my my users, what their average utilization rate is, and how much they're costing is consuming, and how much we're doing. So I want to basically

make sure that our our marginal you know query costs versus all our sunk cost uh align so that I have a sustainable business moving forward in the future right so that I want to know that I can

continue scaling my business as I add that next 10 million users and next 100 million users um that it's actually going to increase the monetization of my company and not uh and not and not and

not make it go the other way.

>> So this is really interesting. Mark went

back to stakeholders. Originally, that

was earlier on in his framework, but responding to the flow of the interview, he decided to pull that back in before his chose his northstar metric. It's a

really good idea to reorganize your framework as needed. Love your response.

I love how you also kept a forward momentum going, which I think a lot of people struggle with doing, you know, after somebody asks a question, kind of like getting zoned in on that question, even letting that question take five or

six minutes, but just continuing to move forward. And um having that balance of

forward. And um having that balance of enough pausing, but then also sticking to your framework. And it was interesting how stakeholders I think it was originally meant to come before actions and metrics, but you kind of

adapted to the questions I answered and you brought it in before Northstar metrics. So, I thought that was also

metrics. So, I thought that was also really clever.

All right. And here's my analysis of your interview that you did, Mark. I

think that what you could have done even better is a couple of different things.

The first is have a little bit more on the Aie Eval style metrics. I think you really stuck to product metrics, but didn't necessarily get into AIE valves.

The second thing you could have done better is have more responsiveness to some of my interview questions. For

instance, when I talked about the API or when I talked about um having a less fuzzy metric, I ultimately needed to come back to those things once more afterwards. It would be better if he

afterwards. It would be better if he could be even more responsive to those things and pre- guess, okay, so here's how I'm going to think about it financially. Here's a less fuzzy version

financially. Here's a less fuzzy version of the Northstar if we really have to go for it. If you answer questions in that

for it. If you answer questions in that way where you have a backup, then people see how flexible you are in your thinking. And then a final thing that

thinking. And then a final thing that could have done even better is potentially a screen share of some sort so that we can really see as you follow along in your thinking. Mark is one of the most humble people about this. He's

helped literally hundreds or thousands of people get interviews. If you haven't seen him on YouTube already, welcome to his YouTube back catalog. There's plenty

more amazing things he has there as well if you want free resources.