Tech Talks EP25: Discovery

[Music] Welcome. I'm Dave Bazooki, CEO and

Welcome. I'm Dave Bazooki, CEO and founder of Roblox. And today you're listening to Tech Talks. Today we're

talking about discovery. How do you find cool stuff on Roblox? How do you search for it? And we are joined by two

for it? And we are joined by two Discovery superstars. We have Michelle

Discovery superstars. We have Michelle Gong on engineering, Chenzang on product. We're going to be talking about

product. We're going to be talking about all things discovery. And uh really we're hoping when you go away you're going to learn a lot about what you watch on your living room, how you go shopping, and what it means when we talk

about that. So, welcome today. Let's get

about that. So, welcome today. Let's get

going. Let's just talk a little about your backgrounds. Um I scanned both of

your backgrounds. Um I scanned both of your LinkedIn pages and you both have something in common. Do you know what it

is? You both have PhDs. Is that right?

is? You both have PhDs. Is that right?

That's right. Okay. So, let's just dive in. Right. Chen, you're in product.

in. Right. Chen, you're in product.

Michelle, you're in engineering. Little

background from you first, Chen, and then we'll go to you, Michelle. Sure. I

would say like 20 years ago, I came to America actually to pursue a PhD in physics. So, I kind of set the foot into

physics. So, I kind of set the foot into the AGO world, I would say. And that

became a constant theme later on when I became a product manager. So I've been working on a related problem spaces for two decades and largely actually related

to discovery as well as other problem spaces uh related to discovery such as content production as well as um uh

merchant fraud. So I actually was very

merchant fraud. So I actually was very lucky to apply the early days of machine learning to Google shopping express uh search and recommendation province and

and later on got to uh actually have the chance to build the first version of search discovery fee and story ranking at Instagram. And when I talk about the

at Instagram. And when I talk about the merchant fraud problems, it was actually in a startup and we had a lot of issues with um all sorts of like merchant

quality policy issues and had to deploy the kind of the first human in the loop uh machine learning driven process to fight it and then eventually yeah at Netflix had a chance to work on both

content production and promotion. Okay.

So those are three companies we'll probably mention today. So Google,

Instagram and Netflix. And I think uh for everyone out there who's in STEM or learning, interesting that such a technical degree ends up being very appropriate for search and discovery,

which is a very technical product category. Okay. And how about you

category. Okay. And how about you Michelle? Like how did you get your

Michelle? Like how did you get your start? Uh so my PhD uh was in on

start? Uh so my PhD uh was in on statistics and machine learning. Uh I

have worked in research labs until Roblox. Yeah. Uh so before Roblox, I led

Roblox. Yeah. Uh so before Roblox, I led applied research labs. uh in Google Deepm and Meta AI uh helping Facebook,

Instagram, Google to improve their production recommendation systems and content understanding models. I have

joined Roblox three and a half years ago time twice. And it's fun working with

time twice. And it's fun working with you. It's a lot of fun working here. We

you. It's a lot of fun working here. We

roll up our sleeves on some pretty big technical problems and and they're both very researchy, but we literally ship your work every week. So it's both

researchy and getting So I thought Discovery was Captain Cook, right?

Captain Cook was sailing around the Pacific Ocean with a fleet of ships looking for new stuff. He was looking

for interesting new things. Wasn't quite

sure what was out there. Found Hawaii.

Found a few other interesting things.

Um, so how does that relate to discovery in a consumer product? Uh, who wants to talk about what does discovery mean when we look at Netflix or we look at Instagram or we look somewhere else?

Yeah, I love that analogy. I think a discovery ultimately means how to help users find new types of content, the connections or services they're looking for. And then generally speaking, there

for. And then generally speaking, there are many different types of discovery.

Yeah. The other types of discovery is very search uh centric. uh that usually for platforms like app stores or Amazon because people have a specific app or

product in mind so they go look for it.

Uh they're also the type of discovery is very recommendation centric. You

mentioned just Netflix right as well as I would say Xbox. So is that how that people think about the industry?

Discovery is the overall term and then there's either search or recommendations. Yeah. Is that the right

recommendations. Yeah. Is that the right way to think about it? Yeah. There are

also other categories like for example I would say ads also adds to the font. So

if I go to a shopping product Walmart, Amazon, you name it, I have both a search bar y and sometimes when I go to the homepage on my Netflix, I just see a

bunch of things. So So those seem like the two different classes. If we think through all of the products that have a search bar, that search discovery, what types of things have recommendations? I

know Netflix does. Are there any other products that pop to your minds? I would

say YouTube. Okay. For sure. Instagram,

Spotify, uh Tik Tok, a lot of social media platforms. Xbox, Xbox too, Steam as well. Okay. And then um so those are

as well. Okay. And then um so those are discovery by images or like I I typically get tiles and I think we're all really used to that. We don't we

don't know how used to it. A few years ago there was a big innovation in discovery and we started doing discovery not just by images but by by video. Can

you can you comment on what class of products those are and are they doing discovery as well? Yeah. Who wants to go? In fact, I would say Tik Tok, uh,

go? In fact, I would say Tik Tok, uh, Instagram res, a lot of these short video products actually changed how especially new generation uh, intact

with the internet. It is a very challenging problem because how could you capture users interest within the first few minutes when they get onto the

platform and then how do you understand which video is most relevant? Uh yeah,

it's uh definitely a different paradigm.

That is still discovery. That's right.

Um it's interesting to think with those products is is it fair to say the product is discovery in that it's I the

whole time I'm using the product in a way I'm doing discovery. Yeah. Well, in

some way like the traditional discovery, right? you saw some kind of a cover of

right? you saw some kind of a cover of the product and then you go somewhere and then you go there and then check if the product is what you want or if that content is what you want but short video change that right what you see is what

you get so there's no such evaluation process in fact you are in consuming and then deciding oh I want it or don't want it and then go to the next step and so two little hints on that type of

discovery first the the idea that instead of being separate an image and then consumption we have seen a lot of the video things I watch on my in my living room move to auto showing a

little clip for example. There's a

little bit of it. I think that uh without giving away that hints a lot of the opportunities we have at Roblox to move from just image to what could be

next. So that's a fun topic. And then

next. So that's a fun topic. And then

one thing I think I'm not sure and you both might know the answer a lot of people um and before we dive into the details of discovery how do those short

form video products so I know there's Tik Tok shorts real spotlight and there's a bunch of others how do they know to show me u videos of cats

fighting with ducks like how do they figure that out any idea like yeah some parts of it actually coming from the user behavior and they basically give

you a some a set of a sampled videos and see how you work. What kind? But I

didn't I didn't know I'm doing any behavior. What uh I I feel like I'm just

behavior. What uh I I feel like I'm just scrolling through things. Is there some behavior I'm doing that's giving them a hint as to what I like like how much you actually watch that video in front of you and then whether you immediately

skip or you actually watch the whole thing and did you actually give it likes or did you actually follow the creator?

Is it possible that only the the speed at which I scroll, how long I stay on any video and whether I like has been enough to infer

what I like on those products. In fact,

uh I want to be a little bit nerdy here.

Yeah. Yeah. Uh that's implicit user feedback. We also have explicit user

feedback. We also have explicit user feedback. Uh when you first come on, for

feedback. Uh when you first come on, for example, our platform, we know very little about you, right? Yeah.

Interactive recommendation is the key.

We want to kneel the recommendation within the first few minutes. And how do we get transfer learning from other similar user cohort? How how do we immediately get the signal from your

behavior and trend the model and get the uh real-time signal to tailor the recommendation towards you? So, would

that mean that if I tend to watch all of the cat videos most of the way through and all of the duck videos all of the

way through, there's other users like me who also like ducks fighting with cat videos and I'll start to see their video. That's right. And then sometimes

video. That's right. And then sometimes they will also inject some of the new uh signals to see if you like those new video and if you like it, they'll show you more and they're also part of the signal actually coming from the video

itself. So a lot of those um short video

itself. So a lot of those um short video platform actually invest in content understanding. They trying to understand

understanding. They trying to understand the creator intent uh the the metadata of the short video using that they will pair with what they know about you as a user. For example, if you tell them when

user. For example, if you tell them when you're on boarding you like these type of topics like cats then they want to show you those you know later on. So uh

basically the final uh put together recommendation system usually have both the user behavior information implicit and explicit signals as Michelle said as well as a content sending signal. So

like a creation related content to put together the recommendations for you.

Okay. So that's a good foundation in discovery. It's not Captain Cook. It's

discovery. It's not Captain Cook. It's

giving people the best content or content they're interested in. So we

jump over to Roblox now. and and Roblox is a little different than video. It's a

little different than shopping. These

are experiences that people can come to together. So, it'd be fun to talk about

together. So, it'd be fun to talk about how Discovery's evolved a bit on Roblox.

And I can start if you want because we didn't even know what Discovery was when we started Roblox. We just knew when UGC

started getting more popular, how do I find something interesting? So we ran for many many years single discovery

algorithm. Our algorithm was uh show all

algorithm. Our algorithm was uh show all of the experiences with the real time stack ranking of the number of people in the experience. It's that simple. And

the experience. It's that simple. And

and that got Roblox going for probably the first eight years or so. Um, it had a couple interesting traits. When

something was really crazy, it went straight to the top. It also had a lot of issues. It it would tend to burn in

of issues. It it would tend to burn in something at the top. Um, so it gave a huge advantage to experiences that had been around for a while. So, it'd be fun

to maybe just talk a little about how discoveries evolved at Roblox. Maybe

what you saw when both of you came on board and some of the things we've done since there. So who has to jump in on

since there. So who has to jump in on that? Uh when I joined I think we had

that? Uh when I joined I think we had our first personalization model. It was

a collaborative filtering model with only one input signal user experience engagement pair. Okay. And then we re

engagement pair. Okay. And then we re recommending top 2K experiences but it's already a step forward than non-personalized model. Right. And then

non-personalized model. Right. And then

since then you so so a personalized model means rather than recommending the same thing to every single thing person on Roblox no matter where you live how

long you've been on the platform what device you we we used to recommend the same thing to everyone. So this was a big step personalization. That's right.

uh and then after I joined in the next two three years we replaced their collaborative filtering model with a multi-step uh stage ranking model. So

that basically means we want to retrieve as many experiences from all the 90 million experiences on the platform as possible and we rank them and

personalize to that user's intent. And

then um we have deep learning, graph learning. Uh in the more recent years uh

learning. Uh in the more recent years uh we implemented self self-supervised learning uh efficient attention modeling, entire space multitask

ranking. So the the whole ranking stack

ranking. So the the whole ranking stack essentially and then uh I think now we are recommending 30k to 40k unique

experiences daily. Okay. Yeah, that's a

experiences daily. Okay. Yeah, that's a lot. That's that's in 3 minutes. Massive

lot. That's that's in 3 minutes. Massive

improvement on Roblox. So, let's let's tease a few of these apart. Um, so going all the way back to the start, we used

to rank real time just all the experiences. We started thinking about

experiences. We started thinking about personalizing this. And then one of the

personalizing this. And then one of the first steps um you mentioned engagement is when you start connecting people with experiences you have to know where

you're going. Sometimes we call this I'm

you're going. Sometimes we call this I'm always asking what are we optimizing for right basically or or what's our objective function. If I was to go to a

objective function. If I was to go to a cocktail party and someone was to say, "Hey, what's your discovery recommendation optimizing for, right?" I

said, "We figured out a way to optimize for five-year user happiness and retention." And so, we're able to

retention." And so, we're able to connect people with experiences that they love and makes them just want to hang out and be with their friends on Roblox. Would that be a good answer?

Roblox. Would that be a good answer?

Yeah, I think that is the first base layer, right? Because if you don't have

layer, right? Because if you don't have the user engagement and retention, then everything else doesn't follow. So we

need that actually to be solid in the center but we're not finishing with our jobs right there right. So we also building a second layer which is called page construction which you trade off

among different objective including also sort of a long-term uh value of the user which includes monetization includes social connection includes other sort of

utilities of the platform and then we're adding the last layer which is really optimizing for the experience join funnel because Roblox experience is way more complicated than a piece of just a

tweet. So through that I think uh we

tweet. So through that I think uh we actually can optimize for a multi-layered objective system to be still a very adaptive to our fastpacing

growth flywheel. So this is really

growth flywheel. So this is really tricky I think um all of the machinery you've shared because if we asked

someone to build an engineering system that optimizes fiveyear long-term happiness and retention that's really hard. Yes. Um, and I'm wondering

hard. Yes. Um, and I'm wondering Michelle if you can mention sometimes we use the term proxies and we we talk about we can't you know we need to know

what's going to be good for long-term retention in the first hour or the first day and so a lot of what we're working on is trying to find proxies for long-term retention. Can you talk a bit

long-term retention. Can you talk a bit about that? Yes. Uh I think there are

about that? Yes. Uh I think there are two factors uh to have a flourishing platform to retain users. uh one we optimize for user, we also optimize for

content creator. We want to improve

content creator. We want to improve ecosystem diversity on our platform so our user can find more and more new engaging uh content. On the other hand,

for each user, we want to make sure they found the most relevant content so that we found the most relevant content is a good proxy for users long-term

satisfaction. So what does long uh

satisfaction. So what does long uh relevant means, right? We need a proxy for relevance. Therefore, we use

for relevance. Therefore, we use engagement. Like if you are deeply

engagement. Like if you are deeply engaged with experience, that means the experience is relevant. If you're

willing to make a purchase, that means is actually in uh relevant to you. And

also if you are inviting your friends into the experience to play with you, that's also relevant. We keep looking for such proxy metrics to essentially

approximate short-term relevance to then approximate long-term user satisfaction.

That's cool. You mentioned something that's very subtle. It's probably the most technical thing we'll go into today, which is ecosystem health, not

just of users, but of creators. Now,

this is really difficult because short-term user optimizations are very different than long-term thinking about search and

discovery that makes the ecosystem healthy for creators to flourish and thrive. And so, there's a lot of

thrive. And so, there's a lot of decisions you both make where we could boost results in a month, but that might

not boost results in six months. And

because the results we see in 6 months are not just related to search and discovery, they're related to the health and happiness and business of all the creators on the platform. So that's

really subtle thinking about optimizing both user health and creator health. Can

either of you just comment like how do we figure that out? Like that's really complicated. Yeah, it is a journey,

complicated. Yeah, it is a journey, right? As part of the discovery

right? As part of the discovery evolution, I think we realize more and more discovery is not isolated island.

You cannot just do this through improving a certain piece of algo twisting certain objective function and achieve the amazing result. That's

right. We have to do it along with our creators. That's the key. And then what

creators. That's the key. And then what do I mean by that? There's a few se uh uh segments of it. Part of this is like actually consciously consider ecosystem health in the design of our system.

That's why I mentioned the separation layers design we just have. And then

part of this is also about having the right sort of measurement of success so that we know we're heading towards the right direction. But then the part of

right direction. But then the part of this is also about transparency like all the stuff we just talked about like the how we optimize what we optimize. So I'm

going to tease this one apart. So we're

saying two things here. Number one,

there may be parts of our algorithm we tune. Yes, that would show a shortterm

tune. Yes, that would show a shortterm user decrease, but we're going to bet that that over 6 months it's going to boost creator health and we're going to be in a better place in six months. So,

there's a little bit of a take the long view value there, which is super interesting. There's another um thing

interesting. There's another um thing you touched on. This is a big thing you tal you just mentioned that word

transparency because up and down our culture there's a lot of discussions around algorithms. This is politically

in the hot seat. Um there's some his history to non-transparent algorithms and we're going to touch on a few of them because I think this is actually really

interesting to the user base. Um, so

what so share a so let's go further.

What we're saying is we're starting to tell the creators on Roblox what our algorithm is. What are the factors that

algorithm is. What are the factors that are being weighted towards personalized what we show people and can you mention the connection all the way to our

dashboard like yes what does that mean?

Yeah, it means like for example, if we use a particular metric in our objective function, let's say the co-play that uh Michelle mentioned uh the action that user invite their friends to join

experience. We want to make sure that

experience. We want to make sure that shows up in the creator uh dashboard so that creator can see the metric of that particular thing uh for the experience

and up and down with the time series so that they can understand oh what is my action calling this metric go up and down so that how can I optimize my action. So this is pretty interesting

action. So this is pretty interesting right because a lot of companies have discussions around how transparent can we be with our algorithm. One of the

typical things for not being transparent would be people are going to game the system. If if we shared our algorithm,

system. If if we shared our algorithm, people would game the system. But I

actually I think we discussed I don't know actually I I'm actually pretty optimistic because um wouldn't gaming

the system if we have a really good algorithm that's transparent being mean the same as building an amazing creation on robots. Exactly. In fact in addition

on robots. Exactly. In fact in addition to transparency we also want to provide insights actionable insights to help creator to improve their experiences.

Therefore, transparency is only the first step. After transparency, we say,

first step. After transparency, we say, "Hey, here are the tools and here are the insights. Why don't you go back and

the insights. Why don't you go back and improve your experiences?" Yeah, that's right. For example, uh I'd like to

right. For example, uh I'd like to mention one of the recent very successful creator tools we built is thumbnail personization and then typically in other company they build it as a platform feature. It's not

something creator use like the platform control how many versions of this uh promotional assets and they pick one of the many for a particular user but we build it with the mindset that we need

to partner with a creator. this is their baby, their experience. They have a voice to decide what kind of a variance of the promotional copy they want to put on top of the platform. And through that

partnership, right, and then the creator really love this uh actually tool. They

adopted really fast. They also kept giving us suggestion, help us make the feature better. And then meanwhile, they

feature better. And then meanwhile, they have this dashboard to see how the action actually will fit into the system and how does that uh help them do the better job next time. It it's

interesting that um in the past before we were talking about full transparency on our discovery and recommendations algorithms, we would think about telling

developers how to be successful on Roblox, but is it is it possible that through full transparency of search and discovery and sharing an analytics

dashboard, we're putting essentially we're sharing exactly what we believe and like we we believe we're trying to do our best to show the best personalized recommendations. So it

personalized recommendations. So it almost comes full circle in look at the dashboards, look at our transparency on discovery. That's what we believe is

discovery. That's what we believe is really special. Exactly. And then part

really special. Exactly. And then part of this intentional transparency is also giving our creators agency as well as letting them tell us where we're actually still missing. I actually

firmly believe our ecosystem is so vast.

As a small team, we sometimes may not necessarily know what all the different types of content might be out there, what creators are thinking about to build, right? It's hard for us to

build, right? It's hard for us to anticipate. But if we tell them what we

anticipate. But if we tell them what we are looking at, they can tell us, wait, hold on, have you considered this other aspect of the experience that could be very engaging, very telling the retention? And then we want that kind of

retention? And then we want that kind of feedback. But without being transparent,

feedback. But without being transparent, you wouldn't be able to hear that. And I

think what's also interesting is as we start searching for the best six or eight or 10 factors to predict you know

fiveyear retention yes a lot of them are human understandable um and you know from memory I think it's do people come back the next day um is this an

experience where people are meeting with their friends is this an experience where you know it's fun that these end up being almost intuitive in a way.

Yeah, exactly. Okay. Super cool. Okay. A

couple other things that um so for people who are out hanging and talking about search and discovery, there's a term that sometimes shows up in all

platforms called cold start. Um so for me, cold start was um living in Minnesota going outside when it's 20°

below and like my car wouldn't start because it the oil was so cold. It was

like Um, it's interesting how that's metaphorically come to search and discovery. What what's what's cold start

discovery. What what's what's cold start mean when someone hits Netflix or Roblox or Instagram? What's that mean on our

or Instagram? What's that mean on our platform? Yeah. So, actually there are

platform? Yeah. So, actually there are two types of co-star. Oh, yeah. So, one

types of co-star is user co-star. Like

what you just said, when a new user showed up in front of a new product like Netflix or us, what do we do? And at

that moment we do know nothing about this user and the best luck is we can ask the user to fill some questionnaire maybe we can have a little bit crew right and then that's traditionally very hard for recommendation system to

recommend very relevant content to you because we know nothing about you. The

other type of co-star is content co-star. It's about a new piece of

co-star. It's about a new piece of content showed up on our platform. Again

we except for the creation related data we have very little user engagement data with that piece of content. And then

this content may have a hard time competing with especially the boring right very successful old content. So

for both type of co-star problem we also have and we've been in kind of the process of addressing them and I don't think it's a ending uh sort of a journey

in the past uh we have actually a dedicated co-star model to solve the user co-star problem over the past I think year or two we added a lot of

streaming features which actually means we index really heavily on new users real time activity y because you know we know so little about you so we want to be very adaptive to the new user. Yeah,

from a engineering point of view, uh content code start uh is actually less challenging problem than user code start because our content understanding model

have improved significantly in the past few years and also uh with such a big platform like roadblocks we can quickly explore and collect some user feedback

and immediately we know whether the content can go viral or not. Uh so in fact a lot we discovered a lot of popular content that was only published like two days ago. So this is really um

this is interesting and this is something that a lot of creators have been are are always asking about is if I make something new and it's played a

hundred times. Do I have a chance to go

hundred times. Do I have a chance to go viral? And I think what you're saying is

viral? And I think what you're saying is yes. Like we're focusing on this. Yeah.

yes. Like we're focusing on this. Yeah.

Now, in the midst of this, um, because we've we've worked so hard to, um, balance the long-term ecosystem health

with bubbling up new content. And I,

there's a bunch of names for this, right? There's like multi-arm bandit,

right? There's like multi-arm bandit, and there's like all these names for this. Um, we have to be careful because

this. Um, we have to be careful because we're now talking about content without a long history of play. Yes. And there's

many ways to make content without a long history of play look good. I can I can have an emptied base plate with an

amazing tie, you know, description and an icon. It says Dave super 50,000

an icon. It says Dave super 50,000 person medieval night battle and I can have a hu like this is the best new thing, but that might not be very good.

So there's there's some things we have to do to try to optimize that. Can you

talk about content understanding? Um,

yes. Uh, I think this is definitely uh there's always trade-off. We want to discover a content as quickly as possible, but meanwhile they have to be

high quality. Yeah. Uh, in this year we

high quality. Yeah. Uh, in this year we plan to invest uh significantly into content understanding and content quality because we start to recommend

more and more tell content and discover them faster. Yeah. Uh so we have set up

them faster. Yeah. Uh so we have set up workstream on all these clickbait detection and then the lowquality content and I think what we mean by

clickbait is rather than someone doing their best to make a cool Roblox experience something where the image the description the what we can see in the

place it just doesn't all add up. It it

seems like a little different. Yes,

there is a a fine line. The challenge

here is that there's a fine line uh between copy and inspiration. That's

right. Right. Because we want inspiration. A lot of a good creation

inspiration. A lot of a good creation whether it's art or architecture building come from inspiration. So

someone saw something successful on Roblox and inspired by it come up with a similar idea and make it even better.

It's good. It should be encouraged. But

if you directly copy it, then that's something bad. We want to not the way

something bad. We want to not the way the way I've thought about this is we only have so much room to mix all of the top experiences the up and cominging experiences with the new experiences. We

want to do that in the most optimal way possible. So if for example half of all

possible. So if for example half of all the new experiences coming out of nowhere are mislabeled or misicon they're actually taking the space of the valid efforts. So I've always thought

valid efforts. So I've always thought about this as making it more efficient for the people who are really trying to create. Yeah. not taking the good oxygen

create. Yeah. not taking the good oxygen from the good experience. That is

totally true and I think we are as Michelle said still early days in fighting against this problem. Uh

because the copycats, the scammy games, the misleading content, teleports, we are discovering them, detecting them, filtering them. But the problem is it's

filtering them. But the problem is it's a little bit like a whack-a-ole. If you

don't know what it is, you don't know how to detect it based on the current traditional approach because you have to like understand it, label it and then learn a machine learning algorithm and

then deploy it. But in the future, we are looking for even more scalable way that without even understanding the pattern. Can we actually spot it before

pattern. Can we actually spot it before it become a big thing? And that is a hard problem. Yeah. And this is rooted

hard problem. Yeah. And this is rooted in actually deeper understanding of our own content which is not just about its metadata, its title, its promotional assets, but also what's actually really

going on within the game, within the experience. What's really given me

experience. What's really given me comfort about this is we're not necessarily something that I don't think we really could do would be understanding content to know what's

going to be great. like that seems impossible. Our users need to do it. I

impossible. Our users need to do it. I

think the way I think about it when we say content understanding, it's understanding really where it just doesn't add up. Like something's really weird here and we don't want to steal

the oxygen. Um, so that's that's another

the oxygen. Um, so that's that's another fun complement to the cold start problem where we're trying to cold start content as well as have users have a great first

experience. Um, so, so to frame this in

experience. Um, so, so to frame this in everyone's mind, we've been talking about discovery, which is you look at something and you see a lot of cool

things that are intriguing to you, but there's a whole other way that people interact with shopping, interact with their home video theater, interact with

games, interact, and that's search. And

search is completely different than discovery. Um, when I think about

discovery. Um, when I think about searching, um, whether it's it's on a shopping site, um, or somewhere else, I

typically when I'm shopping, I sometimes don't know what I'm looking for. I just say, "Hey, I want to look at, you know, ultra

portable cameras with a good lens." Uh,

whereas when I'm on home video, a lot of times I enter the the name of the movie, for example. Right. So maybe to start is

for example. Right. So maybe to start is search really both of these things knowing exactly what I want or roughly what I want. Yeah. Essentially uh you

can imagine for organic discovery user doesn't have to input a query right we understand the user intent from the past user behavior or similar user for search

discovery actually we get some golden queries from users. What's a golden query? Oh, so essentially the user

query? Oh, so essentially the user queries is actually contains a lot more information. I shouldn't say golden

information. I shouldn't say golden because in technical terms golden means there's a set of standard queries. We

have uh correct answers for them. But

users with queries actually give us an extra bit of information. In fact, our team has been collaborating with our PhD interns to see whether we can have the

multi-turn recommendation system. So,

can you give an example of what would be thought of as like one of these types of queries? So, I come on uh to Roblox as

queries? So, I come on uh to Roblox as hey I like dress to impress. Can you

suggest me something similar? Oo. So,

that's the first turn and then they suggest something to me and I would like I like these two but not the other two and then that's the second turn. Through

multi-turn conversation, we actually can gather a lot more explicit information from users and therefore we can tailor our search and recommendation experiences. I never even thought of

experiences. I never even thought of that. So I' I've never thought of when

that. So I' I've never thought of when I'm shopping, for example, or on Roblox of having a more complex dialogue, show me something like this, but not like

this and more like this. Yeah. So is

that is that a general thing we should expect in all these products? Like a

much more complex search line? So I

think this is a speaks to early prototyping days. Yeah. I think it

prototyping days. Yeah. I think it speaks to the evolution of search because we just talked about evolution of discovery earlier maybe more from the recommendation perspective. From search

recommendation perspective. From search I would say up till today a lot of our work is really about making search better as a companion like what you said you have a particular intent in mind

whether it's a particular experience or types of experience right and we're trying to find it for you. Uh but what Michelle's talking about is actually the next stage. We are aspiring to go

next stage. We are aspiring to go towards that. I'm not sure if a

towards that. I'm not sure if a multi-term query is the only solution but we definitely want to connect that.

It's worth thinking about it, right?

Yeah. Like a concage because we're used to link search and whether we're using Bing or Google or some other thing and now all of society is going through complementing that with an AI super

supercharged type search. And when I think of shopping just this morning, I was looking I was literally looking for

a mini uh stereo stereo auxiliary cable 6 ft long white and I could feel that I won't name the shopping site. It was

having trouble putting that all together and it was doing more of a context match rather than saying okay they have to be 6 feet long, they have to be white, they have to be a mini ox cable, let's only

show those. Right. There's a logic in

show those. Right. There's a logic in your intent. So that logic was not being

your intent. So that logic was not being picked up. So I think what you're saying

picked up. So I think what you're saying is search may start to go down though.

Yeah. I think as user are being trained by all this AI products right to be actually feeling natural to have a dialogue with a box. I think this behavior became potential through mature

and that's where potentially we want to explore too and then but at the baby step just also as a PM I like try to put things also in a practical terms you know in the baby step we're building a

search landing page um as a way to sort of starting concierge uh journey so user can come to the search box maybe have intent but immediately get value without

even typing a single term because they can find their recently played experiences trending ing search queries or their own queries or maybe some sort

of categorical experiences related to their sort of history and over time you can imagine that search landing page can become the place where they start this

kind of multi-turn journey as well.

That's super interesting. Yeah, I just learned something. The future of search

learned something. The future of search actually is going to combine a lot more human understanding. Yes. So much

human understanding. Yes. So much

potential for shopping sites I think as well as Roblox. Another feature of search we also start doing is no longer just a one single entity one search product. We used to have several search,

product. We used to have several search, right? I mean, we still do ways in

right? I mean, we still do ways in Roblox uh platform, right? Because on

Roblox, I can search for my friends, I can search for experiences, I can search for a piece of clothing, I can search for even an event. Yes. But in the user's eyes like we're one company, one

app. So we start putting things actually

app. So we start putting things actually together. So we have this whole vision

together. So we have this whole vision of Omni Search. Recently we have already putting people search together with experience search. Next step is also

experience search. Next step is also bringing marketplace items to the same Omni search as well. So user just go to one place and they can find all the things on Roblox. And the the um the

results are really good like the and I know we're automatically understanding a little is that more likely a username or is that more likely an experience for example

and and mixing the results and that's been very promising. I think it it implies a much cleaner UI, right? search

is search wherever it is on Roblox.

Really excited about that as well. Um,

so couple other things. Um, so now let's get a little controversial. Okay. Um,

because every all companies are trying.

Um, but I want to and first I want to highlight what I felt happened in 2016 and has been since corrected and they're doing an awesome job. I know I'm

maybe even one or or both of you worked at a company there. But in 2016, which was eight years ago, I can remember going on Facebook and I can remember um

the election was going on at that point in time and I I felt more than being attracted by pictures of my friends cats, which I I felt was more of a

retention mechanism from seeing interesting stuff. Right. I it was

interesting stuff. Right. I it was getting into the thing of starting to share news clips by friends and I could see all of my friends from high school

and college posting clips in different very different types of clips and it almost felt like the retention was switching to being either really like

this or really don't like this. It was a little bit more of almost a moral outrage or a moral connection switch.

And then I know that over the years Facebook ultimately said, "Yeah, let's move a little bit more to user health or happiness."

happiness." So it seems like this is a little bit of one thing to be aware of when designing a search or discovery system. And it

does feel that might be related to short-term versus long-term optimization. Do you is that how you

optimization. Do you is that how you would interpret it? Maybe a short-term optimization versus long-term optimization. Actually, I was there uh

optimization. Actually, I was there uh when this happened. Uh I was actually at Instagram. Okay. Uh experience society

Instagram. Okay. Uh experience society uh part of that sort of a phenomenon I think a backend uh whether Facebook or Instagram fee optimization objective is

really engagement and then actually short-term engagement just time spent.

That's right. And then because of that you drove into several issues. One is a popularity bias. Now we very familiar

popularity bias. Now we very familiar with that or you call like a burning effect. And we want to highlight this

effect. And we want to highlight this was very early in the search and discovery things and Meta's subsequently become extremely good at it. So this is more like just an industry discussion on

short-term optimization versus long-term optimiz historical sort of episode we went through. And then so the other part

went through. And then so the other part of this is pushing actually some of the creator new creators harder to climb up.

So there's a a declining of the sharing.

And then the third issue was a filter bubble. That's what you're talking

bubble. That's what you're talking about, right? you potentially got pushed

about, right? you potentially got pushed into those extreme sort of bubbles only see certain type of thing or the other certain type of thing and then forces thing is um you start seeing people's uh

connection with their actually very close friends actually falling off. So

this is shows a good example when discovery is a single-minded optimizing full objective how dangerous that can be. Well, it also shows um in a sense

be. Well, it also shows um in a sense one of our values is we are responsible, right? The massive amount of

right? The massive amount of responsibility that both of you are dealing with all the time. And I feel the um the transparency is a good check.

It's a little like open source discovery, right, kind of thing. Um so

that's good. Uh and the notion that um the better our discovery is, the less gameable it will be. like we it'd be cool to imagine someday we can just

share our whole algorithm and because it's optimizing what we believe is healthy is really cool. Um I want to ask a question around um now another

controversial question. Um are you at

controversial question. Um are you at all on either shorts, reels, Tik Tok or spotlight or any of those because I'm on

all of them. I I am on you. Sure. Okay.

So, so I'm on all of them and each one of them has for some reason zeroed in a certain part of me like like I was mentioned one of them is a little bit

more for me um interesting you know cats fighting with ducks kind of thing. One

of them for some reason is just all kinds of therapists telling me how to get better. So that so that's kind of

get better. So that so that's kind of interesting. Um, one of them is all

interesting. Um, one of them is all around cars and rockets and motorcycles and rebuilding engines and things like

that. And then one of them is all about

that. And then one of them is all about political stuff. Um, just like real time

political stuff. Um, just like real time political. So I'm wondering if what that

political. So I'm wondering if what that means is even though it's fairly wellnown, is it possible that each of these companies has a little bit of

their own discovery algorithm?

Is like such a thing possible? Yes. uh I

think uh all of them are actually powered by uh some of the most advanced discovery algorithms in the world.

However, as you can tell as a user uh those most advanced discovery algorithms are not good enough for us. That's

right. Um so we have a long way to go and aspects of it I think speaking to the creator ecosystem they could very like to have the exact same algorithm.

you as a user at the end of the day may still experience different content because different creators on the different platforms. That's why it speaks to the importance of sort of a

transparency and collaboration with creators. So for example, I when I was

creators. So for example, I when I was at Instagram, I remember clearly that there's a community management team is actually go out of their way to work with the community, the creator

community to come up with different themes, different types of content trends to help sort of the creator ecosystem house so to speak. Similarly,

Roblox, we have dev rails, right? work

very closely with our creators listening to what they want listening to what they need to develop the next generation content for Roblox and those efforts don't go un uh noticed because

ultimately when discovery is powerful content is a king so you need both to be a successful content platform that's my firm belief so the more these two can become closer to be together and amplify

each other the more successful that platform can be yeah I think we should also try to get explicit user feedback when you are getting frustrated there's right now there's no way for you to tell

them I don't want this right that's right so definitely explicit but I do give them feedback because they just start flicking through videos so quickly like don't like it don't like it don't like it that's right that's right yeah

or multi-turn uh recommendation actually this is speaking to a a less known sort of a category of discovery we talked about search driven recommendation driven even as driven discovery but then

another type of discovery is kind very serend. So sometimes some of the uh

serend. So sometimes some of the uh platform purposely implement a function called like I feel lucky or random choices or explore or discover they have

different names. The idea is for someone

different names. The idea is for someone like who's frustrated you can get out of your bubble and try something completely new and if you end up finding something new then they can put that into your

main feed. So that it's interesting to

main feed. So that it's interesting to think if that is a response to a nonoptimal algorithm that is ending up in like a topographical minimum versus

you know it's like it's like oh my gosh we failed random jump start. Yes. Like a

local optimal like jump you out of that.

Okay. I want to um jump just into any future visions that you're thinking about either of you for search and discovery. Some of the ones that I have

discovery. Some of the ones that I have tossed around as we think about our long-term objective um is the notion that it may be different for different

ages. So, and this would be not any

ages. So, and this would be not any promise feature, but I I am optimistic about us thinking because in many ways we think about our product relative to

age. It's interesting to think what

age. It's interesting to think what would be our optimal objective function for a six year on six-year-old on

Roblox. And um we might lean more

Roblox. And um we might lean more educational for that objective function in certain ways. We might have a different objective function for someone

who's 40, for example. So that's my idea. Stay tuned. Um because I know we

idea. Stay tuned. Um because I know we we all get together a lot and talk about things. But I'm curious about what your

things. But I'm curious about what your if both of your ideas of where we might see this go. Yeah, I I would say we need

to balance discovery versus uh security and civility. Uh you actually touch upon

and civility. Uh you actually touch upon that point. Uh a lot of the large

that point. Uh a lot of the large language models uh have this jailbreak problem, right? it's going to be even oh

problem, right? it's going to be even oh so that basically means I start to ask question and eventually the large language module say something that's completely outrageous which is which is

I think what hundreds if not thousands of extremely creative Roblox users would love to see happen right they they love to push the system in more of a creative

way yes especially when we have 3D generation right when each individual component looks secure and civil and how do you combine them together to make a

game play that's a jailbreak. Uh so

that's definitely one of the challenging problem we need to handle especially for a metaverse based uh platform. Nice.

I'll speak to where do you think it's going? Yes, I'll speak to two things.

going? Yes, I'll speak to two things.

One is more short-term, one is maybe more long-term future. And first of internally uh we within even the discovery org we're trying to build a lot of the components in a scalable way

so that page construction don't just apply to home but can be applied to search to charts to EDP experience detail page right and meanwhile discovery as a platform can also be

helpful for marketplace um social ads but eventually can also be as a platform for creators as what you just uh alluded

to um the more future sort a vision I have in mind is speaking to this component of discovery and content needs to work together to really create a

magic and I'm seeing this happening because of thanks to all the advancement in AI now with generative AI the same types of creation data not only useful

to create kind of a awesome experience it can also be useful as content co-star and connect back to the user get feedback so the creators can understand

what works what doesn't in a much shorter period of time instead of years.

That's very interesting. So one startup was looking into it like Sunno they does uh this music generation. So within

maybe 10 seconds you can create a very awesome catchy song with a bunch of metadata you put in but then smartly use the same metadata you generate you use to generate that sound for

recommendation. We It's interesting

recommendation. We It's interesting because there's been several conversations with the launch of our 3D generation model, Cube 3D, that that

meta data is very important to couple with that model. And when when I make a 4x4 dune buggy with red chrome and whatever, aligning that metadata and

keeping it with that model, correct, is actually the root um description of what made that thing in the first place. And

I think Roblox has this natural uniqueness and advantage in this because not like other content platform like Netflix. All our content are created in

Netflix. All our content are created in our platform. We know all the data. So

our platform. We know all the data. So

we don't actually need to gather them or convince creator to have a high adoption to gather them. We just need to connect the dots. Right? And then secondly, we

the dots. Right? And then secondly, we have already seen a small feedback loop is f uh formulating with the thumbnail presentation. We gave the creators the

presentation. We gave the creators the tool, they generate those thumbnail, we give them feedback, they come back with better thumbnail. Now imagine if those

better thumbnail. Now imagine if those thumbnail, we also help them generate.

And now they can even do better job with this whole loop. But they can know very quickly if this thumbnail is doing well or not. I mean why the machine learning

or not. I mean why the machine learning algorithm behind to generate and recommend it will get a feedback as well and get better over time. So I'm really

excited about this potential future for discovery as well. So, how far do you feel right now we are along the

discovery vision? So, if I said it goes

discovery vision? So, if I said it goes from zero to 100 when I know all of the stuff we're working on and um so I'll go

first. I actually feel like we're 27% of

first. I actually feel like we're 27% of the way. How do you both feel? As an

the way. How do you both feel? As an

engineer, I'm with you. Uh engineers are always very pessimistic, but I know Chen, she's very optimistic. That's why

whenever she said I said I need to move it by a month. No, actually I think I'm maybe more u well I wouldn't say pessimistic but I feel we have a long

way to go. We still have a lot to build.

So we're probably in the 10 to 20 region as well. Mine is more around knowing

as well. Mine is more around knowing what we're thinking about. Discovery is

going to get a lot more fun over correct. Same here. Same here. Discovery

correct. Same here. Same here. Discovery

is going to be so fun and so intimate and so personal. I think there's a lot of room here. Yeah, because I I don't just look at the algorithm innovation.

I'm very confident we can push out. I

also see there's a tons of new UI patterns, interaction we can put on to the platform given we are 3D experience platform, right? Look at what we have. I

platform, right? Look at what we have. I

think there's a long way to go from 2D to 3D. One day we may have a 3D

to 3D. One day we may have a 3D discovery experience for 3D experience.

That's what I'm hoping one day we can realize. So from that sense that's what

realize. So from that sense that's what we're at 10 to 20%. A lot more product use cases. Yes. Exactly. What your um

use cases. Yes. Exactly. What your um and also on the search side as well as you were talking about AI integrating into search. One could imagine uh

into search. One could imagine uh whatever happens with virtual clothing will happen with physical clothing. And

so as you were talking about that it was making me think I have many avatars including one that's my same shape and size. And if I go to um say I'm going to

size. And if I go to um say I'm going to the Academy Awards, not that I'm going to go there, but if I was in the audience in the Academy Awards, I would probably say, "Hey, I'm going to be

going to the Academy Awards in 3 weeks.

I need a stylish tuxedo that's a little, you know, something I wouldn't normally buy. I want to lean to blah blah blah

buy. I want to lean to blah blah blah and see my avatar with all of those items um as part of maybe even physical world shopping." So, yeah. Um so AI for

world shopping." So, yeah. Um so AI for search that popped into my mind. Very

exciting. Yes. Huge opportunity. Yeah

definitely. And then there's another aspect of this is also live ops. I think

we start this whole experiment from early last year right and seeing so much success working close side by side with our creative community pushing out awesome updates new experience events

for our community. And right now our frequency cannot be too high although it's already quarterly but I imagine one day every single day there's some culture moment there's some event

happening and then we working together side by side with a creator putting it up onto the platform that will be so much fun as well. Yeah that live ops thing makes me feel like one of the four

short form videos that I use you know feels more like real time in a way others don't. So live ops has that real

others don't. So live ops has that real time component. Yes. Exactly. Okay.

time component. Yes. Exactly. Okay.

Well, hopefully if you were listening today, you know now what search discovery and recommendations are and and who could have been better to share it with than Chen and Michelle. Um,

thank you both for joining me. I think

our audience learned a lot today. Really

appreciate it. Thank you. Okay. Thank

you both. We're happy to be here.

[Music]