Yale Engineering Dean's Invited Speaker Series featuring Gus Fuldner, Senior VP, Core Services, Uber

Welcome back to the deans invited speaker series. Uh I'm Jeff Brock. I'm the dean of the school of engineering and applied science. Uh glad to see some familiar faces but also some new faces. So

uh so welcome. Uh quick acknowledgement to Tsai CITY who to whom we continue to be grateful for letting us use this amazing space for these events. Um and uh I'm really pleased this so basically what's the point of these events? Basically, we have lots of academic talks on our campus that you all go to and see, but what we don't have as much of and what we're really trying

to to lean into a bit is uh bringing folks from industry uh to visit the school of engineering so that you get a sense of kind of what's next. So what you're heading toward as you, you know, toil through the salt mines of your academic career and and learn all of these, you know,

engineering tools. It's good to know kind of what what lies ahead and what you can look forward to,

engineering tools. It's good to know kind of what what lies ahead and what you can look forward to, but also what folks are expecting and what what they're hoping you may pick up along the way. Um,

you know, Yale's a very intellectually diverse place. There's more to it than the classroom. Uh,

but sometimes it's good to know kind of what what things you might want to be thinking about. So,

I'm I'm really excited to have our alum Gus Fuldner here from from Uber and class of 04 uh to to talk to us a little bit about that. He was a CS econ major, I guess, in its relatively early days as a major at Yale as a sort of now a very popular joint major, one of our most popular

majors. Um and he is you know a great example of of what you know what careers people can end up

majors. Um and he is you know a great example of of what you know what careers people can end up doing the different exciting things they can end up doing when they go on from engineering. Uh so

after he got his after Yale he got his MBA from Stanford uh and joined a joined benchmark which is a tech VC firm which is one of Uber's first investors. Uh he's also they also invested in mobile apps like Snapchat and Duo Security uh which we use to log into our campus network

as you may know. Uh Gus is a senior VP for safety and core services now at Uber and he oversees many crit critical infrastructure functions within the company along with executive roles. He's done some angel investing including taking risks on some Yale startups. Um and he served on the boards of

many of these ventures. Um we were talking earlier about the way that the sort of proliferation of exciting new AI tools um both in terms of you know their language applications but also in terms of engineering coding and the effects they're having on the way people think about engineering

problems. He has some great examples of this. Uh so um he's gonna present uh some slides uh for about a half hour and then we'll do a little Q&A and we'll open it up to you guys because you tend to have the most interesting questions. All right. So thanks Gus. Take it away. Perfect.

Uh do you hear me? Great. Uh well thank you Dean Brock. I'm really excited to be here and be back uh be back on campus. Um so I've got um a little bit of a talk here. I was trying to think about what would be um what would be interesting. So have a talk about uh very applied AI um and some sort of real world problems uh within Uber um particularly focused around uh the

customer support domain which I thought would be um relatable and um and interesting. Um but uh let me uh let me dive in. Um so a little bit about um my journey. So I was a uh CSON uh major here. They

were actually two separate majors uh at the time. um went to uh um was involved in entrepreneurship at Yale. That was around the time the Yale Entrepreneurial Society was uh was founded. Um

at Yale. That was around the time the Yale Entrepreneurial Society was uh was founded. Um

uh later on my uh later on I went to Stanford uh Stanford GSB for for business school and then um venture cap venture capital at Benchmark. Um I was at benchmark at the sort of very early days of the uh of mobile internet. So some of our investments there were um uh Instagram, uh Uber, Snapchat. Um

and I think VC is really a great opportunity to sort of have a front row seat on innovation. Um

the way I would connect those three companies is that each one of them was sort of the canonical um kind of killer app for a new sensor on the iPhone. Um so uh Instagram is the camera app.

Um Snapchat is the forward facing camera app, the other camera. And um Uber is the u the GPS um app.

So having location aware services. Um so um you know those companies have all gone on to be uh be quite successful. Um and I can guarantee you every time there's a new uh new version of an iPhone um

quite successful. Um and I can guarantee you every time there's a new uh new version of an iPhone um I'm paying attention to what are the um what are the new sensors and capabilities um there.

But um from the very early days of of Uber and um I was one of the first uh first users to sign up in San Francisco um you know I really loved the um the magic of the experience that they created in San Francisco. There was also a very personal connection for me back to New Haven. Um and that

was my frustration with cabs in New Haven. U so um used to be called Metro Taxi. I looked it up last night. I think it's now called M7. um you would uh dial uh 77777 and um you would request a taxi

night. I think it's now called M7. um you would uh dial uh 77777 and um you would request a taxi and um be pretty confident it wasn't going to show up. Um and so u even when I would come back to um campus after graduating because I was involved in uh some startups here um you know I could

very easily get from uh from the train station to campus but could never get never get back.

That's um something that you know when I first uh first discovered the idea of Uber um I was like that's absolutely needed in New Haven. Um so it is I'm pleased that it's now here and was really excited when we um when we finally got to launch um launch in New Haven. But um looking at um kind

of about Uber um we're obviously known first and foremost for um mobility starting with um you know transporting people from point A to point B. We've expanded that to a bunch of uh different flavors of transportation, shared transportation, different modes, um like motorcycles and scooters

and um uh those sorts of uh things. Um and that and that is sort of the uh the go anywhere part of Uber. Um we've expanded that to get anything as well with um Uber Eats. Uh starting there with

of Uber. Um we've expanded that to get anything as well with um Uber Eats. Uh starting there with uh restaurants, expanding to uh grocery, retail items, um all sorts of uh different things there.

And you know underlying u both of these services is a set of uh technologies for us to um aggregate um uh uh gig workers in the real world and connect them with uh people who need something um something to move. So it's all about kind of reinventing the way that uh transportation works um using um mobile phones with kind of geoloccation awareness to uh to make that happen.

So, a little bit about the uh scale of Uber. Um, you know, when I uh joined Uber, uh we were only in a handful of countries. We're now in um 70 countries around the world, over 10,000 cities.

Um millions of trips a day. Um from an engineering perspective, um on a a typical busy day, we're doing um a million concurrent trips. Um so there's a lot of complicated work one needs to do from a distributed systems perspective to uh be able to make that all um happen uh in real time and manage

all the data flows um uh through that. Um but and you know I think it's important to understand that uh Uber is uh more than kind of just an app. Um you see the sort of um straightforward um experience of okay I request a car uh push a button a car shows up. you have that sort of remote control for the real world experience. Uh but in the background there's actually quite a lot

um that uh goes on um under the hood. So these are examples of some of the different um services that uh need to run in the background uh to make uh make that everyday experience um happen. Uh we

need need to know who users are. We need um to be able to u manage safety, manage insurance, u manage fraud and risk. um figure out things like um pricing. Um uh we'll talk later about uh support. Um it's very easy to think about the core experience when everything's go right,

uh support. Um it's very easy to think about the core experience when everything's go right, but what happens when something goes wrong and how do you deal with that from a a support perspective or a safety perspective? So um I lead a team that leads a lot of these functions, particularly those towards the um the left side of the page. Um and um you know, these are functions that um are a

combinations of of operations and technology. So um I've got more than a thousand engineers building the the systems underneath this and then um uh people who um also deal particularly in the in the support domain um with um uh the kind of human dimension of uh services uh around

here. But overall a lot of complexity um under the hood that's not always um obvious when you

here. But overall a lot of complexity um under the hood that's not always um obvious when you um uh just use the consu use the um experience because a lot of this stuff kind of disappears into the background. Um so how are we using um AI at Uber? Um I think there's sort of three broad categories. Um I think it starts with uh machine learning and this has really been a core part of

categories. Um I think it starts with uh machine learning and this has really been a core part of Uber from uh very early days looking at problems like um routing uh matching. So how do we assign um riders to drivers um in Uber Eats? How do we rank uh what restaurants we show you? What menu

what menu items we recommend? um uh problems in in fraud um various sort of classification ranking type problems um lots of uh machine learning uh across Uber. Um we then uh over time as uh uh physical AI and autonomy has um advanced uh spend u more of our time in um in in autonomy and

self-driving vehicles. Uh so uh we have a model where uh we used to be developing this inhouse.

self-driving vehicles. Uh so uh we have a model where uh we used to be developing this inhouse.

We moved to a model where uh it's really a partnership model where we have um publicly announced partnerships with more than a dozen different providers of uh self-driving vehicle technology on everything from sidewalk robots to uh four-wheel cars transporting people um to even

uh 18-wheelers in uh in our freight division. Um and so um there's a lot of um interesting um interesting advancements in this that space. We participated participated in it um by um uh helping um helping these companies uh develop that technology providing a lot of the data uh

to train those models uh those sorts of things as well as being the place to uh commercialize them and there's been um tremendous advancements in this um in the space and I think as we look at safety in the future um there's a lot that will come from autonomy making a fundamentally safer um uh safer road for everyone. Um and then the last area is generative AI and LLM which

obviously is um been a huge area of development over the last uh few years starting from uh the initial transformer research into kind of the chat GPT moment and um we'll double click on some of the different ways that we um are applying those technologies today. So um Genai has a bunch of different um different applications. If I zoom out and look sort of across Uber uh the first

category I would focus on is um productivity. Um, and that's how do you make employees um, uh, more efficient. That's everything from using um, uh, chatpt or Gemini or those types of things in

more efficient. That's everything from using um, uh, chatpt or Gemini or those types of things in your everyday uh, work. Um, but the uh, probably the domain where there's been the most obvious impact has been coding. And um I'm sure many of you have used uh tools like uh cursor and cloud

code and those are great for prototyping. Um but they're also great um increasingly for um very large scale software development where we've got a very large uh code base of you know many many many millions of lines of code um thousands of developers working on it um but lots that you can uh do to make developers more efficient um uh to improve the quality of code to do automated

testing migrations all those uh sorts of things. Second domain is um new product experiences. So

here we've got a screenshot of a uh experience we recently announced uh with uh with OpenAI where you can embed the um Uber um uh uh request experience for both uh um passenger transportation

and Uber Eats um in um in the um OpenAI chatbot. Um, so we use um, MCP to give them um, access to information about the restaurants platform or what's the pricing to get from point A to point B, what's the ETA, those types of things. Um, and you can um, fit that within the um, OpenAI experience.

Um, we're also building agents uh, and agentic experiences within um, within Uber. So, um there are experiences um that you could imagine like I'm taking an Uber home from the airport. Um arrange

that trip for me and order me a my regular order from um Joe's Steakhouse to be available when I get home. Um that's a more complicated order. Um something that you can describe verbally.

get home. Um that's a more complicated order. Um something that you can describe verbally.

Um you could imagine an agent um sort of wiring that up in the background to make it happen. um

something that doesn't necessarily fit as well with the sort of traditional Uber um uh mobile app UI experience um but works really well with uh with an agent. We have teams um building those um those types of experiences. And then the um the last domain um and um one that we'll uh spend more time talking about here is um is customer service. And um customer service I think is um

sometimes uh people might think of as a a bit of an afterthought or you know kind of not the core of the experience but I think um really it is a really fundamental part of the experience. Um when

you think about engineering generally before you think about Uber in you know what are the failure modes what are the defects and how do you address them and uh this purpose of a customer service team is to deal with um the situations where things uh go wrong where you're not on the um we call it the the happy path um and um how do you fix that um scenario and um that's a really wide

um wide class of problems and one that um I think very early on as uh loss were introduced people said hey this would be something that um you could apply to customer support. Um but actually making that happen is um is pretty complicated. So um you give you a sense of what customer support

looks like um at Uber. So um we have uh over half a billion uh customer contacts a year. Um and

um there's a huge complexity um underneath this. So this comes in um many different languages in many countries across uh uh the whole variety of different uh services I showed on the um on the earlier page. And there are um hundreds of different types of um issues that people have.

Uh they can say I left my phone in the back of the car. I was um incorrectly charged a toll. Um any

variety of uh different uh different issues. And you we've had like many companies various forms of automation in kind of the treebased automation form. If this then that you can see that in a a phone tree or in um kind of uh automation that's really fundamentally going through some process

in the in the background. Um and then we have a large um uh network of customer service agents um tens of thousands of them at uh different sites uh around the world to provide 24/7 support and cover all the the languages. Um and so the question is um that that was a very obvious question as um uh

LOM came out was you know how can you apply this to um to customer service and um there are um so let's let's talk about um the case of uh missing tacos. So let's say you order from Uber Eats from your uh favorite uh taco restaurant. Uh you're um going to watch a movie uh tonight. you order

uh order your food and the burrito and quesadillas arrive but not the uh not the chicken tacos and so what has uh what has happened and what do we uh do about that when a customer uh writes in or calls us or however they contact and say hey I'm missing missing my tacos and it turns out that that's a

um that's a reasoning problem um and uh you need to figure out why are the tacos missing what do I do about that what's the the policy um how do I um how do I respond to that? Um and you need to figure that out in order to figure out um what the next steps are. Um and that's ultimately

what what our agents um are doing. And um so how do you make this work in an in an AI um world? Um well, a bunch of different um different techniques. Um started out with

um world? Um well, a bunch of different um different techniques. Um started out with um why don't you use general purpose uh AI models in 2022 2023. Um, you can think of, uh, opening up a a chatbot and saying, uh, creating a prompt saying, "Pretend to be a friendly customer

service agent from Uber. Um, here's some, you know, sort of general, um, general guidelines.

Here's what the customer said." And, um, you turns out that you get a not great um, answer from that. You get something that's not terribly useful. you're taking something that's um trained on, you know, everything on the internet um but doesn't actually know much about

um either Uber Eats uh or about um the actual um taco order we're we're looking at. And so

um you get to you get answers that are um very apologetic um and kind of recommend that you you know you might go um try asking Uber or something like that. just answers that are not terribly um terribly useful. Um so the next step was um why don't you train on prior support interactions? So

we've got um the history of um all the different interactions that have uh happened with our um traditional support agents. Um why don't you use some of these um various approaches like um fine-tuning and reinforcement learning and that kind of stuff to learn from that. Um and you can get to a better result for this. you get um a lot better um you use you get a an agent that uses

brand language. It um is talking about the sort of context of the situation. Um but you tend to

brand language. It um is talking about the sort of context of the situation. Um but you tend to um get something that responds with kind of the modal answer. Um so ultimately LOMs are um at some level token predictors and um so it ends up giving like this is the average answer for a missing item

or this is the average answer for someone who's missing their phone. Um but the specifics of that um scenario are actually um quite important and if you don't have that u that context and the ability to reason about it um it doesn't work. Um and so really the the path that's gotten um far more traction and is is showing progress is um to to look at um reasoning models and what can we um

what can we build from that. So um what do we need to do to create um a uh an AI agent um that can help address uh the missing taco? Um so I think that starts with um sort of the core um agent system. So you have a foundation model that can do reasoning. Um at this point there are a variety of

system. So you have a foundation model that can do reasoning. Um at this point there are a variety of um foundation models that are very good at um at core reasoning. Um but what they don't have is context. Um and so the first thing we need to do is give them context um about um Uber generally

context. Um and so the first thing we need to do is give them context um about um Uber generally but particularly context about the order. Um so um what was the specific order? What's the history of this merchant? What's the history of this courier? What route did the um order take? Was it delayed?

this merchant? What's the history of this courier? What route did the um order take? Was it delayed?

Um what's the history of this consumer? Um to give it a set of uh facts to reason about. Um

we then need to give it a policy. Um this is the similar to the support policy that we would give to a traditional agent uh but a kind of somewhat more general principled uh principles based version of that u to the agent to say you know here's generally how to approach these things.

If it's the u merchants's fault do this. If it's uh a high value uh consumer do that. Um if this consumer complains a lot you might want to check for fraud or something like that. Um, and so you give it the set of uh uh give it the policies um and then you give it the tools um and those are

tools to take action like go refund this user, go um order another taco and send a um send a courier to go replace that as well as tools to sort of go do um do research and gather other facts. And

then the last thing you need is guard rails. Um so if you're giving something a tool to give out $5, you need to make sure it's not giving away $5,000. you need to make sure it's not uh um jailbreaking that you can't jailbreak the prompt. Um uh those uh uh those types of things. And so with this sort of core set of tools, you can build something that can um respond to the basic question of um

someone's writing in saying they're missing their their taco. What do you do about it? You can then um wire that up to a bunch of different uh channels. So you can think about interacting with this in um in a text chat based um interaction or um in voice. And I think this is an area that's um really increasingly interesting as the capabilities get better. Um voice turns out to

be really hard um because you need um really fast latency. Um you get a lot more about like tone and tambber and tenor. you know, that kind of um uh what's the feel of the the voice, not just what is it um what is it literally saying. Um that's an important part of uh people feeling like they're

interacting with a an agent that's um helpful and useful. Um the other thing we have to do is connect um connect this agent to other agents. Um if we design something that solves for a missing item, um a consumer will not just call and say, "I'm just asking about a missing item." and they

say, "And I had this other question. What about my Uber one subscription um or membership product or um something like that?" And so um building this concept of a planner that can say, "Hey, the the topic is shifted to something else. Let me route you to this um this other agent that can um do something else." Consumers don't neatly fit in the design of the way we architect our systems.

They fit in, you know, however they want to um interact with these uh these agents. And so we need the tools to be able to take those exception cases and and move them around or ultimately say this agent this agent bot doesn't know how to solve this problem. Let's get you to a human um that can. And um this is a key part of kind of getting out of that um feeling that people don't

like in a phone tree which is you know says press one for you know one two three or four but you're like well no I want something not on this list. Um being able to handle that is a really poor core part of the experience. Um then there's a bunch of other things that you need to uh uh

need to build as well um to um really understand the function of these agents. So um fundamentally um LLMs are probabilistic systems um and that's a big um transition from you know what we typically what a company typically wants to build which is something that we know exactly what it's going to

do. Um, and uh, here we've got um, a conversation where we're sort of handing the conversation in a

do. Um, and uh, here we've got um, a conversation where we're sort of handing the conversation in a much more open-ended way than a treebased approach um, to a um, uh, to the AI. And so you need to build all these tools to like understand what it's doing, to improve what it's doing um, those types

of things. And so um, for conversations we need simulations uh, because this isn't a um, you know,

of things. And so um, for conversations we need simulations uh, because this isn't a um, you know, one turn ask a question, here's an answer. It's a back and forth if you know you ask a question that the agent says something and and back and forth. And so if you want to create evaluations, you actually need a simulator of um of the consumer um that's uh going back and forth. And once you

have that, you can then create this dynamic where you have a um uh an LLM that that's representing the consumer talking to a LM representing um the uh uh the customer service department sort of talking back and forth to each other. And um we um create automated systems for um evaluating

um LOMs on that um on that experience going back and forth. And it turns out you really have to do that a lot because every time a new model is updated uh or you change the inputs um you want to understand the performance and so um it becomes really tedious if you don't have an an automated way to um just have this uh have this evaluation um work at scale. Um and then you need

a bunch of observability because if we're going to have um you know huge numbers of uh contacts be handled without a human involved in it. Um you know a bunch of bad things could could happen. A

model could start um being good or bad at at some um thing and we need to understand that that's happening. Um so figuring out the observability tools for this sort of interaction that that no

happening. Um so figuring out the observability tools for this sort of interaction that that no one's watching um becomes also quite uh quite important. Um, so, um, you put all these things together and I think we've been able to, uh, get to the point where you get to a, um, system that

can do a, uh, a pretty good job. Um, so, you know, we need to realize that every, uh, every taco matters here. And so, every consumer, uh, who's ordered a taco expects that. They expect that uh,

matters here. And so, every consumer, uh, who's ordered a taco expects that. They expect that uh, that the the taco will come or that there's um, some need to um, uh, to help them. And um you know we've now taken a few of these um AI models and um been able to take certain issue types and get

to an approach where the AI is doing the entire decisioning end to end. Um so there's there's no um sort of uh deterministic rule. We've just given the AI the principles. Here's how you deal with this issue. Um here's a bunch of context. Please go handle it. Um and from the first uh set

of scenarios where we've done that um we've been able to find that um consumers are happier. Um we

get to more consistent outcomes. I think uh often when people are talking about hallucinations and that kind of stuff for AI models, they assume that humans are perfect. Uh but humans are far from that. Um particularly when you're thinking of a large network of um uh call center workers.

from that. Um particularly when you're thinking of a large network of um uh call center workers.

um you spent a lot of time training them, but that doesn't mean they you know exactly do what um what you expected or um or intended. And so um we've been able to get to outcomes where you get to um better uh support quality as you know uh based on you know consumer interviews you rate this rate the support um better uh support quality in terms of people order more in the future um and

um obviously better quality in terms of response times and and those types of stuff because the um automation is great for that. So um uh that's um that's where we are in um you know it really only in the last couple of months have we we've been able to get that to work um in a kind of AI

end to end um way and um you know just a few um few observations from that that I thought might be interesting. The first is um that it it is very clearly possible. I'm not saying we've achieved

be interesting. The first is um that it it is very clearly possible. I'm not saying we've achieved this through all of Uber. So if you have a um uh bad experience uh with an Uber uh Uber trip um only some of our uh support right now is uh done and and through AI but um it is very clear

that we can achieve material improvements in both quality and cost of support. Um that enables us to create better experiences to actually encourage people to reach out to support more um to help with things like we call them silent sufferers. people who um have a problem but don't bother to um reach out to support because it's viewed as too complicated or or something like that. Um and so

it's it's very clear that that is now um possible and sort of the core technical um barriers have been achieved there. Um second observation is that um uh developing a AI agents feels uh feels different than traditional software engineering. There's a lot of orchestration uh when we're setting up models that are talking to models and models that are jud and another

model that's judging that that model and another model that's providing um context um and another model that's doing the planning. Um there's quite a lot of um orchestration here going on of systems that are pretty um non-deterministic and so um it it has a very different field than traditional software engineering. if I build this this this and this. Um and um I think that's created some

software engineering. if I build this this this and this. Um and um I think that's created some uh differences in in how we um plan and design things and you really have a lot of um exploration in getting to sort of what what good looks like in a very different um kind of process there. Um

and then lastly um you know much much of the engineering for um a company like Uber where we're trying to apply this in the enterprise. We're not in the business of selling an a um an AI model. We're in the business of um uh offering transportation and food and uh those those types

AI model. We're in the business of um uh offering transportation and food and uh those those types of things. Um a lot of the work is really around the model, not in the model itself. Um the models

of things. Um a lot of the work is really around the model, not in the model itself. Um the models today are um very very strong and we're not really constrained by reasoning capability. were

constrained by the ability to create context, the ability to create actions, the ability to create observability and guard rails and all those types of things which is really engineering of this broader um system um around the models itself. Um these models can do great things. They can get um you know a perfect score on the math Olympiad and and all those types of things. That's not as

um useful in a kind of everyday business sense. Um and getting to this um kind of broader system requires all this kind of engineering around it. Um so hopefully I've given you some um insight into uh a real world problem in the um support domain. Um it's uh an area that's super impactful to to us it's a lot of cost and a lot of the experience every day um in in that um if we

can provide that great experience that from a bot that is um knowledgeable, understanding, helpful, I think people are fundamentally very happy to work within an automation. um what they don't like about working in an automation is when you get stuck and AI creates the opportunity to just dramatically expand the capability of um what a automation can do and I think uh will really drive

a lot of impact uh going forward. So appreciate the time and happy to take some questions.

All right. Well, I'm I'm sure there are are plenty, but I I guess one thing that stands out to me um you know, when you use the phrase AI engineer, right? The person who's thinking about this way of approaching this problem, who's thinking about how to optimize the problem, improve the responsiveness. Y give us a snapshot of that person, what they know,

what they've learned, how they've gotten there. So I think there is a uh what we're looking for in this these types of folks is a lot of creativity and curiosity. There's not a book that tells you how to do this. This is very much um u you know kind of new frontier

uh work. And so I think it is a lot of creativity and experimentation. Um it's also a lot of um

uh work. And so I think it is a lot of creativity and experimentation. Um it's also a lot of um really um uh paying attention to the um the new capabilities and sort of thinking differently. A

lot of what we're focused on now from a recruiting perspective is trying to figure out um are people are engineers um kind of AI native. do they think about um problems with an AI first um solution of thinking about hey I've got this system that I can use AI not just for code completion but I

can design sort of attack the problem in a totally different um way can I write a an agent that can solve this um uh problem more generally uh for me and um and I think that's one of the real powers uh superpowers of people who are um starting fresh in this uh world where you're kind of learning um

computer science or engineering with these tools is um I think you're much more likely to sort of think about solving problems uh with these tools in a way that um you know some not all of our uh more tenure engineers uh would naturally gravitate to. And I just wonder if you have some thoughts or

ruminations on the kind of like the AI handringers perspective where oh this is going to you know exhaust our energy resources or oh I'll end up getting some hallucination that sends me down some you know destructive path. I mean have you kind of encountered that like standard litany like

item by item and decided these aren't problems for us? I mean I'm particularly interested. It seems

quite clear you view this as a as a cost performer relative to the cost of the kind of call center or sure you know and that that would be good just to sort of kind of demystify that for sure uh so you know Uber's use of AI um we're we're not in the business of training foundation models

um uh so you know the customer service use case um from a compute complex lexity perspective is not um not huge. um we're not building huge data centers and um you know worried about the um uh the you know energy is constraint or something like that for a lot of the things um

we're building um there but the um I I think there is a lot of uh the capabilities here continue to advance um I think the if the there's it's such early stages that there are so many opportunities for increas inreasing um compute efficiency and performance and that kind of stuff. There's just

so many optimizations. um you you've had this kind of scaling path where you get to bigger and bigger models and um and a lot of that has come from well let's just throw more hardware at this um when I there's a lot of opportunities to just uh be thoughtful about um how you structure the um the

computation itself um to be more efficient and I think you see that in um yeah I I don't have teams spending a ton of time saying how much model time are you um using for a use case like this because we see the the cost uh you know per token of uh the um major AI models falling very very quickly

and I think that's likely to to continue and so these discussions about you know how big of a data center do you do you build and and those types of things um it's there's so much emphasis on the scaling law dimension of it and very little on the how much uh uh how much are you going to get from a computational efficiency perspective Um, and you know, we'll see how that plays out. There's

clearly huge bets on the um, infrastructure side of things. Um, but I do wonder if people aren't spending enough time on uh, thinking about what the what what the efficiency side of things. Yeah.

Um, so I heard an interesting comment about that sort of the AI boom as being fueled by FOMO. Uh,

so like everybody kind of wants to get in on it and like that somehow that's kind of fueling the the rush mentality. I wonder if um and then I will turn it over to you guys. You know,

as a kind of a hopeful comment to the Yale undergraduate and I think there's probably a lot in graduate student who are here. What did you you know, what kernels from your experience here as a Yale undergraduate do you think has have come in kind of particularly handy or you know why

should they feel optimistic about being being here and in being in this community? Uh, so I I found my Yale education computer science super helpful. I like use it um use it every day. I'm not in a frontline software engineering role. I manage um engineers and and other uh disciplines. But being

able to understand the foundations of how things work is is really valuable. Um, I remember taking an AI class from Professor Scazalotti and it was actually right on the other side of that wall and um I remember the classroom and um uh you know understanding um then we were talking about neural networks and that kind of stuff. It just helps to understand that um foundation

to understand the the technologies uh today and um you know even uh even for me to be able to engage with with our engineering teams to understand you know computational complexity of an algorithm or um uh distributed systems or that kind of stuff is is important to um really be able to relate to the

the people building the technology every day. And so um I think it's really valuable to have those foundations. I think it's an interesting question as um AI can do more of the the basics um you know

foundations. I think it's an interesting question as um AI can do more of the the basics um you know what are the most important skills going forward. But I do have a pretty strong sense that um sort of understanding those foundational things are still um still pretty important. um you know back uh you know back when I was in in school there was still it was still you were still taught how to

um how to processors and machines um do the the actual computation and it weren't sitting and writing assembly every day um but it was still useful to um to understand that to do the higher level things and we're definitely in this um transition where you're going to sort of higher levels of abstraction but you still need to have some understanding of what's going

on underneath that um both to understand how it works but also to to have ideas of how you might um bring it together and I think there's uh still a lot of lot of value in that. That's

great. Good. Okay. I wonder if there are any questions for the audience. Your hand went up first. I was wondering a bit about the models that these are all running on. Um you talked a

first. I was wondering a bit about the models that these are all running on. Um you talked a bit about how models are scaling very fast and how cost per token is going down. Um and it seems as though and correct me if I'm wrong that like a lot of this is being done on foundation models using like rag and just very very large models. Um is there any work going on in terms of kind of

taking an approach of fine-tuning these models and using smaller weight models um and trying to have like a system of agents rather than a single model with a lot of context poured into it. Um, and also related to that, what is your relationship with the model vendors? Like are you just using I mean like for testing development and like doing doing projects and stuff, you'll just use like a chp API

um or cloud API, but do you have specific relations with these um model vendors to get like the actual weights and to be able to train and fine-tune the models or is it more so just like a third-party um relationship like a lot of individual developers have? Sure. Um so

on the first part um there is a lot of work in um it's not um uh just use one one model that drives um everything. Um I think we tend to use the um higher power uh higher capability models on the

um everything. Um I think we tend to use the um higher power uh higher capability models on the like core reasoning questions. Um but a lot of the things like um evaluation or intent detection or that kind of stuff you can do with much uh smaller models. Um an example of something that um as had team spent a lot of time on is an important problem for us is figuring out if

someone's writing into support whether it's automated or not um how do we find um safety incidents um it's really important that we respond to safety incidents really really fast and so for this narrow question of simply classifying is this inquiry about a safety incident or not you don't need a huge model to do that you need something small that can answer that question really

quickly and then that then I can apply to every conversation that's um that's happening at Uber.

And so we'll use a much smaller um model for that. And for the you know reasoning of deciding um who I give a refund to or not in some complicated scenario um we'll use a much more um powerful and capable model for that. Um I think in terms of the interaction with uh the uh the frontier labs

that sort of provide the core um core foundation models um we have a very um direct interaction in that in that they have um they'll have different names for them but forward deployed engineers or um customer success engineers those types of uh folks who um will work with uh companies to help

solve specific um applications and say you know listen to a problem and say hey um and and like talk to their own research teams and say, "Hey, how does this version of this model um how might it be able to um to do this um better?" Um on the fine-tuning and kind of direct um model

weight things, um there hasn't been as strong of a return to fine-tuning models as opposed, you know, if you put a lot of effort into fine-tuning, um by the time you're done, the like core model has moved on to something else. um and then we have to go run the all the evals again and and stuff like

that. So we're not seeing a great um return from that kind of work um and really more trying to

that. So we're not seeing a great um return from that kind of work um and really more trying to do this build the system around the model assume that the core models will get better um and then build the tools to evaluate the next version when when they come out and and I think you see that on the model developer side as well where they're just really focused on getting the the core models

um to improve across you know all the dimensions of um eval that they're trying to do as opposed to sort of overfitting for um you know medicine or transportation or finance or something like that.

Thank you miss so much for coming to speak with us. Um I really appreciated your point about much of the engineering being in the systems surrounding the models versus being in the model.

Uh and for context, I'm building a cyber security platform with the ELC CS PhD for autonomous vehicles. And I wanted to know um what are the two most important problems you're focused on solving

vehicles. And I wanted to know um what are the two most important problems you're focused on solving with deploying AI agents uh in terms of securing deployment and preserving latency and at any level of abstraction. Okay. Um so a couple of problems um I think latency is a u big problem for a lot

of abstraction. Okay. Um so a couple of problems um I think latency is a u big problem for a lot of uh use cases. There are a lot of things where you can make a demo that works really um really well. Um but if um if a customer can't uh if 2% of the time it takes 10 seconds instead of 1 second

well. Um but if um if a customer can't uh if 2% of the time it takes 10 seconds instead of 1 second to respond and you're sitting waiting you know trying to order food and we're waiting for the model to return an answer. Um that's a problem. Um I think you know you can solve that through

dedicated hardware which is expensive. Um I think that will improve over time as um the um the model serving platforms um get more robust. I think it's really if you think about the degree to which um a lot of these um uh infrastructure providers have have scaled it's like really amazing um and

so you probably should give them a little bit of grace on the um uh latency with the sort of level of volume they've created but it is very important from a business perspective um to and something that's a key criteria for us putting anything in the core purchase flow or I talked earlier about um doing uh voice support um if it's you know you say hey I want to do this and it's just thinking

thinking thinking um that's not a um not a great result so um latency is um uh definitely something um top of mind and then I think um eval is another um area where um really just uh being able to set up the right evaluation for each of the different types of um use cases where possible we we would

love to do um automated evaluations There are um areas where um we do um human evals as well and that's actually something we've had to build that capability internally and we've turned to um our sort of expertise in gig labor um to have gig workers do um evaluations for uh

different types of uh types of models and that's something we built for our own internal use cases and are now starting to externalize with uh third parties um third parties as well but it really you know depends on the specific use case um what what what exactly the eval mechanics would be.

Hi, thank you again for um coming. Uh so I'm a Yale senior and me and my friend we've been building a lot of different um AI tools and kind of use or using AI tools to build more complex systems. And so I have a quick question about how um Uber has been organizing their engineering efforts because I feel like now with the AI coding tools like you're able to be both the

product manager and the software engineer and kind of go really deep into developing that. And so I'm curious how with the different products Uber is building is reliant on the overarching system and infrastructure of the whole company. How have like in management have you been dealing with um like your engineers? Are they more like product focused? you kind of have a team that's deciding

what everyone's is going to try to like push the AI towards exploring and just kind of like how that dynamic has been. So I I think one of the thing so this this kind of engineer can be product manager um dynamic I think we see come in the form of um more prototyping uh so someone has

an idea and they can just create a a prototype of it um using these um coding tools or AI um uh design tools. So, um, uh, Figma and other tools have, you know, much more accessible tools to be able to draw UI, um, that it used to be, you know, you'd have to go talk to the PM and

talk to the designer to build the, um, the UI mockup and now you can just go, um, chat with something and and get it to produce that. And so, I think that changes the dialogue about what a new feature might look like by someone just saying, "Here, here's an example." um when you look at um Uber scale and our expectations about reliability and all the other things we um we need to deal

with um you end up then doing it for a bit of rebuilding to say okay I've created this prototype um it's very rare that we can then you know sort of convert that directly into something we can build but it does um I think accelerate the the process uh quite a bit by being able to really sort of um make it real and gets get over some of the sort of technical capability questions

um very early on um we've had some people do some really cool demos of things that we sort of weren't sure was possible. They demonstrate that it's possible and we say, "Okay, gee, this is worthwhile." Uh, and then we'll put a sort of more traditional engineering approach to u building the

worthwhile." Uh, and then we'll put a sort of more traditional engineering approach to u building the scalable version of it that that meets all our um, you know, engineering objectives.

Thanks so much for the talk, Gus. Um, I thought it was really educational and I think like this like AI customer support thing is something that eventually can be end to end like kind of displacing the whole current system of doing things and I think with um like robo taxis uh

and I guess uh these cursor-l like third party uh AI ids uh like I guess development has been more competitive uh is first of all is uh this like competitive employment and like a concern at Uber where um like these robo taxes might be displacing uh human drivers and like uh comp like uh hiring

is a lot more competitive with the engineers and maybe like what roles do these AI tools like open up um uh for like people who are very interested in AI. Okay. By competitive employment, you mean competition for talent or something else? And also, I guess like robo taxis making like

prices go down for the drivers, right? Okay. Um and I'm not sure what the market share is actually like for robo taxis, maybe it's like very small right now, but if it does scale up in the future.

Okay. So, um look, on the um engineering talent side of things, um there there's always going to be a competition for um for top talent. Um and um While AI coding tools make a bunch of things much more efficient, um we're still uh very much constrained by um the ability to build things as

opposed to the number of ideas of um of things to build. Um at least that's um kind of where we are on the um on the path um right now. Um you can build faster, you can build um with uh you know what we just talked about prototyping uh build with more intentionality. Uh but I think there's still um plenty of demand for um for software engineering talent. Um I think it's really more

a mix of that talent and um again looking for people who are uh kind of AI native and really use these tools effectively. Um robo taxis are a u uh clearly um something in the in the future that will be very impactful. Um we have uh as I said partnerships with a bunch of different

um autonomy providers and we operate um robo taxis um on the ground. So if you go to um Atlanta or um Austin um and request an Uber, we can give you a uh a Whimo um AV instead. Um today that's tiny um in the the grand scheme of uh all Uber trips overall. Uh but if you think about what its

um impact is going to have on the on the future, um I'm really personally focused on the impact on safety. Um the safety performance of these uh vehicles when you design them with the intention

on safety. Um the safety performance of these uh vehicles when you design them with the intention of saying um not just better than a human but the safest vehicle possible. Um the capabilities of these vehicles are really um really compelling. And so if you think um you know on 10 20 year

time scale when a large fraction of uh vehicles on the road are autonomous um what that's going to do at a societal level to the number of road fatalities and um those types of uh things that are really big societal costs. Um I I think there will be a lot of um impact there uh really for the

better. Hey, one more. Sorry, I got I'm late for a meeting, so I'm excited to hear the last book. Um,

better. Hey, one more. Sorry, I got I'm late for a meeting, so I'm excited to hear the last book. Um,

you use the word or the term AI native of tons. Do you mind just expanding upon the definition of that and then how kind of that maybe label would expand across the Uber workforce beyond just software engineer? Sure. Um so I I don't think I have a uh a formal definition for you but let

software engineer? Sure. Um so I I don't think I have a uh a formal definition for you but let me give you a sense. I think it's really um kind of a um an attitude and a world view. Um I think uh when you look at a problem and it's not just software engineering um I could describe AI native

salespeople um that look at their sales activity of um customer prospecting. How do I find a list of customers to talk to and how do I figure out what I should talk to this customer or prospect when I'm um on the phone? Um there are um you know kind of the traditional way of of doing

things and then there's the AI native way which is you know I'm going to have um uh wire up u agents and bots and all these um uh different things. Um, and so I I can think of a a customer service uh person very very deep in my um my org who they're they're respon they're responsible for

creating the tests uh for um agents to say do you understand the policy for tra traditional um call center agents um uh you need to teach them a policy and then do a test and um someone wired up um basically a way to have um ChachiPT and some other tools

um create these tests for every policy. And so it basically replaced these people who were their job was to create the tests and they're like well now instead of having one test that know people might share answers with it just says here's a policy go create a bunch of questions it automatically created the like question bank and sorted them and all these things. And this was someone who

was not a um an engineer um by trade. They were not an engineering function. They were basically a a line manager of customer support agents in an emerging market and they um just hacked together a whole bunch of um uh different tools and built this system and and so that's really a um when I'm talking about AI native they're just like hey here's this problem in front of me and I

can wire up all these tools to solve this problem in a very different way. um and that you know the person sitting next to them would have said hey here's the traditional way of how we've done this all the all the time and this person was like hey I'm just I've got these new tools let me use them

in a new way and experiment and try out and so I think we're um trying to figure out ways to both encourage our existing staff to um think in that latter way um and as we look at um bringing in talent um how do we test and evaluate for that um I I would say that traditional interviewing techniques are not great at that and so that's something we're um wrestling with to figure out

the best way to uh um to do as we bring in new talent. All right, a great question. Maybe uh

maybe with that uh like to thank Gus for coming. I think you're really exciting viewpoint into what's going on at Uber and in society more generally, it seems. So, uh let's thank Gus again. Thank you.