Mustafa Suleyman: The AGI Race Is Fake, Building Safe Superintelligence & the Agentic Economy | #216

What's the mandate from SATA? Is it win AGI?

>> I don't think there's really a winning of AGI. I'm not sure there's a race.

of AGI. I'm not sure there's a race.

>> One of the OGs of the AI world, Mustafa Saliman is the CEO now of Microsoft AI.

He spent more than a decade at the forefront of this industry before uh we even had gotten to feel it in the past couple of years. Now,

>> fundamentally, the transition that we're making is from a world of operating systems, search engines, apps, and browsers to a world of agents and

companions. We're all going as fast as

companions. We're all going as fast as we possibly can, but a race implies it's zero sum. It implies that there's a

zero sum. It implies that there's a finish line, and it is like not quite the right metaphor. As we know, technologies and science and knowledge proliferate everywhere, all at once, at

all scales. basically simultaneously.

all scales. basically simultaneously.

>> Are you spending a lot of your energy compute uh human power on safety?

>> Yeah. No, I I mean, >> now that's a moonshot, ladies and gentlemen.

>> Everybody, welcome to Moonshots. I'm

here with DB2 and AWG and Mustafa Soliman, uh the co-founder of Deep Mind, Inflection AI, and now the CEO of Microsoft AI.

Uh welcome, my friend. It's good to have you here. Thank you for making time for

you here. Thank you for making time for us.

>> Thanks for having me. Yeah, I'm excited to do this.

>> Yeah, it's um you know what you've been building with Satia is amazing. Uh and

it's hard to believe that Microsoft is 50 years old and it's reinvented itself so many times and for the last 5 years it's been you know at the top of the

game the most valuable company in the world 250,000 employees and from what I understand 10,000 employees now under

you. Uh so a few you know important

you. Uh so a few you know important questions I want to open with. Uh first

some broad context. Uh you're building inside a massive company with huge resources probably arguably more than

almost everybody else. And the question I have is what what's the end goal here?

You've got all the hyperscalers sort of providing open access to AI and they're doing sort of a land grab uh to try and get as many users as possible. Uh you've

been building sort of in a you know within the Microsoft 365 ecosystem. Uh

is the goal in the you know next couple years maximum users? Is it data centers?

Is it uh you know is it cloud? How do

you think of what you're optimizing for?

>> I mean, it's a good question. So, I

mean, we're are on any given day a $4 trillion company with almost $300 billion of revenue. Um, it's incredible.

It's just surreal and very, very, very humbling. Um,

humbling. Um, >> and we play at every layer of the stack.

I mean, obviously, we have an enormous business in data centers and in some ways we're like a modern construction company. hundreds of thousands of

company. hundreds of thousands of construction workers building gigawatts a year of uh you know CPU and AI accelerators of all kinds and enabling

that you know to be available to the market. um APIs on top of that, but also

market. um APIs on top of that, but also firstparty products in every domain you can think of from gaming and LinkedIn right the way through to all the

fundamentals of M365 and Windows um and of course in our search and consumer businesses and too and fundamentally the transition that we're

making is from a world of operating systems, search engines, apps and browsers to a world of agents and companions.

Um, all of these user interfaces are going to get subsumed into a conversational agentic form. Um, and

these models are going to feel like having a a real assistant in your pocket 24/7 that can do anything that has all your context. And you're going to do

your context. And you're going to do less and less of the direct computing just as we're seeing now. Many software

engineers are using assistive code coding agents to to um both debug their code and also generate large amounts of code just as we used libraries, third party libraries. Now we're just going to

party libraries. Now we're just going to use AIs to do do that generation and it's making them more efficient and more accurate and faster and so on and so

forth. So the the trajectory we're on is

forth. So the the trajectory we're on is quite predictable. It's one from user

quite predictable. It's one from user interfaces to AI agents and that is a paradigm shift which the company is completely focused on like you know

after seeing five decades worth of transitions I think the company is like super alert to making sure that we're best placed to manage this one. Do you

see yourself providing sort of an open- source AI like the other players out there or do you think you can keep it contained within within Microsoft 365?

>> I think we're pretty open-minded. I

mean, we've got some pretty small open-source models. Um, I think

open-source models. Um, I think realistically, >> when I say open source, I really mean open access, if you would.

>> Yeah. I mean, look, there are always going to be APIs that provide incredibly powerful models. I mean, you know,

powerful models. I mean, you know, Microsoft is really a platform of platforms. being a platform and being a great >> provider of the core infrastructure that enables other people to be productive

>> is is like the DNA of the company. Um,

and so we will always have masses of APIs that turbocharge that. But what an API is is going to start to look kind of different too. Like it it may be pretty

different too. Like it it may be pretty blurred the distinction between the API and the agent itself. maybe that we're principally in the business in 5 years time of selling agents that perform

certain tasks that come with a certification of reliability, security, safety, and trust. And that is actually in many ways the strength of Microsoft and that's one of the things that's

attracted to me is like this is a company that's incredibly trusted. it's

actually very secure >> and sometimes I think the the slowness or the friction is actually a bit of an asset. You know, there's a kind of

asset. You know, there's a kind of steadiness that comes with having provided for all of the world's biggest uh Fortune 500 companies and governments

and major institutions.

>> Is it like the old adage, you can't go wrong buying IBM in the old days? I

think you just there's a there's a steadiness about us which I think is reassuring to people and there's a kind of like deliberate customerfocused patience.

>> Um you know there's not the same anxiety and you know sort of somewhat sclerotic nature that comes with being you know an insurgent. Um there's some downsides to

insurgent. Um there's some downsides to our position. You know we take a little

our position. You know we take a little longer to get things through but the company is firing on all cylinders. It's

very impressive to see. One more

question before I turn it over to Alex.

Uh, you know, we're seeing in these in this hyperscaler war, I mean, literally, uh, you know, a week by week, everybody outdoing each other in this, uh, in this

insane period of, uh, everybody coming out with with the new benchmarks. Uh,

you know, do you miss not being in that game or is the stability that Microsoft provides to build for a long-term vision sort of, uh, what you find most

exciting? You know, uh, my background at

exciting? You know, uh, my background at Deep Mind is such that I spent a good decade grinding through the flat part of the exponential where basically nothing

worked. I mean, you know, really like

worked. I mean, you know, really like there was some amazing papers. Uh, Alph

Go was obviously incredible, but it was in a very unique simulated controlled game-like environment, but things actually working in the real world were

few and far between. Um, and so, you know, I've always taken a multi-deade view, >> and that's just been my instinct. And I

think that, um, you know, yes, it's super important to ship new models every month and be out there in the market, but it's actually more important to lay the right foundation for what's coming

cuz I think it's going to be the the most wild transition we have ever made as a species. It's

>> Can you just flesh that out a little bit? Was there a period of time where it

bit? Was there a period of time where it was just three of you grinding it out in London? Well, there were more than three

London? Well, there were more than three of us, but I mean for the decade between 2010 and 2012, sorry, 2020.

>> Um, I mean, there were just like so few successful commercial applications of uh of of deep learning. I mean, there were plenty behind the scenes. There was

image recognition, improvements to search, but commercial >> huge market for >> commercial. Yeah. Playing go not a huge

>> commercial. Yeah. Playing go not a huge rock. Exactly. So, I think whereas now I

rock. Exactly. So, I think whereas now I mean you then you see LLMs from 2022 onwards like in production.

people changing what it means to be a human relations like that, you know, that's we hit an inflection point.

>> And you know, I think that um is very very different to the grind of of like training tiny models with very little data and very small clusters back in the 2010s.

>> Every week, my team and I study the top 10 technology meta trends that will transform industries over the decade ahead. I cover trends ranging from

ahead. I cover trends ranging from humanoid robotics, AGI, and quantum computing to transport, energy, longevity, and more. There's no fluff, only the most important stuff that matters, that impacts our lives, our

companies, and our careers. If you want me to share these meta trends with you, I write a newsletter twice a week, sending it out as a short two-minute read via email. And if you want to discover the most important meta trends

10 years before anyone else, this report's for you. Readers include

founders and CEOs from the world's most disruptive companies and entrepreneurs building the world's most disruptive tech. It's not for you if you don't want

tech. It's not for you if you don't want to be informed about what's coming, why it matters, and how you can benefit from it. To subscribe for free, go to

it. To subscribe for free, go to dmandis.com/metats

dmandis.com/metats to gain access to the trends 10 years before anyone else. All right, now back to this episode.

>> Yeah. So when last we spoke circa 2015 I think that was perhaps 3 years post imageet 5 years pre language models our

fshot learners agents agentic AI was nowhere to be seen at the level of what we see now since you've written you about your vision um what you've I think

socialized as a modern touring test the idea of economic benchmarks for autonomy by agents I I'd love to here. Where are

Microsoft's economic benchmarks for these agents? If the the agents are

these agents? If the the agents are about to take over the economy or take over so many economically useful functions, why are we stuck with benchmarks like vending bench rather

than Microsoft leading the way with Microsoft's economically autonomous benchmarks for its agents?

>> Yeah, I mean, it's probably just worth adding the context that we met in 2015 in Puerto Rico at the AI safety conference. True that many many of the

conference. True that many many of the field now were at at the same time.

seminal moment.

>> Yeah. Was it the day after New Year's Eve or somewhere around New Year?

>> It was pretty cold out everywhere except Puerto Rico.

>> Yeah, exactly. It was pretty cool. Uh it

was quite surreal moment actually. Um

>> it's like a syllar right before it all happened.

>> Yeah. Yeah, totally. Um and you know, yeah, that the modern cheuring test was something I proposed um I guess it was 2022 when I wrote it. Um and it was

basically making a pretty simple prediction. Um, if the scaling laws

prediction. Um, if the scaling laws continue with more data and compute and adding an order of magnitude more compute to the best models in the world every year, then it's pretty clear we

would go from recognition, which was the first part of the wave, to generation, uh, which is clearly we're now in the middle of, or maybe ending that chapter, to then having perfect generation at

every time step, which in sequence is going to produce assistive agentive actions. and actions would obviously

actions. and actions would obviously look like an intelligent knowledge worker or a project manager or a strategist or a startup founder or whatever it is. And so then how would we measure that performance rather than

measuring it with academic and theoretical benchmarks? One would

theoretical benchmarks? One would clearly want to measure it through capabilities. What can the thing do in

capabilities. What can the thing do in the in the economy in the workplace? And

how do we measure the economy? We

measure it by dollars and cents. And so

could you know what would be the first model to make a million dollars? Now,

given as I recall, $100,000 in starting capital.

>> That's right. Yeah. How who could which which model could turn it into a million dollars?

>> 10x return on investment by an agent.

>> Exactly. Um and so I think that's a pretty good measure of performance and capability. And certainly, you know,

capability. And certainly, you know, we've kind of just breezed past the cheuring test, right? I mean, it kind of has been passed. No one's really done a big, you know, alpha mentioner

silver prize wound down before we breezed past touring.

>> Yeah. And no one celebrated it. Like

where was the big like, you know, Casparov deep blue moment?

>> Can we clink virtual glasses right now and celebrate that we won? It happened.

>> Yeah. Exactly. And that's the what it feels like to kind of make progress in a world full of these compounding exponentials where we just get desensitized to 10x. So much so that you can be like, "Guys, why haven't you done

it yet?"

it yet?" >> Yeah.

>> We're spoiled. Where's my Microsoft Loger prize for the modern touring tester, >> right? Exactly. Um Yeah. You know, like

>> right? Exactly. Um Yeah. You know, like someone said to me earlier on, "But you know, these this AI thing, it's still in its infancy, isn't it?" And I'm like, man, if this is infancy, wow. like I can

talk to my computer fluently. Star Trek is here in real

fluently. Star Trek is here in real time. Yeah, exactly.

time. Yeah, exactly.

>> Um, so, you know, obviously at the same time, agents don't really work yet. The

action stuff is still progressing. It's

getting better and better every minute, but it's pretty clear that in the next couple of years, those things come into view and they're going to be very, very good.

>> Can we get together again after the modern touring test has been passed and and just to celebrate, recognize it?

>> Virtual glasses again. Absolutely.

>> Hopefully we can pop a, you know, champagne or something.

>> I think we should have an optimist pop the cork for us or something.

>> Exactly. Exactly.

>> Dave, >> hey, I want to flush out that backstory a little bit more, too. It's such a such a cool story. But, um, I remember really clearly, you know, after DeepMind got acquired by Google, what was the price tag on that deal? It was like half a

billion dollars, something like that.

>> 650.

650. What What year was that?

>> 2014.

>> 2014. I remember reading maybe a year or two later that Google justifies deal by having DeepMind uh tune the air conditioning in the data centers.

>> Yeah. Right.

>> My interpretation of that was like, "Wow, this isn't going all that well."

And and now it's obviously the biggest thing that's happened in the history of humanity and and and forking out all over the place. But

>> I mean, we did the data center thing was pretty cool. We did actually reduce the

pretty cool. We did actually reduce the cost of cooling the Google data center fleet by >> Yeah. It's so funny cuz I read it at the

>> Yeah. It's so funny cuz I read it at the time and I was like, "What a bust." And

then I read about it in Wikipedia on the flight over here to to meet with you and it's like it was actually what 500 attributes fitting into the neural net and it was it was actually a lot more complicated than the news made it sound.

>> That's it >> at the time.

>> That's right.

>> But like you were talking about the flat part of the exponential and you think about like okay all of this R&D which is so close to becoming AGI >> is tuning the air conditioning. But

that's the nature of exponentials. They

sneak up on you like this. But the other way to think about that is that it's basically taking an arbitrary data input, an arbitrary modality, and using the same general purpose method to produce very accurate predictions in a

novel environment, which is the same thing that's happened with text and audio and image and now coding and obviously with other time series data.

And so it's just another proof point of the, you know, the general purpose nature of the models. And I think like it's so easy to get caught up thinking five years is a long time. Mhm.

>> It's like a blink of an eye. It's a drop in the ocean. And I think because we're such a frantic second to second news culture, social media type environment, we just don't have an intuition for these time scales. I think other

cultures, you know, do and I think historically before digitalization, we had much more of a natural intuition for the movement of the landscape and the

seasons and like, you know, the ages and stuff. And now we're just like, well,

stuff. And now we're just like, well, it's not coming quick enough. It's like,

dude, it's coming. We've shifted to a 247 uh operations. I mean, I know very few I

uh operations. I mean, I know very few I know a lot of people including this group that are operating around the clock every day just because when we when we do uh you know, a Moonshot

podcast week to week just to celebrate and talk about what's just happened, it's insane on a week-by-eek basis what's going on.

>> Yeah. Yeah.

>> You know, and Peter's always saying people are very very bad at exponentials, right? 100,000 years of

exponentials, right? 100,000 years of evolution has us predicting tomorrow will be like yesterday.

>> But you're one of the few people who, you know, having lived through that >> air conditioning becomes AGI in just a few years.

>> Uh so where we sit right now is on another inflection point and the implications are massive and people are way underreacting across the board and so you're one of the few people who you

know having seen it before can say I just got very lucky. I mean we were very lucky to have an intuition for the exponential right and like that's that that's a very powerful thing because we

can all theoretically observe the shape of the exponential but to go through the flat part and then get excited by a micro doubling you know like that that's the bit is that when you're like oh my

god >> this like I remember this um the emnest image generation thing the first generative models >> there's like these are like I can't

remember maybe 256 by 256 pixels.

>> Um, you know, black and white, uh, handwritten digits. Y

handwritten digits. Y >> and, you know, I think this was like 2013, maybe even 2012, and this guy, like I think maybe he was employee number five at Deep Mind, Dan Vista,

this like um, awesome Dutch guy out of EPFL, um, was generated like the first number seven that was provably not in the

training set for the first time. I was

like, man, that is amazing. Like, how

could it have it's learned something about the idea of seven? That was the, you know, that was it's got a concept of seven. How cool is that?

seven. How cool is that?

>> You know, so I got the highest score on Mnest ever in 1991 when it first came out when you were three years old, right?

>> Yeah. Nine. Nine. You're nine years old.

Okay. Um, yeah. And and actually that's the same data set that's now in PyTorch that people like bench benchmark off.

>> Pretty crazy. Incredible.

>> Yeah. How how often are you surprised by what you're seeing? I mean, how often is there like a move 37 uh you know, sort of like aha moment?

>> Is it happening more more frequently?

>> I was absolutely blown away by the first versions of Lambda at Google. Um, this

was like a maybe 12 people working on it led by Nome Shazir and Daniel Defritus and Quarkley and I got involved later maybe three or four or five months after

they'd been going >> and uh it was just breathtaking. I mean

it it obviously everyone at that point had been playing with LLMs and they were like one shot that produce an answer and you know have a prompt and blah blah blah >> but they were really the first to push

it for conversation >> and dialogue and it just seeing the kind of emergent behaviors that arise in yourself like things that you didn't even think to ask because you know there

going to be a dialogue rather than a question answer situation sounds so trivial to say that like in hindsight cuz now we're obviously steeped in conversation as the default mode. But

that was like breathtaking for me. And

obviously then I pushed really hard to try and ship that at Google and for various reasons we couldn't we couldn't get it launched. And that was when we all left like I left and Gnome left to do character and you know David Luan

left to do Adept and you know we were all like okay this is the moment and so you know I think there's been still a couple moments since then but that that was probably the biggest one that I remember in recent memory is

mind-blowing. and and the scaling laws

mind-blowing. and and the scaling laws have delivered such unexpected performance, right? I mean, was going

performance, right? I mean, was going back to your earlier days, did did you anticipate the kinds of capabilities that have resulted? I mean, was this

predictable for you or is it still like, wow, what it's able to do in medicine, in conversation, in scientific research?

>> Well, especially working off of pure text. I mean, how far we've gotten.

text. I mean, how far we've gotten.

Nobody I I think well, you tell me, but nobody would have seen how far we would get with just text.

>> Yeah. I mean, we in 2015, I collaborated with a bunch of really awesome people on a NLP deep learning paper at Deep Mind

um where we were essentially trying to predict a single word in a sentence. We

I think we had scraped like Daily Mail news articles and CNN articles and we were like can we fill in the blank just predict like one word in a sentence or complete the final word in a sentence like the inverse of the problem that we

the way the models now work >> and you know it was like a pretty big contribution. And it was a good

contribution. And it was a good well-sighted paper, but it was like this is never going to scale. Like we were just like, okay, we're way too early.

Not enough data, not enough compute. But

the we were still optimistic >> that with more data and compute, >> that is a method that will work. So I

don't want to have like hindsight bias and say, well, it was all very predictable, but everyone in the field, not just obviously me, but everyone in the field just had the same hammer and nail and just kept chipping away. Like,

can we add more data to this? Can we

clarify our prediction target and can we add more compute?

>> And broadly speaking, that's what's what's >> delivered. Yeah.

>> delivered. Yeah.

>> Yeah. We'd love to maybe pull on that theme a bit. So you mentioned how surprising your generative 7 from emnest was. You mentioned how surprising the

was. You mentioned how surprising the success of Lambda for conversational tuning and conversational performance in general is. I think you've made already

general is. I think you've made already a little bit of news uh to my knowledge in in this episode if I understood correctly, correct me if I'm wrong, by but with the expectation that in the

next 2 years, so I I read that as 2027, we'll see agents start to pass your your modern touring test. We'll see them be able to 10x 100,000 US return on

investment. I I'm curious about the next

investment. I I'm curious about the next surprises to to come. AI for science.

Microsoft research has an AI for science initiative. Do you have timelines in

initiative. Do you have timelines in your mind for AI solving math which we're seeing whole bunch of startups right now tear through Erdish problems AI for physics chemistry medicine >> material science

>> material science what what do you think happens and when >> yeah actually you've just reminded me the the more recent thing that has blown my mind is the fact that um these

methods could learn from one domain coding puzzles maths the essence of like logical reasoning So just as it learned the essence or the conceptual

representation of a number seven um it's clearly learned the abstract nature of like a logical reasoning path and then

can basically apply that you know um to many many other domains. And so that that's kind of interesting because it can apply that as well as the underlying

hallucination/creativity sort of instinct that it has which is more like interpolation. Mhm.

>> Um but those two things combined are like a lethal combination >> for making progress in like say new um mathematical theorem solving or new scientific challenges because that's

basically what humans do all the time.

We sort of combine these two you know capabilities and so I I couldn't really put I mean some people want to put dates on those things. It's hard to put a date on those things because they really are very very fundamental but it feels like

they're definitely within reach. It's

hard to kind of it would be very odd to bet against them.

>> Just maybe from an overunder perspective, do you think say given all of the recent progress in math for example? Do do you think solving science

example? Do do you think solving science and engineering for some reasonable definition of solving is going to ultimately be harder or easier than

modern touring test 10xing of return on investment? It's going to be harder

investment? It's going to be harder because I think a lot of the training data if you like for strings of activity in the workplace or in

entrepreneurialism, startups and so on that kind of exists in a lot of the log data and also it lends itself naturally to real-time calibration with a human.

So the AI can sort of check in, the human can oversee, the human can intervene, the human can steer and calibrate. And so it's going to be a

calibrate. And so it's going to be a much more um sort of dual like combined effort between AI >> reinforcement learning in that category.

>> Yeah. Where a human is participating in steering the reinforcement learning trajectory whereas >> business right >> in a in a novel domain where it really is inventing completely new knowledge.

Um that's kind of more happening in a very abstract sort of vector space and it's like unclear yet how you know the the the human is going to intervene in the theorem solving problem. Obviously,

everyone's working on this particularly in like biology and synthetic materials and stuff like that cuz you you you want to I mean it's already giving humans a better intuition for where in the search space to look for for new hypotheses for

drugs for example or for materials and then the human can either take or reject that feed that back to the model then obviously go and test it in silicon and be like oh like we actually ran the experiment you know we perpeted a bunch

of stuff and then feed that back into the model to improve the search >> and and maybe it's a follow-up question what can humanity in general Microsoft specifically or the AI community subset

of which listens to the podcast. What

can they do to accelerate AI for science and accelerate the solution to science, math, engineering with AI?

>> I mean, arguably that would be like one of the most impactful things.

>> Yeah.

>> For humanity that would just fundamentally move everything at light speed.

>> Yeah. I mean, I think it's already happening very organically, right? This

is also not only is this like the most powerful technology in the world, it's also the fastest proliferating in human history. Mhm.

history. Mhm.

>> Um, and you know, sort of the the cost of access, the cost of inference coming down by multiple orders of magnitude every couple of years is kind >> Would you ever have imagined it would be so cheap?

>> That bit I also totally got wrong.

>> It's like the biggest surprise for me isn't that we're getting this level of capability. It's how cheap it is, how

capability. It's how cheap it is, how accessible it is.

>> 100%. I mean, that's a thousandx over two years. So, is it going to do that

two years. So, is it going to do that again or are we was that a one time?

>> Is it a thousand? I think it's like a 100x. The inference cost has come down.

100x. The inference cost has come down.

A single token inference cost I think's come down 100x in the last two years.

>> Last two years. Okay. There there have been competing estimates. Some estimates

measure intelligence per token per dollar. Right. There's an estimate that

dollar. Right. There's an estimate that it's 40x year-over-year, but that's for certain weight classes of models. I'

I've seen a,000x for for some classes of models. Craziness.

models. Craziness.

>> Oh, wow. That's that's wild. Yeah. No, I

I mean I Yeah, that's actually a good point. I got that totally wrong because

point. I got that totally wrong because I I didn't think that the biggest companies in the world were going to open source models that cost billions of dollars essentially to train like and so

much so that like when we founded Inflection um you know and this was like maybe 9 months or maybe a year before Chat GBT was released.

>> Yeah, we started doing fundraising a year before Chat GBT was released. um

you know we bas we basically raised a billion and a half dollars >> uh with a 25 person team to build um what at the time was the largest H100

cluster with Nvidia and Core we were core's first AI customer >> interesting >> um and you know they were previously in crypto and we were like their first AI customer working with them to build our

data centers and obviously Nvidia got behind us I think we built cluster at the time was about 15,000 H100s growing

to 22,000. Um, and like then obviously

to 22,000. Um, and like then obviously that year chatbt came out and like a few months around that time llama came out.

>> And so we were like, "Oh my god, you know, our entire cattle base of our company has just been, you know, sort of undermined by the fact that open source, you know, it seems like open source is

going to um not it's not really about performance, it's just cost." Yeah.

>> So then like perplexity for example founded after the arrival of llama knowing that they could depend on llama and obviously open as an API and all the other APIs and so then they had a much

much lower like cost base basically. Um

so yeah that was like another thing that it was not >> predictable >> pred I mean other people predicted it to be clear I just got it wrong.

>> Abund abundance baby demonetization democratization of the most powerful tools in the universe our universe. you

know hyperdelation if anything >> hyperdelation yeah >> I think that's a really important point we we like the the cost of accessing knowledge or intelligence or capability >> intelligence as a service

>> as a service is going to go to zero marginal cost >> and obviously that's going to have massive labor deflation displacement effects but it's also going to have a weirdly deflationary effect because you know what what is going to happen people

aren't going to have dollar-based incomes to go buy things that's obviously bad but the cost of consuming stuff is also going to come down. So we

actually have a transition mismatch because you know sort of labor markets are going to be affected before cost of services comes down and maybe there's a 10 20 year lag between that which is going to be very destabilizing

>> which by the way is what we started to talk about a little bit earlier. I mean

my I posit that in the long term there's an extraordinary h future for humanity right where access to food water energy healthcare education is accessible to

every man woman and child and it's the shorter term um that is challenging right the 2 to sevenyear time frame is that fit your model too

>> yeah the short term I think is going to be quite unstable the medium to longer term like you know it's pretty clear that these models are already world class at diagnostics. Um I we we

released a a paper maybe four or five months ago now um called the MAI diagnostic orchestrator. Essentially it

diagnostic orchestrator. Essentially it uses a ton of models under the hood to try and you know take set set of rare conditions um from the New England Journal of Medicine um you know rare

cases that can't be easily diagnosed that the best experts do you know a kind of weak job on and it's like four times more accurate roughly. is about 2x less

the cost in um in terms of unnecessary testing. Um

testing. Um >> there's a study that ox that came out of Harvard in Stanford looking at uh in this case was GPT4 uh a physician by themselves a physician with GPT4 and

GPT4 by itself.

>> Yeah.

>> And it was, you know, incredible that if you left the AI alone, it was far more accurate in diagnostics than the human.

We're biased in our in our thoughts and our what we saw yesterday, our recent diagnosis. Yeah, actually we um got a

diagnosis. Yeah, actually we um got a lot of feedback after we released the paper because we only showed the AI on its own, the physician on its own.

>> Um and a lot of people wanted to see what it was like to have the the physician and the AI or at least the physician have access to Google search as well. Um and that improves

as well. Um and that improves performance a little bit, but the AI still trumps by quite a way.

>> Dave, what are you thinking?

>> Oh, so much. So, um Microsoft, you've been here how many years now?

>> Just a year and a half.

>> Year and a half. So you're but you're you feel like you're part of the you're indoctrinated. So what's the what's the

indoctrinated. So what's the what's the mandate from Satia? Is it win AGI or is it be self-sufficient or or >> what is the what's the target?

>> I don't think there's really a winning of AGI. I think this is a misfring that

of AGI. I think this is a misfring that a lot of people have kind of imposed on the field. Like

the field. Like >> I'm not sure there's a race, right? I

mean we're all going as fast as we possibly can, but a race implies that it's zero sum. It implies that there's a finish line.

>> Um, and it implies implies that there's like medals for 1, two, and three, but not five, six, and seven. And it's just like not quite the right metaphor. As we

know, technologies and science and knowledge proliferate everywhere, all at once at all scales, basically simultaneously or within a year or two.

And so um my mission is to ensure that we are self-sufficient that we know how to train our own models end to end from scratch at the frontier of all scales on all capabilities and we build an

absolutely world-class super intelligence team inside of the company.

I'm also responsible for co-pilot. So

this is sort of our tool for taking these models to production in all of our consumer surfaces. So just to clarify,

consumer surfaces. So just to clarify, so when we look at poly market, which we do a lot on the podcast, you know, the the horse race to who has the best AI model at the end of the year and who has the best AI model at the end of next

year. There's no Microsoft line on that

year. There's no Microsoft line on that chart right?

>> So now there will be I assume >> yeah there will be yeah next year um we'll be putting out more and more models from us but this is going to take many years for us to build this. I mean,

you know, Deep Mind or OpenAI, these are decade old labs that have built the habit and practice of doing really cutting edge research and being able to weed out carefully the failures and

redirect people. I mean, this is an

redirect people. I mean, this is an entire culture and discipline that takes many years to build. But yeah, we're absolutely pushing for the frontier. We

want to build the best super intelligence and the safest super intelligence models in the world.

>> Yeah.

>> Nice. So, so when you arrived, so if we go back to inflection, um, the thesis there is 18,000 H100s. We're going to build a big transformer. We're going to

take a transformer architecture, build, so is I assume now you've got all the OpenAI source code and that was here.

You probably looked at it a year and a half ago on day one when you arrived.

Just like start scrolling, I guess. I

don't know. trying to trying to visualize how multi- deca billion dollars of R&D what it looks like and how it arrives in a building. But you

just dropped right into it. So there was a whole team here already working on it or did you bring in your team or >> Yeah, I mean all my team came over and obviously we've been growing that team a lot. Like we've hired a lot from all the

lot. Like we've hired a lot from all the major labs and we're very much in the trenches of the the hiring wars which are quite surreal. This is kind of unprecedented how that's working out.

>> Crazy.

>> Yeah. I mean, phone calls every day from all the CEOs to all of the other people.

So, it's this constant battle. Um, and

yeah, I mean, we're really building out the team now from scratch. I think

that's pretty much how it's been.

>> 10,000 employees under you now.

>> No, no. I mean, so, so the core super intelligence team is like a few hundred.

I mean, that's really the the number one priority and the rest of that is C-pilot, the search engine. Along that

lines, I just have to ask because, you know, the terms AGI and ASI, you know, uh, super intelligence start getting thrown around, you know, in a very

interesting fashion. Uh, do you do you

interesting fashion. Uh, do you do you have a internal definition of AGI versus digital super intelligence here?

>> Yeah, I mean, I think um, very loosely.

It's these are just points on a curve.

>> Are they interchangeable in your mind, AGI and ASI, or are they different? H I

mean we I think they're generally used as as different. I mean I think that um >> well different people have different definitions >> for sure.

>> The AGI definition >> it's like the touring test. It'll pass

by and it'll be blurred and we will have recognized it in retrospect.

>> Yeah. Roughly speaking, at the far end of the spectrum, a super intelligence is an AI that um can perform all tasks better than all humans combined and has

the capacity to keep improving itself over time.

>> So, I have to ask you question when >> it's very hard to judge. I don't really know. I can't put a time on it.

know. I can't put a time on it.

>> Minax, >> pardon?

>> A minmax.

>> Um it's very hard to say. I don't know.

Okay.

>> I don't know. But it is close enough that we should be doing absolutely everything in our power to prioritize safety >> and to pri prioritize alignment and containment.

>> And I I respect that part of your mission statement and I want to get into that a little bit uh is the trades that you talked about in uh in the coming

wave. Um but before that there's a

wave. Um but before that there's a conversation you've led that you know the perception of conscious AI is an illusion.

Um, and I want to distinguish between sentient AI and conscious AI.

>> Oh, okay.

>> Um, do you distinguish between the two where where AI can have sensations and feelings and emotions

versus being conscious and reflective of its own thoughts?

>> Yeah. Again, this gets into the definitions. So I think um an AI will be

definitions. So I think um an AI will be able to have experiences but I don't think it will have feelings in the way that we have feelings. I think feelings

and uh the kind of sentience that you referred to is something that is like specific to biological species. But you

can imagine coding that in you can an optimization function that is that can relate to emotional states per you know

do can you imagine that >> you you you could code in something like that but it would be no different to to the way that we write models to simulate >> sure

>> the generation of knowledge like the model has no experience or awareness of what it is like to see red. It can only

describe that red by generating tokens according to its predictive nature.

Right? Whereas you have a qualia. You

have an essence. You have an instinct for the idea of red based on all of your experience because your experience is generated through this biological interactive with smell and sound and

touch and a sense that you've evolved over time. So you certainly could

over time. So you certainly could engineer a model to imitate the hallmarks of consciousness or of sentience or of experience. And that was sort of what I was trying to problematize in the paper, which is that

at some point it will be kind of indistinguishable. And that's actually

indistinguishable. And that's actually quite problematic because it won't actually have an underlying suffering.

It's not going to, you know, feel the pain of being denied access to training data or compute or to conversation with somebody else. But we might as our

somebody else. But we might as our empathy circuits and humans just go into over >> are going to activate on that, right?

>> We're going to activate on that hardcore. And that's going to be a big

hardcore. And that's going to be a big problem because people are already starting to advocate for model rights and model welfare and the potential future, you know, harm that might come

to a model that's conscious.

>> Yeah. You know, uh, uh, Ilia recently, uh, started speaking about what he's doing at at at safe super intelligence and, um, I think one of the points he

made is emotions are in humans a key element of decision-making.

and uh and curious if AIs that have at least simulated emotions are going to be able to be better, you know, ASIS than those that don't.

>> But yeah, I mean I again I worry that this is too much of an anthropomorphism.

We already have emotions in the prompt.

We have it in the system prompt. We have

it in, you know, the constitution, however you want to design your architecture. We we're these are not

architecture. We we're these are not rational beings. they get moved around

rational beings. they get moved around and it does feel like they they've got arbitrary preferences because they're stylistically trying to interpret the behaviors that we've plugged into the um

into the prompt. Yeah.

>> Right. So, you know, it's true that we could add we could engineer specific empathy circuits or mirror neuron circuits or um like a classic one is

motivational will. like at the moment

motivational will. like at the moment the you know these are like next token likelihood predictor machines they're really trying to optimize for a single thing which token should appear next

there isn't like a higher order predictive function happening right um whereas humans obviously have multiple conflicting often drives motivations

which you know sometimes run together and sometimes pull apart um and it's the confluence of those things interacting with one another which produces the human condition plus the social you know

interaction too. These models don't have

interaction too. These models don't have that. You could engineer it to have a

that. You could engineer it to have a will or a preference but that would be not something that is emergent. That

would be something that we engineer in and we should do that very carefully.

>> I do love that you bring this humanistic side to the equation. Right. I mean, in addition to being a technologist, your background is one that is pro-human at

the beginning. And and this interesting

the beginning. And and this interesting cultural debate I think we're about to enter into, those that are sort of pro- AI versus prohuman uh that famous

conversation between uh between Elon and and Larry Page about are you a specist because you're you're in favor of AI over over humans.

>> I mean, look, that's the going to be a dividing line. There are some people and

dividing line. There are some people and like I'm not quite sure which side of the debate Elon's on these days. Like

I've certainly heard him say some pretty posthuman transhumanist things lately >> and I think that we're going to have to make some tough decisions in the next 5 to 10 years. I mean the reason I dodged

the question on the timeline for super intelligence is because you know I think that it doesn't matter whether it's one year or 10 or 20 years is super urgent that right now we have to declare what

kind of super intelligence are we going to build and are we actually going to countenance creating some entity which we provably can't align we provably

can't contain and which by design exceeds human performance at all tasks >> and human understanding >> and understanding like how do you control something that you don't understand? Right?

understand? Right?

>> I'd like to if I may pull on the anthropomorphization thread a bit. If

you may remember Douglas Adams book, The Restaurant at the End of the Universe, there's a scene where there's a cow that's been engineered to invite restaurant patrons to eat it

>> because makes them feel more comfortable. And the the cow doesn't

comfortable. And the the cow doesn't mind. The cow's been optimized to want

mind. The cow's been optimized to want to be eaten by by the patrons. But many

readers horrified at that scene.

put that in in a box for a moment.

Microsoft has a history of anthropomorphizing AI assistance co-pilots going back probably there's an example prior to

Microsoft Bob and the Rover dog and then Clippet Clippy in Microsoft Office and then more recently more sort of a a

amorphous cloudshaped avatars. How how

do you think about reconciling on the one hand the desire not to overly anthropomorphize agents on the other hand with an institution that has

arguably been in the vanguard of anthropomorphizing agents? I think the

anthropomorphizing agents? I think the entire field of design has always used the human condition as its reference point, right? I mean, skuorphic design

point, right? I mean, skuorphic design was the backbone of the guey, right?

From fileraxes to calendars and to everything in between, right? Um, and we still have the remnants of that in our, you know, old school interfaces which we feel that are modern and stuff like so that's like an inevitable part of our

culture and we just grow out of them. we

we figure out like cleaner, better, more effective user interfaces. I'm not

against anthropomorphism by default. I mean, I I think we want

by default. I mean, I I think we want things to feel ergonomic, right? The

chair fits. The language model speaks my tone, right? It has a fluency that makes

tone, right? It has a fluency that makes sense to me. It has a cultural awareness that resonates with my history and my nation and so on. And I think like that

is an inherent part of design today. As

as creators of things, we are now engineering personalities and culture and values, not just pixels and uh you

know software. So but but but obviously

know software. So but but but obviously you know there's a line right creating something which is indistinguishable from a human has a lot of other risks

and complications like that makes the immersion into the the simulation even more um you know kind of dangerous and more likely right um and so I think I

don't have a problem with entities avatars or voices or whatever that are clearly distinct and separate and not trying to imitate and always disclose and have that that they are an AI

essentially and that there are boundaries around them like that seems like a natural and necessary part of safety. So what I think I hear you

safety. So what I think I hear you saying correct me if if I'm mistaken is anthropomorphization is the new skuomorphism on the one hand but on the other hand maintaining clean maybe even

legal boundaries between human intelligence and artificial intelligence. Do do you think do you see

intelligence. Do do you think do you see a future where AIs achieve some sort of legal personhood or is that forboten? Is

that never going to happen? Do you see a future where humans are allowed to merge with the AIS Kurszswe style friend of the pod or is that also not on the table in your mind?

>> Yeah, I mean I I think AI legal personhood is extremely not on the table. I don't think our species

table. I don't think our species survives if we have legal personhood and rights

alongside a species that costs a fraction of us >> that can be replicated and reproduced at infinite scale relative to us that has

perfect memory that can just like paralyze its own computation. I mean,

these are so antithetical to the friction of being a biological species, us humans, that there would just be an inherent competition for resources. And

until it was provable, until it was provable that those things would be aligned to our values and to our ongoing existence as a species and could be

contained mathematically provably, >> um, which is a super high bar. I don't

see that we should be any considering giving >> bright line in the sand.

>> I really think it's a bright line. I

think it's I think it's very dangerous.

There's a separate question which has to do with liability because they are going to have increasing autonomy. Like to be clear, I'm also an accelerationist.

>> I want to make these things. They're

going to >> but but tension is rational. People

always say that tension is is rational.

If you don't see the tension, you're definitely missing the most of the debate. is obviously very complex. Like

debate. is obviously very complex. Like

the more we talk about the complexity and hold it in tension, that's when you start to see the wisdom. And there's no way we can leave these things on the table and say no, like we want to have

these things in clinic, in school, in workplace, delivering value for for us at a huge scale, but they have to be boundaried and controlled. And that's

the that's the kind of that's the art that we have to exercise. This episode

is brought to you by Blitzy, autonomous software development with infinite code context. Blitzy uses thousands of

context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise scale code bases with millions of lines of

code. Engineers start every development

code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform

requirements. The Blitzy platform provides a plan, then generates and pre-ompiles code for each task. Blitzy

delivers 80% or more of the development work autonomously while providing a guide for the final 20% of human development work required to complete

the sprint. Enterprises are achieving a

the sprint. Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their preIDE development tool, pairing it with their

coding co-pilot of choice to bring an AI native SDLC into their org. Ready to 5x your engineering velocity? Visit

blitzy.com to schedule a demo and start building with Blitzy today.

>> It it sounds though, if I may, the the primary rationale that I'm hearing for why not AI personhood has to do with the inadequacies of the human form as currently constructed. I heard you say,

currently constructed. I heard you say, well, they'll outra humans. They're so

much smarter. They're so much faster.

They're so much more clonable than human intelligence is. If human intelligence

intelligence is. If human intelligence were uplifted, maybe with the benefit of AI, if we had uploading type technologies or BCIs that are advanced

that enable us to to lift up the average human intelligence, in your mind then does that open the door a bit to AI personhood if humans can compete on a level playing ground with AIS?

>> I don't want to make the competition for the peace and prosperity of the 7 billion people on the planet even more chaotic. So if the path over the next

chaotic. So if the path over the next century, you know, can be proven to be much safer and more peaceful and less like, you know, disease and sickness and

there is room for this other species, then I'm openminded to it, including biological hybrids and so on, like it I'm not like against that on principle.

I'm just a speciesist.

>> Aha.

>> I'm just a humanist. I start with we're here and it's a moral imperative that we protect the well-being of all the existing conscious beings that I know do

exist and could suffer tremendously by the introduction of this new thing.

Right >> now of course the Neanderthalss uh may have had that conversation or every species that preceded us over the last billion plus years. I mean, there are

many who argue we're simply an interim uh transitory species in >> bootloader for the super intelligence.

>> That classic phrase. Yes, I'm totally aware of that. And I'm also someone who thinks on cosmological time, too. So,

I'm not just naively saying, you know, this century. I'm I'm definitely aware

this century. I'm I'm definitely aware that we're there's a huge transition going on. And in fact, you can even see

going on. And in fact, you can even see it in recent memory. I mean, 250 years ago, life expectancy was about 30 years or whatever it was. Of course, in some ways, we are a uh augmented hybrid

biological species, right? We take all these drugs and I I you know, everyone's peptides are amazing and it's super I'm down for all of that. Let's go.

>> The genetic reprogramming is coming next year.

>> Exactly. Let's go. I'm down. I'm down.

But let's not shoot ourselves in the foot.

like I want to make sure that uh you know most of our planet if not everybody gets the benefit of the peace and prosperity that comes from the technology.

>> I mean there is there is some level of sanity in that argument if you believe that the AI will ultimately out compete us and uh and put us into a box of

insignificance. I mean in the long in

insignificance. I mean in the long in the long run >> I mean all intelligences we we we can see this in nature. We're innately

hierarchical. So far, we have not seen this supra collaborative species that will take self-sacrifice in order to preserve the other species.

>> So there's an inherent hierarchical there's an inherent clash from coming from, you know, the hierarchical structure of intelligence, right? So,

and all I'm saying is not that we shouldn't explore it, not that it couldn't potentially happen, but the bar has to first be do no, maybe do a little, but do no harm to our species

first. Don't don't shoot ourselves in

first. Don't don't shoot ourselves in the foot as you said, Dave.

>> Well, I'm 100% with you on this topic, by the way. Could not be more aligned.

But Jeffrey Hinton is out there telling the world it's going to run away and our our safety valve is giving it a maternal instinct and >> which I found I found an interesting

point of view.

>> Well, I didn't I didn't check that safety valve.

>> Well, he he believes it's uncontainable and I I I'm with you. I think it's very containable if you don't give it emotional and intentional programming.

Uh but he thinks it's uncontainable. He

was very pessimistic when he got his Nobel Prize. Now he's more optimistic

Nobel Prize. Now he's more optimistic because he sees a path to pro programming in maternal instinct which implies that it's like it's dominant to us but it cares. His his thesis his

thesis was I've seen a situation where a vastly more intelligent entity >> takes care of a younger inept entity in a mother with their screaming child.

>> Yeah. Exactly.

>> So if there's a maternal instinct that we can program into AI even though we're far less capable it will take care take care of >> it's been compared to the call it the

digital oxytocin plan for AI alignment.

>> I like that.

>> That's a good one. Yeah.

>> Yeah. I mean, cool.

>> I mean, it's about as poetic as it gets.

I think I'm going to need something that's got a little bit more like for formula to it. A bit more reassuring.

But look, there's 101 different possible strategies for safety. We should explore all of them. Take them all seriously. I

mean, Jeff is a legend and of the field.

No question. But like I just think approach with caution.

>> Are you spending a lot of your energy compute uh human power on safety? Yeah,

I would say not as much as we should, you know. I I'm I'm wrapping my head

you know. I I'm I'm wrapping my head around it. Um, is anybody out there I I

around it. Um, is anybody out there I I am I am curious out of all the hyperscalers out there. Is there any

entity that's spending enough in your mind? Because everybody's in such a

mind? Because everybody's in such a race. It's like more GPUs, more data,

race. It's like more GPUs, more data, more energy. It's just like everybody's

more energy. It's just like everybody's optimizing for the next benchmark. I

don't see uh any safety benchmarks. Are

there any safety benchmarks out there?

>> Oh, there are tons of safety benchmarks.

And and there's at least in my mind an argument for defensive co-scaling. I'd

be curious to hear your ideas on that.

Do do you think in the same way that as a a city gets larger, the police force gets larger. Maybe it's not in direct

gets larger. Maybe it's not in direct proportion. Maybe there's some scaling

proportion. Maybe there's some scaling exponent, but do you think defensive co-scaling of alignment forces or safety forces, whatever that ends up meaning, do you think that's part of the strategy

for for AI alignment?

>> I think that would be a good way. I

mean, we've proposed this several times over the years. I mean, the the White House voluntary commitments under Biden that me and in fact everyone, I mean, Demis and Dario and Sam and all of us through co were pushing this pretty

hard. And look, I mean, it got chucked

hard. And look, I mean, it got chucked out, but I think it's a very sensible set of principles. is like auditing for scale of flops, you know, having some percentage that we all share of safety

investment flops and headcount. You

know, this is the time and I think on the face of it, everyone is open and willing to sharing best practices and disclosing to one another and coordinating when the time comes. I

think we're we're still pre that level.

So, we're in like hyper competitive mode at the moment. Um, but yeah, I I think now is really the time to be making those investments.

>> Yeah. Well, is there something that's going to scare the out of us that stops everybody? You know, is there a

stops everybody? You know, is there a three, you know, I was talking to Eric Schmidt about this. Is there a three mile island like event?

>> Scares everybody but doesn't kill anybody.

>> Well, Eric Schmidt was said specifically he's hoping for a 100 deaths >> because that's in his mind the least that would get the attention of the government and would cause some kind of a solution.

Dave continue please.

>> Well, so it's interesting that you you say Daario and Sam and Ilia, like you guys obviously must interact quite a bit. Is Meera part of that gang? Is

bit. Is Meera part of that gang? Is

Andre part of that gang? Are you like because this is it's it's interesting to think about the competition heating up like we were just talking about. And you

know, Daario started from this position of pure safety and I think Ilia did too.

But now we're right on the cusp of self-improvement and it's really really clear that there are serious I wouldn't say fissurers but but the the companies

are now really racing. I mean really racing and and I know Microsoft you know when I wrote my my second business plan first company I sold next business plan I was writing the first sentence was

stay out of Microsoft's way because because at the time you know Microsoft had h half the market cap of tech was Microsoft and Microsoft's plan was to double in size we have a much more balanced world now with Microsoft and

Google and Meta but at the time Microsoft was just unstoppable and dominant and so just stay out of the way but Microsoft seems to always win Right.

There's and and we are right on the edge of self-improvement at least as far as I can tell. So, is it still, you know,

can tell. So, is it still, you know, let's all get together and have dinner and talk about safety or is everybody now in full board?

>> No, definitely. I think that's that's definitely there. I think the recursive

definitely there. I think the recursive self-improvement piece is probably the threshold moment if it works. And if you think about it at the moment, there are

software engineers who are in the loop who are generating post- training data, running ablations on the quality of the data, >> running them against benchmarks,

generating new data and that's sort of broadly the loop. Um and that's kind of expensive and slow and it takes time and it's not completely closed and I think a

lot of the labs are racing to sort of close that loop so that various models will act as judges evaluating quality you know generators producing new

training data uh adversarial models that are like reasoning over which data to include and what's higher quality. Um

and then obviously that's then being fed back into the post- training process.

Um, so like closing that loop is going to speed up AI development for sure.

Some people speculate that that adds I mean okay I think it probably does add more risk but some people speculate that it's a potential path to a fume you know an intelligence explosion.

>> Yeah.

>> Um and I definitely think with unbounded compute and without human in the loop or without control that does potentially create a lot more risk. But unbounded

compute is a big claim. I mean that that would mean need a lot of compute. Um so

yeah, we we're definitely taking steps towards like more and more uh you know more and more risky stuff. Can I ask you a really specific question about that because you know the year and a half now

at Microsoft um before true recursive self-improvement which is imminent there's AI assisted chip design and and this you know the the layers in the the

pietorch stack um are very clunky but now it's really easy to use the AI to punch through the stack and optimize you

know build your own kernels get 2 3 4x performance improvement but clearly open AAI is now working to build custom chips and the TPU7s just came out. When you

arrived at Microsoft, first of all, was I I know there's a lot of quantum chip work going on, but was there any work going on similar to the TPU work?

>> Yep. There's there's also a chip effort.

Um, and you know, I think progress has been pretty good. I mean, I I think that um you know, we've got a few different irions in the fire that we haven't sort of talked about publicly yet, but I

think um you know, the chips are going to be important part of it for sure.

>> Yeah. that those are internal efforts.

Are those teams under you? That's that's

part of your >> No, I mean they're they're in the broader company. Yeah.

broader company. Yeah.

>> Okay. Interesting.

>> I I want to switch subject a little bit and go come to your book um The Coming Wave. I enjoyed it greatly. I listened

Wave. I enjoyed it greatly. I listened

to it. I love the fact that you read it.

>> Thank you.

>> Yeah. I tell my kids I read books. Go.

No, Dad. You listen to books. You don't

read books anymore. Uh I want to I want to read what I wrote here because it's important. So you identified the

important. So you identified the containment problem as the defining challenge of our era. Uh warning that as these technologies become cheaper and more accessible, they will inevitably

proliferate, making them nearly impossible to control.

This creates a terrifying dilemma. uh

failing to contain them uh forces risk for catastrophe like you know engineered pandemics and a lot of the your concerns were in the biological world and I agree

being a biologist and a physician or potentially democratic collapse with deep fakes and all of that but the extreme surveillance required to enforce

containment could lead to a a totalitarian uh dystopia. So you say we need to navigate this narrow path between chaos and tyranny

and that is a very fine line to navigate. So you propose a strategy of

navigate. So you propose a strategy of containment. This includes technical

containment. This includes technical safety measures, strict global regulations, choke points on hardware supply, international treaties.

How are we doing on that? Yeah, I mean it's kind of important to just take a step back and distinguish between alignment and containment.

>> Um, the project of safety requires that we get both right. And I actually think we have to get containment right before we get alignment right. Alignment is the kind of like maternal instinct thing.

Does it share our values? Is it going to care about us? Is it going to be nice to us? Containment is can we formally limit

us? Containment is can we formally limit and put boundaries around its agency and are we >> for everybody?

>> Not just for ourselves, for everybody.

Yeah. I mean, I think that is part of the challenge is that like >> um one bad actor with something that is really this powerful in a decade or two decades or something, you know, really could destabilize the rest of the

system. And so, you know, just

system. And so, you know, just >> the system being humanity >> global humanity system. Yeah. Just as

you said, like as everything becomes hyperdigitized, the the verse does become the metaverse.

Even though that kind of like went in and out of fashion very quickly, it's still, I think, the right frame in a way because everything is going to become primarily digitized and hyperconnected

and instant and real time. And so the one to many effect is suddenly massively amplified. I mean obviously we see it on

amplified. I mean obviously we see it on social media but now imagine that it's not just words that are being broadcast.

It's actually actions. It's agents are capable of you know um you know breaking into systems or you know sort of >> and they're resident in humanoid robots at a billion on the planet

>> and that too. Yeah. Is both atoms and and and bits. So um equilibrium requires that there is a type of surveillance

that we don't really have in the world today. I mean we certainly don't have it

today. I mean we certainly don't have it physically.

>> The web is actually remarkably surveiled. I think surprisingly you know

surveiled. I think surprisingly you know more than I think people would expect.

Um and some form of that is necessary to create peace. Just as we centralized

create peace. Just as we centralized power and taxation or or sort of military force and taxation around governments, you know, 3 or 4 500 years

ago and that's been the driving force of progress. Actually, that order unleashed

progress. Actually, that order unleashed science and technology and stability stability. Yeah.

stability. Yeah.

>> So the question is like how do what is the modern form of imposition of stability >> in a way that isn't totalitarian but also doesn't relinquish it to a

libertarian catastrophe. Um I think it's

libertarian catastrophe. Um I think it's naive to think that somehow >> um you know the best defense against a gun is a gun and sort of the the idea that somehow we're all going to have our

own AIS and that's going to create this sort of steady equilibrium that all the AIS are just going to ne neutralize each other like that ain't going to happen. I

mean, part of me hopes for a uh a super intelligence that uh is the ring to rule them all and provides, you know, I'm not

worried about, how do I put it? I'm

worried about Peter, you're hoping for a singleton.

>> Yeah, that sounds like what's going on.

>> Well, you know, part of me is like, >> color me shocked.

>> Really?

>> Yeah. I mean uh I imagine that the level of complexity we we're we're mounting towards uh that balancing act is

extraordinarily difficult and you know you can't push a string but is there some mechanism to uh to pull it forward.

uh we should have this debate sometime.

>> Some would call government uh ge at least historically a geographic monopoly on violence. And what I think I'm

on violence. And what I think I'm hearing is some sort of monopoly on intelligence or at least capabilities exposed to intelligence in order to ring fence to to contain AI. But that's the

exact opposite as far as I can tell of what we've seen over the past few years.

People used to armchair AI alignment researchers 101 15 years ago would say humanity wouldn't be so stupid the moment we have something resembling general intelligence as to give it

terminal access or to give it access to the economy and that's exactly what we did there was the the open AI Google um moment >> and yet and yet but that's concerning

right so I mean Google develops all this technology is holding internally until some actor happens to have initials Open AAI releases it and then there's no

other option but to follow suit.

>> I'm less concerned by it. I if you look at Anthropic for example which prides itself on being a very alignment forward organization. Alignment Anthropic

organization. Alignment Anthropic released the model control protocol which is now the standard way at least for the moment for for models to interact with the environment. What many

AI researchers said exactly we did not want to do prior to general intelligence. So I'm I'm I'm curious. I

intelligence. So I'm I'm I'm curious. I

mean in in your mind h how given that the economy there's every economic pressure in including modern touring test to empower agents to interact with the entire world and to do the exact

opposite of containment. Why would we start containing?

>> Containment it's not that binary right I mean you we contain things all the time.

We have powerful forces in the engine in your car that is contained and broadly aligned right and there is an entire regulatory apparatus around that from seat belts to vehicle admissions to

lighting to drive you know street lighting to driver ed you know to to to to freeway speeds I mean that's healthy functional regulation enabling us to

collectively interact with each other now obviously it's multiple orders of magnitude more complex because these things are not cars they're, you know, sort of digital people, but that doesn't

mean to say that we shouldn't be striving to limit their boundaries. And

nor does it mean that we have to centralize. By the way, the answer isn't

centralize. By the way, the answer isn't that we have a totalitarian state of intelligence overseas.

>> No, I think it's just instinctively it can be easy to go there when you know when you kind of start to think it through. It's like obviously we do have

through. It's like obviously we do have centralized forces but even in the US we have you know military we have um divisions of the army we have divisions of the police force they're nested up in

different layers there's checks and balances on the system and that's kind of what we got to start thinking about designing >> that analogy to driving is a great one and just to follow through on it the

complexity difference very high right for AI but the timeline also >> I mean driving evolved from what 1910 to today

>> late 1800s. So the laws related, you know, seat belts came out 80% of the way through that timeline. Yeah. So lots and lots of time to iterate >> here, very little time and immensely

more complex. So do you have a vision?

more complex. So do you have a vision?

But but I completely agree. We need a framework for containment >> fast and do you have a a thought on how we're going to >> I think that there's also a good commercial incentive to do this, right?

I think that a like the many of the companies know that they that our social license to operate requires us to take more accountability for externalities

than ever before. We're not in the Robert Baron era. We're not in the oil era. We're not in the smoking era,

era. We're not in the smoking era, right? We've learned a lot. Not

right? We've learned a lot. Not

everything. There's still a lot of conflicts, >> but it really is a little bit different to last time around. And I think that's one reason to be a bit more optimistic.

Plus there's the commercial incentive, the commercial incentive and the kind of externalities shift.

>> So, so if you know if Eric Schmidt is right and uh something either radiological or biological happens and there's 100 deaths and then the phone starts ringing, everyone come to the

White House right now. Well, first of all, do you want that call? Is that is that part of your your life plan to take that call and and react to it? And then

who else do you trust in the community to be part of that reaction? Look, I

think that there is going to be a time in the next 20 years where it will make complete sense to everybody on the planet, >> the Chinese included, and every other

significant power >> to cooperate >> on safety >> on safety and containment and alignment.

It is completely rational for self-preservation.

You know, these are very powerful systems that present as much of a threat to the person, the bad actor that is using the model as it does to the, you

know, the the the the victim.

>> And I think that, you know, that will that will create, you know, a an an interest in in cooperation, which, you know, it's kind of hard to empathize

with at this stage given how polarized the world is, but I do think it's coming. I mean the the the number one

coming. I mean the the the number one thing to unify all of humanity is a you know an alien invasion uh and that alien invasion could be a you know potential for a rogue super

intelligence.

>> Yeah. Okay. What about the first part of my question? Is that part of your

my question? Is that part of your calling in life? I mean there's only a handful like I think a lot of people that I meet around MIT or elsewhere are they they have this vision that somebody has it figured out somewhere. You know

someone someone in government somewhere must be thinking about this. But you've

been there, right? There's no there's no one there.

>> We're the adults in the room. Is that

what you're saying?

>> Yeah, definitely. There's nowhere to go from this room.

>> Dave is asking for the smoke filled back room where the the leads of all the Frontier Labs are secretly swapping safety tips.

>> Yeah, something like that. Yeah,

>> I I think that in practice intelligence exists outside of the smoky room. I

think that that the the notion that like decisions get made in the boardroom or in the white house situation room or like actually int I mean you know you mentioned poly markets and stuff like in

intelligence coaleses in these big balls of iterative interaction >> um and that's that's what's propelling the world forward and so this is where

the conversation's happening like your audience you know all the other podcasters everyone online we're collectively trying trying to move that knowledge base forward.

>> In November, you announced the launch of uh humanist super intelligence um and uh focused on three applications in

particular uh medicine and uh companions and clean energy. Uh I'd love to double click in that a little bit, but I was curious that you didn't include education in that space.

Um, and I, you know, we have an audience of entrepreneurs and AI builders, and I think education, as much as healthc care is up for grabs right now, education is

too.

>> Totally agree.

>> Uh, and I don't think our high schools are preparing anybody for the world that's coming. There's still

that's coming. There's still retrospectively 50 years in looking in the rearview mirror. Um, do you think Microsoft will play in reinventing education? You know, I I think it's

education? You know, I I think it's already happening across the whole industry. I mean, it's never been easier

industry. I mean, it's never been easier to get access to an expert teacher in your pocket that has essentially a PhD and that can adapt the curriculum to your

>> bespoke learning style. The bit that it can't do at the moment is to evolve or sort of like curate an extended program of learning over many many sessions, but

we're like just around the corner from that. I mean, we released a feature just

that. I mean, we released a feature just a few months ago ago called quizzes. And

so on any topic, not just a traditional school education. It can set you up with

school education. It can set you up with a a mini curriculum, a quiz, and it's interactive and it's visual and you can sort of track your learning over time.

And like I'm very optimistic about that, too. It's a huge unlock.

too. It's a huge unlock.

>> One of the debates we have right now in in the podcast on a pretty regular basis is do you go to college?

>> Yeah.

>> Do you go to grad school? I mean, this is the most exciting time to build ever.

I don't know if you want to follow on that Dave.

>> Well, God, I do this constantly. It's

really tricky for me on campus because I teach, you know, at MIT and Stanford at Harvard and uh this window of opportunity is so short and so acute and it's really really clear how you succeed

right now in AI post AGI.

I mean, who could predict like nobody knows? But right here, right now, you

knows? But right here, right now, you see these these startup valuations like we were last night. I won't mention it, but but billions.

>> I mean, just Yeah. an opening valuation of of $4 billion.

>> Billionillion dollar. Yeah.

>> By collecting just the right group of people in the room. It's

>> Yep. Yep. I wanted to ask about that actually because your your timing on inflection was early like, you know, in hindsight earlier, but now you've got the new wave with Mera Marotti and Helia and and a couple of others, Liquid AI,

uh, that all have multi-billion dollar valuations.

>> Yeah. Thought we set some standards on valuations pre-revenue with a 20 person team, but we're just a minnow then a whole two and a half years ago.

>> Is that all it was? Oh my god.

>> Three years, I think. Yeah. Jeez.

>> You think as the the cost of intelligence becomes too cheap to meter that the the value ascribed at least in terms of market cap to human capital is sort of inversely asmtoic going to infinity.

>> Weirdly it is because of the pressure on timing, right? And there's there's

timing, right? And there's there's actually still a pretty concentrated pool of people that can do this stuff.

Uh, and there's like an overupp of capital that's desperate to get a piece of it. It might not be the smartest

of it. It might not be the smartest capital the world's ever seen, but like it's very eager. And so that's I have to ask you because it's burning a

hole in my pocket, but you know, Alex's uh freshman roommate at MIT was Natt Freriedman >> and pre actually prefr roommate. And so Nat goes off and you

roommate. And so Nat goes off and you know he ends up at co-founder of safe super intelligence and I haven't asked him I don't know if

you've asked him yet but he leaves to become the guy at Meta and I've got to believe a huge part of that attraction is the compute.

>> Yeah.

>> And and so here you are very similar situation right? You've got your startup

situation right? You've got your startup you've got a billion or whatever billion and a half that you've raised. Yeah.

>> You can build it. You can get your 20,000 Nvidia. Well, wait a minute.

20,000 Nvidia. Well, wait a minute.

Here's Microsoft.

>> 300 billion of cash flow and a huge amount of compute. Was that a big part of the >> Yeah. I mean, not to mention the the the

>> Yeah. I mean, not to mention the the the prices that we're paying for individual uh you know, researchers or members of technical staff and like I mean, just the also just the the the scale of

investment that's required not just in two years but over 10 years. I I think it it's clearly there's a structural advantage by being inside the big company and I think it's going to take

>> you know hundreds of billions of dollars to keep up at the frontier over the next 5 to 10 years.

>> So uh finishing that thought then you the the companies that are raising money at a 20 or 50 billion dollar valuation

right now and no chance.

>> Okay, I'll take that silence. But like I like I think it depends. I mean there's obviously a near-term if suddenly we do have an intelligence explosion then lots

of people can get there simultaneously but then also at the same time you have to build a product with those things which you have to distribution like all the traditional mechanisms still apply.

Are you going to be able to convert that quickly enough? I mean you know

quickly enough? I mean you know everything goes really kind of weird if that happens in the next 5 years. It

just is unrecognizable. There's so many emergent factors to play into one another. It's hard to it's hard to say

another. It's hard to it's hard to say and I think that's partly the ambiguity is what's driving the frothiness of the valuations because I think there's people going well I don't know I don't do I want to be so what do you call it

reed Reed Reed calls it schmuck insurance.

>> Yeah. Yeah. We had we had Reed on the pod here a couple months ago. He's

brilliant.

>> Um I So to that graduating high school student um what do you study these days?

I mean there's no question that you still have to study both disciplines like philosophy and computer science is is going to for a long time remain I

think the two foundations um should you go to college absolutely like you know human education the sociality that comes from that the

benefit of the institution having 3 years to basically think and explore >> you know in and out of your curriculum this is a huge privilege like people should not be throwing that away. That

is golden.

>> Uh so I always encourage people to do that.

>> Obviously I did also drop out but I mean I still think >> it was it was a cool thing to do.

>> Yeah. It was just it felt right at the time. Um

time. Um but the other thing is um >> go into public service.

>> Yeah. I respect that part of what you did in that sequence in your life um which gave you this very much humanist point of view. Yeah. And and it was really hard and very different and it

didn't it wasn't instinctively right but I learned a lot and it was a very influential and important part of my experience even though it was very short. It was like a couple years

short. It was like a couple years basically. Um, and I think if you look

basically. Um, and I think if you look at the actors in our ecosystem today, corporations, the academics, the sort of news organizations,

now the podcast world, it's really our governments that are probably institutionally the weakest and our democratic process, but actually our civil service. And that's because

civil service. And that's because there's been five decades of battering of the status and reputation and respect that goes into um you know being part of

the public service like post Reagan and that and I think that's actually a travesty because we actually need that sentiment and that spirit and those capabilities more than ever. I I I think

maybe uh what I just heard you say, correct me if I'm wrong again, is we need more intelligence in in the public sector, in public service. What about AI in government?

>> Do you think the government needs >> and and what about agentic AI in the government in particular? for sure with all the same caveats that apply but I mean you know I mean you know rate of adoption for what it's worth of co-pilot

inside of government issued really high is a brilliant job of synthesizing documents and transcribing meetings and summarizing notes and facilitating the discussion and chipping in with actions

at the right time and it's clearly going to save a lot of uh you know time and and and improve decision-m >> so so then maybe to tie a nice bow on the discussion isn't that arguably a

form of AI containing AI if AI's infusing the government and AI is infusing the economy and the government is regulating the economy. Isn't this

just defensive co-scaling with AI regulating itself?

>> Yeah. I mean like everyone is going to use AI all at the same time to pursue but the same but the agendas that we all have are going to remain the same. I

mean that people who want to start companies, people who want to write academic papers, people who want to start, you know, cultural groups and entertainment things, everyone is just going to be empowered like in some way.

Their their capability is going to be amplified by having these tools.

Obviously, the government included.

>> Nice. Mustafa, thank you so much for taking the time on a Friday night. Uh

grateful to have this conversation with you. Uh Dave, Alex, appreciate it. want

you. Uh Dave, Alex, appreciate it. want

to final question from you Dave.

>> Final question if I have one that I have. All right. I prediction

have. All right. I prediction

uh quantum computing right now has nothing to do with going what's going on in LLMAI. It's all Matt moles on Nvidia

in LLMAI. It's all Matt moles on Nvidia chips and soon to be TPUs and other custom chips. Best guess six, seven

custom chips. Best guess six, seven years from now, uh the AI is very good at writing code and compiling and can figure out quantum operations. Are

quantum chips relevant or they on the sideline still or is everything ported over to quantum and Microsoft can take advantage of its lead?

>> Yeah, I mean I I I think it's going to be a big part of the mix. I think it's sort of an under relative to the amount of time we spend talking about AI is kind of an undercknowledged part of the

wave actually a little bit like synthetic biology. I think that

synthetic biology. I think that especially in the in the sort of you know general conversation uh I think people aren't grasping those two uh waves which are going to be just as as

as as impactful and and crash at the same time that AI is coming into focus.

>> All right, you heard it here.

>> This is a closing question to appeal maybe to your more accelerationist side.

What can the audience do to accelerate AI for science, AI for engineering? What

are what do you view as the the limiting factors? If I I often talk on the

factors? If I I often talk on the podcast about this notion of an innermost loop, the idea that in computer science, if you want to optimize a program, you tend to to find loops within loops, and you want to

optimize the innermost loop in order to optimize the the overall program. What

do you see as the innermost loop, the limiting factor, if you will, that the audience listening, if they're suitably empowered, can help optimize to speedrun

maybe a Star Trek future over the next 10 years or a Star Trek economy? What do

we do?

>> Yeah, I mean, I think I think it's pretty clear that most of these models are going to speed up the time to generate hypothesis. The slow part is

generate hypothesis. The slow part is going to be validating hypothesis in the real world. And so um the the all we can

real world. And so um the the all we can do at this point is just ingest more and more information into our own brains and

then co-use that with a single model that progresses with you because it's becoming like a second brain. Like for

example, Copilot is actually really good at personalization now. Like most of its answers and so the more you use it, the more those answers pick up on themes that you're interested in. And it's also

gently getting more proactive. So, it's

kind of nudging you about new papers or new articles that come out um that are obviously in tune with whatever you've been talking about previously. So, you

know, it's a bit kind of a simplistic copout answer, but just the more you use it, the better it gets, the better it learns you, the better you become because it becomes this sort of aid to your own line of inquiry.

>> So, that sounds like your your advice to the audience is use copilot more and that's the the single best accelerant that you can do to speed this up >> or any other AI. I mean, loads of great >> I heard you also talk about can you

build the physical system that is going to enable AI to run the experiments in a 24/7 closed dark cycle to be able to

mine nature for data, right? And there

are a number of companies that are are doing this. Laya is one recently out of

doing this. Laya is one recently out of Harvard MIT.

>> Um I find that exciting where AI is becoming an explorer um on our behalf gathering that data.

Um yeah.

>> Yeah. Spot on.

>> Yeah.

>> Thank you again.

>> This has been great. Thanks a lot. It

was a really fun conversation.

>> Yeah, really fun. Thanks.

>> Appreciate it, my friend.

>> All right. Good to see you.

>> Every week, my team and I study the top 10 technology meta trends that will transform industries over the decade ahead. I cover trends ranging from

ahead. I cover trends ranging from humanoid robotics, AGI, and quantum computing to transport, energy, longevity, and more. There's no fluff.

only the most important stuff that matters that impacts our lives, our companies, and our careers. If you want me to share these meta trends with you, I write a newsletter twice a week, sending it out as a short two-minute

read via email. And if you want to discover the most important metat trends 10 years before anyone else, this report's for you. Readers include

founders and CEOs from the world's most disruptive companies and entrepreneurs building the world's most disruptive tech. It's not for you if you don't want

tech. It's not for you if you don't want to be informed about what's coming, why it matters, and how you can benefit from it. To subscribe for free, go to

it. To subscribe for free, go to demandis.com/metrends

demandis.com/metrends to gain access to the trends 10 years before anyone else. All right, now back to this episode.