Inside Nemotron & NVIDIA’s AI Lab | Bryan Catanzaro

If you accept as the truth that we're going to be running at the limit, then what that means is that the way to get more intelligence is to be more efficient. We can't get more

efficient. We can't get more intelligence by applying more force if we're already at the limit. We have to be more thoughtful about how we use what we have. We build tools. We build

we have. We build tools. We build

external organs that help us solve problems. You know, we we have an external stomach. We call it a kitchen.

external stomach. We call it a kitchen.

Now, we're creating an external brain.

What is the implications of an external brain? Pretty profound. Nobody actually

brain? Pretty profound. Nobody actually

really knows.

Hi, I'm Matt Turk. Welcome back to the Mad Podcast. Open source AI is having

Mad Podcast. Open source AI is having yet another moment with powerful new models arriving almost weekly. And my

guest today is one of the very best people to unpack it all. Brian Kentaro

leads Neimotron, Nvidia's family of open foundation models. Now, not everyone

foundation models. Now, not everyone realizes Nvidia has a massive effort to build frontier AI models, but it employs hundreds of AI researchers and Neotron 3

Ultra immediately became the number one US openweights model when it was released just a couple weeks ago. We

begin this conversation with the state of open source AI and the race between the US and China and then we go deep inside Neotron for bit training hybrid member transform architecture mixture of

experts multi-token prediction and multi-teer distillation all in plain language and finally we get a rare look at how a modern AI research organization actually runs how you get many brilliant

minds to build one model instead of a 100 papers please enjoy this awesome conversation with Brian Katansarao All right, Brian, excited to do this. It

seems that open source is having a banner year. So, you guys at NVIDIA just

banner year. So, you guys at NVIDIA just released Neotron 3 Ultra, which is an important moment and the best open-source openweight model in the US.

That was just a few days ago. And then

uh even more recently, GLM 5.2 came out and that was another moment. So it seems that things are accelerating in open source AI. It feels like a great place

source AI. It feels like a great place to start. What's your assessment about

to start. What's your assessment about where we are and how wide the gap between closed source and open source currently is?

Well, it's really exciting to see all of the energy going into open technologies for AI because we know that um open technologies make it possible for people

to innovate. You know, the internet is

to innovate. You know, the internet is such a great example of that. Um, we

actually did have closed internets. I

don't know if you remember things like America Online and Prodigy back in the day. Um, and they were great. Um, and

day. Um, and they were great. Um, and

open internet has also u been amazing, right? Like so many different companies

right? Like so many different companies have been able to figure out how to transform their work um, thanks to uh, an open technology. The application of the internet to retail is very different

from the application of the internet to healthcare or manufacturing. But all of them have been totally transformed um by the internet. Um AI uh I believe is uh

the internet. Um AI uh I believe is uh also a very transformational technology and also a technology that needs to be applied in very diverse ways and because

of that I believe that open technologies for AI are really fundamental. Um and

it's very exciting to see continued um investment and development of open technologies from uh for AI from so many different organizations around the

world. Um uh and uh you know I I hope

world. Um uh and uh you know I I hope that that continues.

And what's your sense for how far behind opensource is compared to closed sources? It's been the the big trend of

sources? It's been the the big trend of the last few years has been this sort of narrowing gap. Do do you think that open

narrowing gap. Do do you think that open source is almost there or the bar keeps getting raised by the closed source models? Well, I I feel like this

models? Well, I I feel like this question um uh it's maybe a tempting question because, you know, it's fun to set up kind of competition, but but I actually feel like the whole AI

community is moving very fast. Um and if you look, for example, at the progress in AI, whether it's closed or open just over the past 3 months, it's been incredible. Um, and so if you're in a

incredible. Um, and so if you're in a field that's moving really, really fast, I think that's more important than any particular gaps that might exist between different models because the most

important thing is, you know, how is AI developing as a field?

What do you think the drivers are to continue progress in open-source uh AI?

Is that the community? Is that big companies like Nvidia being behind it?

Is that the global competition with China? what propels open source AI

China? what propels open source AI forward?

You know, I think there's a number of things that that are pushing open technologies for AI forward. One is just the demand. You know, there's so many

the demand. You know, there's so many organizations that want to customize AI and want to integrate it deeply into their work in a way that really requires

open technologies for AI. And so, so I think um the demand is certainly there.

I think also it's just um uh the best way to develop technology. Um, and we've seen this, you know, for for many decades that technologies developed in the open move quicker because we can all

learn from each other. And um, in an era where we're undergoing the most exciting thing to happen in technology in our lifetimes with the development and the deployment of AI, um, what else do do

computer scientists want to work on other than making AI awesome? And if

working together as a community is the best way to do that, then that's also a driver that pushes the community towards openly developing technology. to ask

maybe a slightly cynical question. there

is at least a a a part of the community that's wondering whether open-source as an ecosystem not not Nvidia but in

general has been progressing in part based on the ability to distill closed source models and in a world where we

seeing the anthropics and fable fives of of the world starting to discourage distillation do you think there is a chance that opensource AI progress may

slow down in that context or as a result.

You know, in my mind, there's no question that when the um technology community decides to make huge investments in the most transformational technology of our time, that there's

going to be rapid progress. Um and also that that technology is not going to be controlled by a small group of people.

Um because that's just not the way that um the industry works. you know, we we um do our best work. Um uh we have the most impact with our work when we're

able to uh each think about it in our own way and apply it in our own way. So

um you know uh I love uh the uh closed AI APIs uh whether from anthropic or other people. I think they're amazing

other people. I think they're amazing you know um really really impressed with the work that those labs are doing. But

they're not the only labs in the world.

There's lots of labs around the world and lots of people have a good idea. Um,

it's not the case that there's only a few labs that have the monopoly on all good ideas. That's just not true. That's

good ideas. That's just not true. That's

not how humanity operates. There's a

there's a lot of bright people on this planet. And um, you know, the community

planet. And um, you know, the community uh, of course cares deeply about this technology. It's obviously so

technology. It's obviously so transformational, has such profound impacts on so many things um that that of course uh, many people uh, want to be

involved in that. And um so I think over time we're going to see that um community oriented approaches to developing and deploying AI are going to continue to strengthen and be widely

adopted because that's really the history of how we built things as a as a human uh species.

Do you think that is globally true as well? So um you know in particular with

well? So um you know in particular with respect to China this perception that yes a lot of people have great ideas around the world. Uh however a lot of

progress from Chinese models were directly inspired or perhaps uh generated through distillation from from the closed source models. Is that just a

kind of like press rage bait or from the perspective of a leading AI researcher?

You're very impressed by the the the novel ideas that come out of China as well.

You know, um uh perhaps unusually, I uh actually did work at a Chinese company for about two and a half years. I worked

at BYU. Um I worked in the Silicon Valley AI lab uh along with Andrew Ing and as well as Daario Amadei and um uh

we all worked uh for a Chinese company and saw how smart, hardworking, creative, inventive our colleagues were

uh at the rest of BYU and you know that experience has has stuck with me. Um I

think it's absolutely false to say that um you know uh the achievements of of some other country are all being um created by sort of you know copycat

mentality. It's just not it's just not

mentality. It's just not it's just not true. Um now do we all learn from each

true. Um now do we all learn from each other uh in the technology community? Of

course you know of course of course we learn from each other. But um you know uh I I would say uh you know it's been a really good thing for the world that the

Chinese uh AI community has been so um open with what they've been building. I

think it's enabled a tremendous number of companies to build things that uh they couldn't have done without um that community and I think it's also spurred um technological progress throughout the

AI uh ecosystem. So, you know, I'm really grateful for um the contributions that um our um colleagues in China have made over the years and um you know, I I

would love to encourage a a spirit of openness amongst AI labs around the world outside of China as well. You

know, I was really excited when uh OpenAI released the GPT OSS models um a while back and then of course Google's been doing great work with Gemma.

Absolutely thrilling to see that. Um,

and you know, we're pushing Neotron along here at Nvidia as well. Um, so I I think there's a there's a chance for um uh the rest of the world to catch up to China uh in the sense that um you know

we can understand the benefits of working together uh as a community to build technologies for AI uh in a way that I think China has frankly been leading.

Great. What is the case for a customer to be using open-source models uh these days? What is your fundamental

days? What is your fundamental advantage?

Every company is built around a secret.

Uh this is a secret that has to do with not just their intellectual property but also their platform. Uh which has to do with how do they interact with problems and customers? um how do they think

and customers? um how do they think about solutions uh to what what their customers need and it is always the case that the value of AI is greater when it

can be more tightly connected with those secrets because you know AI depends on data critically so the more valuable the data that goes in the more valuable the

solution becomes now um every company when it's thinking about how to deploy AI has to think through what are the implications for the core secrets of our

company and um there's a lot of circumstances where um due to trade secrets or or you know trying to think of think through the business model or or even regulatory requirements that you

know there's data that you really have to treat very carefully by law. Um and

it is much better to do that when you are able to think that through and implement it yourself. um thinking about the integration of AI, the way that AI interacts with um customers, um the

guard rails that are put in place, you know, every company um has a specific understanding of its customers and and therefore what what the customer needs.

And um the amazing thing about open technologies for AI is that they allow custom uh customization, right? So

companies can think this through. They

can build things that that really matter for them. And you know, I started out

for them. And you know, I started out this conversation talking about the internet and about how the internet, the deployment of the internet has been done in very different ways for very

different industries. Um, and there's a

different industries. Um, and there's a lot of desire to do that. Um, uh, as we see AI change the way that we work and play throughout the entire economy. Um,

this is really spurring a lot of demand for open technologies for AI.

Great. I'd love to go into a bit of a deep dive into Neotron, but before we do that, maybe a few minutes on your story,

your background. What was your path to

your background. What was your path to where you are today? Um, including the the BYU detour.

So, um, I started work at, uh, Nvidia in 2008. Uh, at the time I was a graduate

2008. Uh, at the time I was a graduate student trying to figure out, um, parallel computing for artificial intelligence. And I thought um Nvidia

intelligence. And I thought um Nvidia had a chance of changing the way computers work AI which was a lonely presumably a lonely quest right in 2008.

Oh, it was it was very chaotic back then. People thought I was crazy and you

then. People thought I was crazy and you know I remember going to ICML in 2008. I

published my first paper training models on the GPU and people asked me why I was there. People said this is not a good

there. People said this is not a good paper for ICML. We're we're we just do fancy math here. And I was like, well, but I think computing actually matters a lot for AI. If we could train bigger

models that had more capacity to learn, we could probably solve more problems. And um they kind of nodded their heads and like, well, I'm not really sure why you're here.

Isn't a GPU a thing for gaming as well, presumably, right? Yeah. There's there's also that,

right? Yeah. There's there's also that, right, which we we continue to um run into that uh that idea. Actually, a GPU is whatever Nvidia says it is. You know,

we make them. So a GPU is is a is a thing that we make in order to accelerate the world's most important computations. Um which in 1995 was

computations. Um which in 1995 was graphics and you know for a long time now it's been AI. So um anyway I started at NVIDIA um I was in the research group

um doing uh strange things about trying to make um uh compilers libraries uh for AI on the GPU. Um that led to um the

creation of uh first copperhead which was a it was a a Python uh embedded language that um compiled to the GPU um which I think foreshadowed a lot of

things in TensorFlow and and PyTorch um and then um uh then that led to the creation of QDNN which was Nvidia's first product for um uh for deep

learning on the GPU. Um and uh I I really enjoyed working on that. Um but I was always wanting to see more uh

firsthand about the applications of AI and at NVIDIA I was mostly working on you know uh libraries and compilers for AI. So I thought um well you know when

AI. So I thought um well you know when Andrew Ing asked me to go uh build the Silicon Valley AI lab with him uh at BYU I thought oh this is a great opportunity

because even back then BU um was very advanced in its application of AI to its core business. And so um uh so that was

core business. And so um uh so that was a a fantastic opportunity for me. The BU

Silicon Valley AI lab was an amazing place um full of brilliant people that were working really hard.

What was it like working with a young Daario? Uh was there like any signs that

Daario? Uh was there like any signs that he could become uh you know who who he has become?

Daario um was brilliant from the beginning. I remember um I interviewed

beginning. I remember um I interviewed him uh I was on the panel um and um at the time he uh had been working in bioinformatics so he he hadn't been

working on deep learning or or the things that we call AI these days um but it was very clear that he learned extremely quickly and also that he

thought extremely deeply um I think you know uh the thing I admire most about Daario uh is the strength of his conviction um you know I've been working

in this field uh for a long time and I've believed also that AI is going to transform the world but I don't think that I believed uh in it as completely

as Daario did and perhaps that was because you know my academic training during my PhD was full of a lot of caution uh I don't know if you remember

but AI was old and bad in 2005 it will never work that people did with computers they started doing it in 1945 Right. Um and

and so there had been so many grandiose promises that failed to deliver over the years. And so I came to AI with a lot of

years. And so I came to AI with a lot of caution. In fact, back then we used to

caution. In fact, back then we used to call it machine learning, which was basically a dodge. Like we just didn't want people to know that we were we were working on AI because then they would be like, "Oh, we've heard about that. It

never works." Right? So, um I came to AI with a little bit of this like uh you know, academic caution like, "Oh, we should you know, we should hedge a little bit. like I don't know if now's

little bit. like I don't know if now's the time like um and and Daario, you know, he his uh strength of conviction and his understanding of the moment of

how um the technology was developing this time it was actually going to work and then the implications of that on um you know how the technology should be

developed um uh what kind of institutions uh to build I think he's done a spectacular job and so um yeah working with him uh it was always always a a fun experience.

So then you went back to Nvidia and walk us through the journey.

Yeah. So uh 10 years ago actually in 2016 uh Jensen called me up and said hey uh would you like to come back and build an applied research lab and I thought that would be a fantastic uh

opportunity. I you know I've always

opportunity. I you know I've always loved Nvidia. I've loved um the way the

loved Nvidia. I've loved um the way the company works, the convictions the company holds. You know um Nvidia is a

company holds. You know um Nvidia is a very unique company. it follows through over long time periods, you know, uh, and I've seen that with CUDA. I've seen

it with our deep learning technologies.

I've seen it with our ray tracing graphics technologies, our AI for graphics, you know, over and over again, Nvidia is not afraid to put in five or 10 years worth of research in order to

change the world, you know, and um, working at a company that has that strength of conviction and the ability to follow through is kind of an ideal thing for me. Um I just really I just really love the support that the company

gives uh gives its researchers um to invent the future. And so um I thought I'd come back um uh the first project that uh that I worked on um actually

became DLSS uh which uh some of your um audience may know about but DLSS is our real-time uh AI for graphics and it makes a small GPU run like a big GPU.

It's about 10 times more efficient because rather than computing uh the color of every pixel for every frame, we use AI to infer the color. Uh and uh you

know these days 23 out of every 24 pixels uh is being generated by our AI model when you're using DLSS um to play games. And gamers love it. It's become

games. And gamers love it. It's become

the standard way of playing games because it's just so much more responsive and it's more beautiful. Our

AI, we train it offline on huge data sets. Um and it's able to render uh

sets. Um and it's able to render uh graphics in real time uh more beautifully than uh traditional methods do. Um uh we recently actually announced

do. Um uh we recently actually announced DLSS 5 which is a fully generative version of DLSS and um I am so excited about it. It represents culmination of

about it. It represents culmination of 10 years worth of research on how to make um real-time graphics much more beautiful. Um and so uh so that's part

beautiful. Um and so uh so that's part of uh part of the journey uh here for me was real- time uh uh AI for graphics. Um

but then at the same time we also started a language modeling project. Um

and this was back in 2017 um you know before transformers uh were big and before um language modeling uh uh started taking over the world. But um

you know I just had this intuition maybe built on you know some of the things that that I had seen uh while working uh at BYU. I I just had this intuition that

at BYU. I I just had this intuition that you know working with text and understanding text was going to lead to better reasoning which was going to lead to better application of AI in all sorts of domains. And so um uh so we started

of domains. And so um uh so we started this project uh called Megatron.

Megatron stands for the biggest baddest transformer. That's why we named it

transformer. That's why we named it that. Um, and it was really a systems

that. Um, and it was really a systems project to show the world how to train the largest transformer models on Nvidia's hardware. Uh, back at the time,

Nvidia's hardware. Uh, back at the time, uh, some of your, uh, audience may or may not remember this, but, uh, there was there were being claims made that the only way to train big transformer

models was on the TPU. Um, because after all, the transformer had been invented at Google. And, uh, so, you know, we

at Google. And, uh, so, you know, we looked we looked at uh, you know, we loved the the the transformer paper. We

thought, wow, this has amazing potential. We tried it out on our own

potential. We tried it out on our own language modeling tasks and it worked so much better than the RNN's that we had been using before. And also we saw immediately that there was an enormous systems opportunity to co-optimize the

GPU, the networking, all of the compilers and software that would enable people to scale uh transformer-based language models uh really dramatically.

And we we thought, you know, this is this is something that that um could really have an impact. So we started the Megatron project um which then led to I think uh basically helping the whole

industry figure out how to train um extremely large uh LLMs um and also led to the foundations of today's Neotron project um where you know Nvidia trains

uh its own LLMs um uh for its own purposes. So that's kind of the the

purposes. So that's kind of the the history great journey. Okay, so let's go into

great journey. Okay, so let's go into all things uh Neatron. And before we get into the specifics, that's the obvious question that I'm sure you've been asked

uh many times uh which is why does Nvidia care in the first place to be building model and investing very significant efforts into creating its

own family of of frontier models.

You know, Neimotron has two jobs. The

first job is to help us understand how to build the systems of the future.

NVIDIA is an accelerated computing company and that means thinking through the world's most important computational challenges from first principles and designing systems which includes a lot

of software uh in order to make it possible for people to invent and deploy things that never could have been done with standard computing. But in order to do that, Nvidia has to deeply understand

everything about how AI works. That's

how we co-design all of the systems and software um uh for our main product line. So the first job of Neotron is to

line. So the first job of Neotron is to make sure that Nvidia continues to exist so that we can continue delivering meaningful acceleration in an era where Moore's law has died. uh and the the

acceleration that we get these days comes through specialization but again specialization comes through understanding. So that's Neatron's first

understanding. So that's Neatron's first job is to help Nvidia understand how to build its core products. Nvidia's second

or Neatron's second job is to support the ecosystem. Uh, one of the most

the ecosystem. Uh, one of the most valuable things that Nvidia has built over the years is um, all of the people around the world who build and deploy

amazing AI um, using Nvidia's uh, technologies and uh, we think that it's necessary for uh, open technology for AI

to continue to exist uh, from Nvidia to help uh, support that. Neotron's not

trying to be the only open technology for AI. We love all technology for AI

for AI. We love all technology for AI for the very straightforward reason that whenever AI uh is further developed and further deployed, it's an opportunity

for our business. So, so this is this is um you know, we're we're very explicitly trying to develop our ecosystem because that's good business for us. Um but

we're not trying to be the only provider of technologies for this ecosystem. We

love seeing um other companies contribute as well. Um the the most important thing for Neimatron's second job is just making sure that it continues to be possible for companies of all shapes and sizes to build and

deploy their own AI.

By the way, Moors law is dead. Is that

is that is that official?

It's been dead for years.

It's been dead for years. Like why why is that?

Well, you just look at the the progress uh in semiconductor manufacturing. You

know, the the original statement of Moore's law was economic, right? It was

about we can afford to put twice as many transistors on the same chip in every whatever 24 months, whatever the the time period is. And um these days that

is absolutely not the case and it hasn't been for probably 5 or 10 years right now. We are still scaling our systems

now. We are still scaling our systems right um through a number of ways. One

is just applying a lot more silicon to it, right? uh we are also getting

it, right? uh we are also getting transistors are continuing to get smaller and and more efficient although at a slower pace but they're also getting quite a bit more expensive at the same time.

Um uh so the uh you know in an era where where Moore's law was alive, the best way to make the system of the future was to take the system of the present and then just shrink it and and maybe double

it at the same time, right? But in an era where where we've been living for a while now where you don't get economic benefits from taking your existing design and shrinking it, uh you really

have to be more clever about how you use every part of the system. uh that that's uh you know an era where accelerated computing is is much more valuable than

ever because the the work of thinking through the pro problem from first principles and co-designing absolutely everything uh from transistors to

algorithms and applications uh in order to reduce waste and and deliver meaningful acceleration that's more valuable than ever.

Fantastic. to play back what you were saying uh a minute earlier. It makes

good business sense for Nvidia to be in the model business because one it helps design better chips and two uh whatever is good for AI is ultimately good for

Nvidia which makes a lot of sense. That

Neotron effort is reasonably recent, right? It started in 2023, I believe.

right? It started in 2023, I believe.

Maybe walk us quickly through the key releases. I believe in 2023 there was

releases. I believe in 2023 there was Neatron 38B as a key release or am I missing a step?

Yes. Yes. Yes. So, you know, the the you know, the the numbering is somewhat lost to time. It I almost feel like we're in

to time. It I almost feel like we're in the Lord of the Rings and it's like, you know, there's like some ancient like relics that we're digging up out of an old mine. Um, you know, this is a long

old mine. Um, you know, this is a long time ago. You know, the original what

time ago. You know, the original what what uh we originally called Neotron 1 was actually a project that we did with Microsoft. Um, we jointly trained a 530

Microsoft. Um, we jointly trained a 530 billion parameter model. Um, I believe that was released in 2021. And so this

is GPT3 era. Um, and that's what um, at at the time we called it Megatron Touring NLG. uh touring was uh what

Touring NLG. uh touring was uh what Microsoft was calling their their language model efforts um at the time.

But uh uh that uh in retrospect we called Neotron 1. Uh then along the way we built a few more uh we got up to

Neotron 3. Um and then uh Llama came

Neotron 3. Um and then uh Llama came along uh and we were really excited about that. We were very um happy that

about that. We were very um happy that Meta was supporting the open uh AI technology space. Um and so we we

technology space. Um and so we we started um you know taking our language model technology and adding it to llama models which then resulted in llama

neatron 1. Um and you know that was the

neatron 1. Um and you know that was the first uh reasoning model uh built on llama. Uh we were really proud of that

llama. Uh we were really proud of that and that was 2025.

Uh might have been 24 uh I I believe I can't remember uh somewhere around there. Um and then uh uh yes and then we

there. Um and then uh uh yes and then we we continued uh to to develop that um uh and uh you know last we we so we the

numbers kind of started over again. We

released a a Neotron 2 um I believe it was last year. Um and then we quickly followed that up with Neatron 3 because um

uh we we needed to put support in.

Neatron 2 didn't have support and that made it kind of uncompetitive against uh other models like GPT OSS 20B was just like so fast because of so we were like okay we've got to we've got to put put

thee in. So that became Neatron 3. Um

thee in. So that became Neatron 3. Um

now we're in a a slightly difficult state because you know we're working on Neotron 4 right but we already released a Neotron 4 which was um in 2024 we released a 340B

uh model called Neotron 4. Um, and so I'm not exactly sure how we're going to um solve this marketing problem. I

didn't create this marketing problem.

Uh, so uh I'll I'll I'll do my best to to make it clear that Neotron 4 of of of whatever the NE whenever we release that is different from the the 2024 Neotron

4. Um, but uh in any case, we've been

4. Um, but uh in any case, we've been working on this for a long time. I I

think um more important to us than any particular generation is just the sustained commitment that Nvidia has to developing these models. We've been

doing it for a while. I think our models have gotten dramatically more useful in the past year. Um which is a reflection of two things primarily. One is that uh the whole company has come together. So

there are many different teams around Nvidia that now understand how important this is to Nvidia's future. And so

there's dramatically more people and better ideas that are going into Neotron. Um and then uh number two along

Neotron. Um and then uh number two along with that we've been able to scale the um compute resources that go into it. Um

obviously it's um very important to have good computing infrastructure to build AI. We've um recently increased our

AI. We've um recently increased our investment substantially because uh we believe that that this is really really key to our company's future.

Fascinating. But I just to continue the thought I think it's really important that everybody knows that we've been doing this for a long time. We are

increasing our investments substantially and Nvidia is a company that follows through. You know we followed through

through. You know we followed through over 10 plus years with CUDA and we're doing that with Neotron. Now

that's very helpful because uh I think the the broader world is is just starting to catch up to the fact that there is a very substantial open-source

frontier AI research effort that that's been happening. So it's really

been happening. So it's really interesting to to hear that uh you know there's been this progression and now there's this family of models that we're going to talk about uh in a second.

Another important moment seems to be uh the creation just in March 3 months ago of the Neotron coalition. Do you want to explain briefly what that is? So,

Neotron exists to help support the ecosystem and we were thinking, well, this is a different kind of AI project than other projects around the industry,

right? Because um we're not actually

right? Because um we're not actually trying to dominate in any way. We're

just trying to support we don't we're not trying to control uh uh the way that AI is um uh being integrated into all these companies. We're just trying to

these companies. We're just trying to make sure there's good AI. But we

thought well maybe if we worked with people while we develop it then it's going to be more useful for them. It'll

be easier to integrate because we will consider what they need from the beginning. And um you know Neotron has

beginning. And um you know Neotron has always been collaborative. I was telling you that you know long long time ago our first big model that we trained we we did with Microsoft right? It was a joint effort where Nvidia and Microsoft

researchers worked side by side to build that and that um that ended up I think helping both Nvidia and Microsoft. I

think we both learned a lot from that um experience. And

experience. And so because Neotron is not trying to compete with uh other companies, but rather support, uh because we're going to be putting it out there openly

anyway, why not collaborate before the thing is built? Rather than Neotron being a project that Nvidia does all on its own and then posts on the internet and says, "Hey, why don't you try this?

We think it might be good. Why don't we make sure that it's good for the partners that um uh that are interested by working with them before Neotatron is even created um and incorporating you

know any sort of feedback evaluations environments um uh benchmarks um or any other um kinds of technology that other people want to bring. It turns out that

the entire ecosystem there's a lot of companies that really want open models to succeed and so they have a self-interest. they have their own

self-interest. they have their own vested self-interest in making sure that open technologies are excellent and so why not um work with them and and let them contribute uh however they'd like

to making Neotron better. So that's the the idea of the Neotron coalition. It is

not an exclusive coalition. We're not

trying to be the only model out there.

All the companies that we work with are free to to continue doing the work however makes sense to them. And yet um you know these companies want to work with us because they want to make sure that um open technologies for AI keep um

developing quickly and that they have a chance to influence how that happens.

Great. What's the current state of the Neimontron family? You got nano, you got

Neimontron family? You got nano, you got super, you got ultra. What do those models do and what are the use cases for them?

So nano is a 30 billion um uh total 3 billion active parameter model. supers

120 and 12 and ultra is 550 and 55. Um

they're designed um really to fit, you know, it's kind of small, medium, and large um uh deployment scenarios. Um uh

you know, Nano can be really capable for things that um you know don't require nearly as much knowledge or reasoning, but obviously for the for the most um

capable model, you go for ultra. Um,

super in a lot of ways is our most popular model because it represents kind of a great balance between um cost and and intelligence. So we we kind of like

and intelligence. So we we kind of like um having this small, medium, and large um approach to building a family just because our customers um seem to respond

to that um pretty well. But um you know uh the most important thing from Nvidia's point of view that people are

doing with uh uh with LLMs is agents right is um building agentic workflows having a having an agent working on your behalf solving problems for you night

and day. Um such an exciting way of

and day. Um such an exciting way of approaching the problems that we have to solve. Um and um it's our dream to make

solve. Um and um it's our dream to make Neotron amazing for that purpose. That's

that's our goal to double click on this at a high level.

Neotron family is focused on agentic reasoning with a particular focus on making it efficient. Is that is that the right headline?

That's right. Yeah. Um Neotron has always been um uh speed first approach to building models because NVIDIA is an accelerated computing company. As I was saying, we're trying to think through

what is the problem here computationally from first principles and um you know Neotron uh 3 family has a lot of things in it that are uh we're really proud of.

For example, um Neotron Ultra and Super uh were pre-trained using 4bit arithmetic. We pre-trained those in

arithmetic. We pre-trained those in MVFP4 um which you know uh is a not trivial thing to do to invent the algorithm so that your model can converge to an

excellent result using such coarse arithmetic uh required a lot of invention. Really proud of that. Do you

invention. Really proud of that. Do you

want to explain maybe for for people what 4bit is versus 16 bit for example?

You know, actually there was a fantastic post I saw in Hacker News yesterday where somebody let you um upload a picture and then it would basically posterize it, basically reduce the

colors to fit different um uh number formats including NVFP4 and MXFP8 and some of the other formats that are out there. And so you could kind of swipe

there. And so you could kind of swipe around and look what it does to the colors of a picture. Um and you know it's it's really quite dramatic. Four

bits is not a lot of bits, right? That's

only 16 values. Um now, of course, these are all um what are called block scaled formats. So, um groups of numbers also

formats. So, um groups of numbers also come with uh an 8bit um scaling factor.

And the the specifics of this can get rather complicated. So, maybe they're

rather complicated. So, maybe they're not quite as important. Uh but the the reason why we want to do this is because first of all we have dramatically higher

um throughput for these formats in our GPUs um specifically on Blackwell Ultra.

Um and uh uh secondly, we know that it's going to save an enormous amount of energy. Um one one way to think about uh

energy. Um one one way to think about uh the computational problem of AI is that we are going to be running at the limit.

Whatever the limit is, it could be uh an economic limit like we only have so many billion dollars to to buy servers with.

It could be a power limit. We only have so many gigawatts that we can afford to to train a model with. Whatever the

limit is, um uh we're going to be running at that limit. The or every organization is is because why? Because

the value of intelligence is so high, you know, that that people are going to they're going to invest because they they know that they're going to get return. um uh the value of intelligence

return. um uh the value of intelligence is is enormous. Um so if you if you accept as the truth that we're going to be running at the limit, then what that means is that the way to get more

intelligence is to be more efficient. We

can't get more intelligence by applying more force if we're already at the limit. We have to be more thoughtful

limit. We have to be more thoughtful about how we use what we have. And you

know, 4-bit number formats are dramatically cheaper to move around.

They take up less space in memory. um

they take up less uh pico jewels when you move them from the memory um in or even on the chip uh around the chip much

less uh energy when you compute on them and so uh so that's really um driving you know the investment in 4-bit formats and I think these days 4-bit formats for

uh deployment are very well established um it's it's pretty pretty um straightforward these days to make a a good quantized uh four-bit uh checkpoint that you can deploy and that gets you a

lot of inference cost and speed advantages. Um but using 4-bit formats

advantages. Um but using 4-bit formats for pre-training uh that's quite a bit uh more challenging because uh you have this um numeric uh solver that's you

know optimizing the weights and um you know it it can be quite sensitive. So if

you if you don't uh treat the numbers right uh your model can diverge uh and instead of actually getting a model done through pre-training you end up with you know basically just uh that run diverged

which is you know always always scary.

So it it took a lot of invention for us uh to be able to pre-train Neotron uh in 4bit. We're really proud of that.

4bit. We're really proud of that.

Okay great. All right. So, as we uh get into slightly more technical things, the uh architecture of Neotron uh is hybrid.

Is that is that right? So, it's a combination of transformer and member state space which is a slightly more exotic uh form of of of architecture.

Walk us through that. Yeah, you know, we published a paper in 2024 that showed that you actually get a smarter model by combining um state space models with

transformers. And we we actually did a

transformers. And we we actually did a sweep of uh you know, how much of the model should be full attention and how much of it should be um a state space model in order to get the the lowest

perplexity, basically the best the best language model um that you could get.

And we found that you actually want it to be mostly a state-based model with a little bit of attention. And so kind of the intuition behind that is that um the

state space models seem to be better at um kind of this intuit um intuitive kind of impressionistic understanding of a sequence um uh because they're um you know they're kind

of summarizing the entire sequence into a constant space. That's how they work, right? So instead of having the ability

right? So instead of having the ability to look at the entire sequence randomly, uh they summarize everything at every step into a constant um cache or little scratch pad that they're they're working

on. And um that constraint seems to

on. And um that constraint seems to actually make them smarter at some tasks that involve like global understanding.

Um on the other hand, the advantage of full attention is that it can pick out very specific uh bits of information and look at those exactly. It doesn't lose anything. There's no lossy compression

anything. There's no lossy compression going on. and you can actually see the

going on. and you can actually see the whole thing. Um, and so we found that

whole thing. Um, and so we found that um, uh, you know, using both of these together was actually better than using either one on their own. Um, and that is

independent of the speed benefit. That

is just the model is smarter. And you

know, since we published that, I think a lot of other um, uh, labs have also found this to be true. Um, you know, a lot of uh, uh, models these days are

being built with hybrid SSM approaches.

For example, QN has done that. Um, Kimmy

is using what they call Kimmy linear attention these days. So, um, it's become, I think, quite widely adopted to use some sort of state-based model in

conjunction with um, full attention for um, uh, for the the base architecture.

Um now it also has some speed benefits because um the uh uh amount of memory that you need to hold uh that state

space uh cache is actually constant with respect to your sequence length. um

which then means that um generally you can fit much higher batches on the GPU when you're training and doing inference uh because the memory um requirement is

lower and it keeps the GPU uh fuller and busier um and therefore um you know provides some some pretty important signific uh some some pretty important efficiency benefits as well.

So the models are also based on ane mixture of expert architecture. walk us

through that and maybe remind people what is in the first place.

So mixture of experts is a form of sparsity. Um the idea is wow, you want

sparsity. Um the idea is wow, you want to train a model on the entire internet.

You want it to remember absolutely everything about the history of everything. But when you're answering a

everything. But when you're answering a particular question, does it seem reasonable that it needs to actually think about the entire universe in order to answer that question? Actually, no.

It seems like it's quite sparse, right?

It seems like um we're we're we're using a language model to explore a very tiny space of ideas in order to answer a question or solve a problem. We want the model to be able to draw from the entire

universe. We want to train it so that it

universe. We want to train it so that it understands everything that it possibly can. But when it's actually running, it

can. But when it's actually running, it doesn't really need to see all of that information. There's been a variety of

information. There's been a variety of approaches to sparity that try to take advantage of this property, but mixture of experts has been the most successful.

And the way that it works is that the neural network has what's called a router that is learned that um is going to decide to send activations to a

subset of the experts uh for every token that's flowing through every layer of the model. It's going to be making

the model. It's going to be making choices about um which fraction of the model is going to actually get to interact with this token as we try to understand it, build up representations of the problem, and then generate the

next token that we're going to output.

So it's a little bit like if I have a company with 550 employees but uh 55 of them are in engineering I want the 55

employees who are specialists to come to my meeting about engineering and not the rest of the company.

That's right. Yeah. Or you could think about it as a library. Like if you go into a library to do research, you don't read all of the books in the library.

Like your first job is to figure out which books do you need to look at in order to find the answer to your question. And so um so that's kind of

question. And so um so that's kind of the the idea behind. Now have

fascinating implications for the systems that we build. So with Blackwell for example, Nvidia went all in ones. That's

why we built NVL72 which allows up to 72 of our GPUs to read and write each other's memory uh at very high speeds, very low latency. Now

why is that important? It's because as you put a token through the stack of layers, at every layer, you have a router that's routing that token somewhere else. Why don't you partition

somewhere else. Why don't you partition your experts so that you know the experts are not sitting every expert on every GPU, but you have a subset of the experts assigned to each GPU and then

you're routing the tokens between the GPUs very dynamically as you push uh the token through the network. Now um this is impossible to predict in advance

where the tokens need to go because it's very specific to that particular token for that particular model. And so that's why we built NVL72 and that's why um Blackwell is so

amazing for inference for you know today's AI models uh is because we thought deeply about mixture of experts when we were building it and this is speaking to Neatron's first job. you

know, if if we hadn't been working on understanding AI, we wouldn't have been able to build Blackwell properly. And

that, you know, has has translated directly into um you know, increased deployment um of Blackwell, which you know, we're we're we're very excited about.

Is what you just described called latente or is that a different concept?

Latent MOE is a specific uh innovation that we have in Neotron 3 family and uh what it does is actually um reduces the amount of communication that has to be sent through envy link during

computations by basically down projecting it. So you know every token

projecting it. So you know every token is it produces a vector and the idea is like we're going to take that vector and learn a way to compress it and then send that compressed thing through the network and then we're going to

uncompress it at the other end and as a result we save on network bandwidth and we also get four times the number of experts for the same inference cost. So

you could think about it as like you know our library of books got four times bigger um and we get to you know read four times more books uh at the same inference cost because of because of

this particular innovation is in general becoming the default architecture for Frontier AI.

Yeah I believe have been the default um in Frontier AI for a long time. Um

they're just a really good combination of inference cost and intelligence.

Great. Great. But they have drawbacks as well. You know, they they take a lot

well. You know, they they take a lot more memory. Um uh if you have a very

more memory. Um uh if you have a very small amount of memory, a dense model is going to be smarter. Um and they also um they tend to work best either if you're running at batch size one. So you're

running basically a single job or you're running a huge data center with like infinite queries coming in in the middle. They can be a little bit tricky.

middle. They can be a little bit tricky.

Another important characteristic of Neimatron 3 Ultra is the 1 million token context.

the the long context window. How

important is that in the overall mix and what does it enable the model to do?

The longer the context length, the more challenging problems we can solve with a language model. Um, that allows us to do

language model. Um, that allows us to do things like append all sorts of information to a query, which could be a codebase, it could be instructions. Um,

you know, uh, in in the long term, I'm hoping that I have my own personal LLM that's able to read all of my emails, you know, and help me answer questions about that. You know, the more

about that. You know, the more information that we can attach to a particular query, um, the the more useful the model can be. Um, now, uh, it

can get more and more expensive, right, to reason over large amounts of of input data. And um and so that's one of the

data. And um and so that's one of the the reasons why there's usually a limit on how big the context length can be.

But with Neotron 3, we we tried to push it as far as we could go. Uh we think a million tokens is a lot of tokens. Um

and you can do a lot of things with that.

Prisma is particularly helpful in sort of multi-step agentic workflows and there's this whole separate discussion around context compaction to make sure

that the model doesn't get lost in too many tokens. So like how do you how do

many tokens. So like how do you how do you all think about this?

100%. I mean um uh you know compaction is that's a thing if you're using an agentic workflow you deal with all the time. Um and compaction tends to work

time. Um and compaction tends to work pretty well you know because language models um are pretty good at at identifying the most relevant things and summarizing and you're basically trying

to summarize your context when you compact it. Um so uh compaction uh is is

compact it. Um so uh compaction uh is is not a bad approach. I think having models that can just natively uh reason about larger amounts of data is just inherently more useful. So of course we

want to push the the boundary on that as well.

Great. Can you talk about the multi-token prediction uh which is also very interesting if you're running at a low batch size um which is when you are trying to get the

most interactivity if you're in a data center. So you want you want the model

center. So you want you want the model to respond as quickly as possible and it's okay for it to be more expensive.

your token your cost per token might might be higher but you want the result as quickly as possible or if you're running um locally um so you might be running at batch size one just because

you're the only person using it. It

turns out that uh the GPU has extra execution capabilities that are just lying there unused. The bulk of the work when you're running in these scenarios is actually fetching the weights from

memory. And then you push the token past

memory. And then you push the token past those weights and then you fetch more more uh weights from memory. But it

turns out if you if you push two tokens or even five tokens through those same um weights, it would cost basically the same amount of time because the the expensive thing is not doing the math to

push the token through the weights. The

expensive thing is just reading all of those weights from memory. all those

parameters they have to come in. And so

the idea with multi-token prediction is to take advantage of this by having the model predict multiple tokens at once.

Let's say that the model predicts five tokens. We know the first token is

tokens. We know the first token is correct. The next four tokens may or may

correct. The next four tokens may or may not be correct. So then what we do is on the next pass we take those four tokens and we stick them into the model uh and

then uh run it through and at the end we check you know the model then predicts another set of tokens right then we check were the extra tokens we predict last time correct. If so then we just

accept them uh and then we get like a 4x speed up. um and if they were incorrect

speed up. um and if they were incorrect then we only accept the ones that were correct and then um you know uh uh proceed from there. So the benefit of

this is it it doesn't degrade accuracy at all because you're using the model to doublech check right. So all this speculation is going to get checked uh during the next token that you you uh

run through the model. So it doesn't degrade your accuracy at all to turn on multi-token prediction, but it can give you a speed up and it's probabilistic depending on the acceptance rate of your um predictor, you know. So if your

predictor is more accurate, the acceptance rate goes higher, you get a higher speed up. Um so with, you know, um uh with our recent Neotron models, you know, we're pretty proud of our acceptance rates, but we're always

trying to make them better. You know,

always trying to improve that acceptance rate. This is a really good example of

rate. This is a really good example of accelerated computing. You know, with

accelerated computing. You know, with multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model

your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this case, that's how it works. And what that implies is that

it works. And what that implies is that you know if we're trying as Nvidia as a company to provide meaningful acceleration to the world's most important computational workloads this has to be an important part of how we uh

think about it. you know, if there's a 3x cost reduction or speed improvement uh for inference, which is the most important computational workload of 2026, um if that's on the table and it

depends on the accuracy of the multi-token prediction network, then that's something that Nvidia needs to understand very deeply because it's going to affect uh our business directly.

Fascinating. To to continue on the on on the tour, multi-teer distillation, we talked about distillation a little bit up front. uh what does that mean in the

up front. uh what does that mean in the context of uh Neotron 3? So with Neotron 3 ultra we did postraining using something called mo multi-dommain on

policy distillation and what that uh entails is that you know we have many different aspects of the model we want to improve um for

example science understanding is different from math theorem proving which is different from coding which is different from agent harness uh

interactions right there's there's um withotron 3 I think we had about um 10 10 or 15 of these teachers and um so the idea is that you take these teacher

models and you push them as far as you can go on some specific domain. So you

just don't worry about making it good at everything just make it really really smart at this one domain. Then you have a collection of these models and you want to create one model that learns uh

to be good at everything. And we do that using a specific reinforcement learning technique that a lot of labs these days use called um uh MOPD. Um and the the good thing about this is that because

the teachers are supervising uh they can give really dense uh rewards to the the student model. Basically every token is

student model. Basically every token is getting supervised and so the student can learn really quickly um and then become uh you know almost as good as all

of the teachers at all of the things. Um

so uh one benefit of this is that it really helps the team work together better. um you know if if if um you

better. um you know if if if um you don't have a technique like this and you have let's say 500 people working to try to make a model better and one team's like well I'm trying to make it better at this thing and then another team's

like I'm trying to make it better at that thing there can be a tugof-war where it's like well who wins you know and and if you have to make a choice like oh I'm going to make them I'm going to choose to prioritize this one over that one then you make the other team

feel like their work doesn't matter you know it's just really hard one of the challenges of of building AI in in 2026 is that you have to figure out how to get the people to work together even though you're only building one thing at

the end of the day. And so this particular technology um has been really instrumental in helping more people work together to make Neotron stronger.

Fascinating. So it's just as much a technology question as a human organization question.

Exactly.

Okay. Fantastic. Let's put a pin in this and and and get back to to this in a second because it's a fascinating topic in terms of uh the post training uh that

that you just alluded to. One of the uh exciting things that you all did uh in the context of Neotron is also to

publish the data the training data. Does

that include per industry data for specific reinforcement learning tasks?

Yes, that's the beauty of a conversation like like like this today where where you guys can actually talk about those those things. So where does the get one get

things. So where does the get one get the data from for uh post- trainining reinforcement learning focused um efforts right obviously one of the key

questions in in the world today is that LLMs or or or AI systems have become great at coding and great at math like the next big question is can they become

great at uh law and consulting and then uh you know all sort of different domains and part of the the black box of closed models is like how people go about doing all all of this? Where did

they get the data from? To the extent that you can uh talk about all of this, I'd be very curious about how you guys have have gone about it.

It's not an easy question to answer because it is quite complex. But I would say um we rely on uh a number of things.

One is that we do purchase data from uh companies that um uh that you know are are building data sets that you can purchase. Um and to the extent that you

purchase. Um and to the extent that you know we have the rights to redistribute or to to to open up that data we do as part of um our our Neotron data effort.

Um, you know, with with Neotron, we are trying to be maximally open with the data that we release because our goal is to support the ecosystem, right? Our

goal is is not to be the only model out there. And we love it when we hear of

there. And we love it when we hear of other models around the industry that are using our data sets um to to make their AI stronger because that means we're succeeding at our job to keep the

ecosystem thriving and growing. Um now

uh we also are big believers in synthetic data generation. Um we use an enormous amount of compute um uh running

language models on our own systems to create synthetic data that then helps our models be better at uh solving problems in specific domains and we

release a lot of that data as well. Now

it's of course not very straightforward to do this like you know AI is always garbage in garbage out. So uh you have to work really hard to make sure that any synthetic data that you create is

actually adding value that's actually helping the model generalize and solve problems uh more intelligently. Um but

those are the the primary ways that we go about um building our data sets.

Since we're talking about post training and RL in different domains, just curious to get your thoughts on where we go from here in terms of generalization.

So just to you know build on what I was saying a second ago like the industry seems to be marching from coding and math which are domains with verifiable rewards to to different uh industries.

Do do you think that this is where things are going and that the industry as a as a whole is going to be able to cover those next few domains as

efficiently as coding or math? Coding is

really special because um it's a very intellectual exercise that created uh a lot of economic value which then meant that we had an enormous amount of tokens

that we could um learn from as well as uh tooling that allows us to verify whether um you know our our models are actually solving problems. Um so coding

um uh uh is always going to have a special place in our heart um and and something that I think AI is going to continue to get much better at um because we we have this uh special

relationship with it.

Um you know with regards to other domains I think what I'm excited about has to do with significantly more

diverse environments for AI to learn in during reinforcement learning. Um, I

believe that, um, you know, I mean, reinforcement learning is is such a general form of, um, teaching an AI how to solve problems. Uh, we're just

getting started at figuring out how to apply that. Um, and I think as our

apply that. Um, and I think as our environments get more sophisticated, the AI then learns more understanding of um

uh the problems that it's trying to solve as well as the implications of the actions that it can take. Um, then it becomes much better at uh at actually um solving those problems. When I look at

the the uh environments that we're using today, they're still uh fairly simple um all things considered. And I think um uh that's going to become significantly more complex and diverse over the next

few years.

All right. So you mentioned uh you know making 500 people work together and I I said we would get back to it because it's so interesting. So just uh taking a step back like tell us about the

research organization at uh Nvidia like how is it structured? How how does it all work?

Well uh Nvidia is not structured according to an org chart. um we have one but it's not actually the best way of understanding how we work. Um uh my

team for example is not part of the official NVIDIA research team. My team

is actually part of the organization that builds the GPU and my team is not the only team building Neotron. There's probably 10

building Neotron. There's probably 10 teams around the company that have significant involvement in building emotron um in different parts of the

company in in enterprise software um in uh uh our AI software um uh division the part of uh Nvidia that actually designs

the GPU also significantly is involved in in building Neotron. Um so there's there's so many different teams that that have to work together. Um we um

always like to say that the mission is the boss um rather than uh the organization. But um what that implies

organization. But um what that implies is uh that people have to figure out how to work together um which is challenging um in the sense that humans are

naturally tribal creatures and it's uh not natural for us to um be friendly with people we don't know very well or trust uh co-workers that we don't have you know success working with in the

past.

And um you know uh actually the name Neimotron uh reflects that. We had um the Nemo team which was building software for AI and the Megatron team

which was building uh primarily focused on systems research for um uh for building large language models and um you know we decided to work together and then start calling our projects

Neimotron reflecting you know sort of the um uh the collaboration between these teams. Since then, Neimocron has has dramatically expanded. There's so

many more teams that are part of the effort. Um, and it's really important um

effort. Um, and it's really important um that we have structured it uh in this open way inside of Nvidia. Um, you know, we are inviting uh volunteers from

around the company to come help build NVIDIA's AI. Uh, we think it's very

NVIDIA's AI. Uh, we think it's very important to the future of the company and, you know, as that vision continues to develop, more and more people want to join. That's fantastic. We're really

join. That's fantastic. We're really

excited about that and it means that we then have to figure out uh how to organize the work so that everybody has a chance to contribute and feel heard and feel like their ideas um are are uh

you know fairly evaluated um on the on the path towards impact. Um uh we we have a a formal process for doing that.

We have an internal website where people um share ideas and then those ideas um are assigned to um one of 25 different um leads that are uh you know over

various parts of building Neotron. Um

they interact with those ideas. Some of

those ideas get further developed. Um

some of those ideas get deferred until you know the next uh the next time we we go around building a new model.

Um but we're trying to build Neotron in an open and inclusive way um uh so that u you know we can really come together as a company to build it. I think um you know organizations that figure out how

to collaborate to build AI succeed.

Organizations that struggle with control over who owns the AI tend to uh waste a lot of effort. Um and so Nvidia's success and Nematron success I think is

directly proportional to our ability to collaborate something that I care deeply about.

Fantastic. But you you mentioned uh earlier that um despite the fact that you work at the you know number one undisputed leader in in GPUs you all as

a research organization don't have all the GPUs that that you would want in the world. So like how does the allocation

world. So like how does the allocation of GPUs and compute uh happen? And is

that based on how promising an idea is or early success? Do you give GPUs, withdraw GPUs based on success?

It's a really complicated question and it's it's obviously a a difficult problem for everyone in the industry to figure out how to allocate their their

compute. Um uh inside Neotron uh you

compute. Um uh inside Neotron uh you know so we we have a budget uh for Neotron. Um and inside Neotron we

Neotron. Um and inside Neotron we allocate compute based on what we think the needs of the project are. We have a um a hierarchy. So we had a we have a

set of programs and and in inside of each program we had have a set of projects and each of them put forward their requests. Um and then you know we

their requests. Um and then you know we have a two-eek cycle where we review requests and we review the budget and then we make decisions um in kind of a hierarchical way and then um you know uh

compute gets decided that way. Um now

having said that um this is something that I think we can still do better at.

Um uh it's hard when we're making decisions about compute allocation because um every researcher is convinced that their idea uh could change the world if it just got a thousand times

more GPUs attached to it, right? And

they they might be right. It might

actually be true. And yet we're running at the limit. We don't have a thousandx more GPUs for every idea that that we have. We we have to operate within uh

have. We we have to operate within uh the limits that that we have. And so it is um a challenging process. We try to incorporate as many people's um

perspectives into that as possible so that it's as much as possible a shared sense of uh understanding maybe not agreement. So there may be times when

agreement. So there may be times when one project feels like it really deserved more GPUs because the impact of that would have been so high but it didn't get it. We hope in that

circumstance that they have an understanding of why some other project did get more GPUs and why that was considered more of a priority during this particular allocation round um for

the company. Um so that people can at

the company. Um so that people can at least uh understand you know that there's a reason uh for the allocations that we have. Um uh having said that um you know this process is always

improving. There's always more work to

improving. There's always more work to be done to make this more transparent and more fair. Um and and then of course u my my number one is just to get more GPUs so that um you know we can also

fund more things because I would like to do that too.

How do you balance useful research with great exploratory research?

My belief is that research needs to be bootstrapped. Um it research is a

bootstrapped. Um it research is a chicken and egg problem. So um it is always the case that every researcher believes if I just had a lot more resources my idea would change the

world. Actually, it's important that

world. Actually, it's important that researchers feel that way because if you didn't feel that way, you wouldn't have the conviction that's required to go do something crazy and new, right? So, you

have to believe um and and so of course um uh you start with that belief. Uh but

then how do you translate that belief into something that other people can understand, right? That other people are

understand, right? That other people are willing to invest in. Um this is what I call the the chicken and egg problem, right? Because like once your research

right? Because like once your research idea is obviously good and impactful, it's easy to get resources. But how do you get it to be obviously good and impactful without those resources, right? So so the way you solve chicken

right? So so the way you solve chicken and egg problems is by bootstrapping.

This is an iterative uh problem solving approach where you do something small.

you get some sort of signal about this is a good idea and you tell people about that and then you ask for just a little bit more and um if people saw like oh yeah that you know that experiment

turned out pretty well that's pretty intriguing we should probably do a little bit more there um then you're on track right and uh that that um over time you know iterate a lot iterate

quickly iterate many times uh you can bootstrap to you know finding significant resources for your idea and also usually attracting more people um to come along with it uh on the way

because they have a chance to see that this idea is going to change the world and then they want to be part of it.

Is that how the moonshots at Nvidia got started as well over the years whether that's in in AI or otherwise? So it it was bottoms up somebody coming up with a

good idea versus Jensen saying this is what we need to do. Well, you know, Jensen has lots of good ideas, too. And

so, the company is very responsive to to his ideas, and that's uh that's important as well. But Jensen very explicitly says all the time, this is a company of volunteers. You know, uh each

of us is here because we choose to we could we could be doing something else with our lives, but we we choose to be here. Um, and so, uh, you know, we we

here. Um, and so, uh, you know, we we tend to make decisions, um, uh, especially for early stage research, it it tends to be very bottoms up, uh,

because, um, you know, it's it's sort of an invitation like bring bring your best ideas. Let's let's figure out, you know,

ideas. Let's let's figure out, you know, what are all of our best ideas and then we'll take a step from there. Um, now do we sometimes have top- down um ideas uh that are important for the company

strategy? Of course, you know, of

strategy? Of course, you know, of course. Um, uh, NVFP4 pre-training is

course. Um, uh, NVFP4 pre-training is one of those, you know. So, we decided as leadership of the company, we're going to really invest in in FP4 hardware. Um, now it's time to go invent

hardware. Um, now it's time to go invent some optimization algorithms that succeed in using it. And, um, so, uh, so we told the team, we didn't say to the team, you have to work on NVFP4

pre-training. What we said is, there's

pre-training. What we said is, there's an opportunity, we're making a big investment, and if we can figure this out, it will be significant for our company. And then we let the people who

company. And then we let the people who are interested in that work on that and as a result we succeeded you know so um so it is a balance of like uh bottoms up

and tops down um u uh but uh it always has this bootstrapping feeling even even with something like NVFP4 where there's a significant like strategic top down component the actual technical solution

which is very intricate and complex and has a lot of moving parts that came uh from the researchers themselves and you know that's my belief is that research always comes from the researchers themselves. You can't tell research

themselves. You can't tell research exactly to how to go solve a problem because then it wouldn't be research, it would be engineering. But uh in a world of of AI where the most important

problems we have to solve all have this research component, there needs to be freedom for researchers to innovate if we're going to make progress.

Listening to everything you're saying, I'm struck by how entrepreneurial the culture at Nvidia still seems to be.

So like I'm it's a very large company so I'm I'm sure there's also of politics and you mentioned the tribal instincts like I'm sure all of this is happening but um especially given you know how long the company's been around a

phenomenal success the fact that people have been making a lot of money internally it still seems to be very entrepreneurial bottoms up driven maybe meritocratic is that the right uh

takeaway yeah I mean um one thing that's very unusual about Nvidia is the tenure of its leader leadership. Jensen Wong has been running the company for 33 years,

but he's not alone. There are a lot of other very senior leaders in the company who have been there for three decades or longer. Uh including my boss and um

longer. Uh including my boss and um these people remember what it feels like to work at a very small NVIDIA and they know what it feels like to work at a

very large NVIDIA. Um they have a shared sense of ownership for the company. you

know um Nvidia is a place we often say uh no one fails alone um and the the the point of that uh that's just a statement of fact right you work at a company it's

a one company you all succeed together you all fail together you work in accelerated computing accelerated computing is the composition of thousands of technologies if any of them

fail to deliver acceleration the value is destroyed it doesn't matter whether the chip is great if the compiler sucks at the end of the day the thing that you're selling is time and capability to researchers that are trying to build the

future of AI. And if they don't get that, it doesn't matter whether it was the, you know, the transistor or the math unit or the compiler or the library or the networking or anything else along

the way that that failed to to live up to its um expectations. The whole thing in composition fails, the whole value is destroyed. And so we have a deep

destroyed. And so we have a deep understanding of that culturally at NVIDIA and it is something that motivates the way that we work together.

Maybe to close the conversation, I'd love to zoom out, get your take from, you know, the perspective of somebody who's like as deep into all of this as

it as it gets about uh where things may be going. So like who knows in a few

be going. So like who knows in a few years, but I don't know in the next year or two maybe there's some visibility. I

read somewhere that you're not necessarily a big uh singularity uh kind of a kind of person. Is that is that fair?

True.

And what why is that?

Well, I think that intelligence is just so incredibly multifaceted. Um, you

know, I always think uh about this question like um uh if uh a company were to be looking for its next CEO, would it find the next CEO by looking for

somebody who won the International Math Olympiad?

Probably not, right? Even though like it's incredible for people like I could never even compete in any way at the International Math Olympiad and those people are amazing, right? they have

just incredible brilliance. That's not

the right kind of brilliance to run a company. Um if we look for example at um

company. Um if we look for example at um other aspects of our culture that that are really important. Um for example, musicians. What kind of intelligence

musicians. What kind of intelligence does it take to become a hit musician?

Um don't assume that it's all luck. It's

not. These people are working hard and they're very smart in ways that I might not understand with my PhD, right? I

might not have that kind of intelligence. Um, and so when I think

intelligence. Um, and so when I think about intelligence, I think it's just so multifaceted and so contextual. You

know, it really depends on the situation. It's not just about raw

situation. It's not just about raw intelligence. Raw intelligence is kind

intelligence. Raw intelligence is kind of like the horsepower of an engine, but an engine running without wheels doesn't go anywhere, right? So, so intelligence um the impact of intelligence has a lot

to do with um the context that the intelligence is put in the harness, the platform. And so when I think about

platform. And so when I think about that, I think um you know uh the singularity is although it's an attractive idea, I think that it's it's a really a wrongheaded idea because it

it doesn't really um take into account um these other factors. Um so I believe that artificial intelligence is going to continue to develop at a rapid pace.

It's going to unlock significant capabilities uh for people in every aspect of um of our world economy, people doing every kind of work. Um I'm

very excited about the opportunities that it's that it's going to bring. Um I

am also a little bit uh concerned with how we're going to manage the transition. So I do think that

transition. So I do think that transitions are hard for humans in general. Like we're we're conservative

general. Like we're we're conservative generally. Um and uh you know there

generally. Um and uh you know there there is going to be a lot of change.

This is a profound change in the way that we think and the way that we work, the way that we learn. Um uh ultimately I have faith in our ability as humans to

figure it out. Um you know we've done it in the past. This is how this is who we are. Um we we build tools. Uh we build

are. Um we we build tools. Uh we build external organs that help us solve problems. You know we we have an external stomach. We call it kitchen. It

external stomach. We call it kitchen. It

creates enormous value for us. We can

eat things that we couldn't eat without a kitchen. Right now we're creating an

a kitchen. Right now we're creating an external brain. you know the the

external brain. you know the the implications of the external stomach were pretty profound uh for us as a species. They led to agriculture which

species. They led to agriculture which led to organized societies the way our cities are built. So we think about what is the implications of an external brain pretty profound. Nobody actually really

pretty profound. Nobody actually really knows. Um but what I do believe in is um

knows. Um but what I do believe in is um uh the power of humanity to solve problems and to learn and to incorporate new technologies in ways that benefit

us. Um, I also believe that the problems

us. Um, I also believe that the problems we face as a planet all require more intelligence. Every single one of them.

intelligence. Every single one of them.

Whether that's inequality uh or climate um change or um uh you know any of the other um structural uh uh problems that that I think are very worrisome that we

face. The solutions to those are going

face. The solutions to those are going to require invention and intelligence.

And what that means for me is that the only kinds of tools that we can really create moving forward are going to be AI. Uh because the problems that we face

AI. Uh because the problems that we face are all about intelligence. And

regardless of the technological approach to solving those problems, uh the solutions will always be called AI. Um

and uh so that um uh makes me hopeful for the future but also you know somewhat um you know respectful of the challenge that it is going to bring to us as we we try to figure out how to

live in a new way with this new external brain. Uh but I believe in our our

brain. Uh but I believe in our our ability to to learn and to change. Um

and and I I think um ultimately this is going to make our lives better.

Do you guys feel the AI backlash that seems to be forming internally? Is that

is that something that you all perceive think about? And if so, do you think

think about? And if so, do you think it's a communication problem that our industry may have? You know, in particular, given what you just said about all the obvious potential of AI,

you know, I'm always worried about the way that the public thinks about technology and interacts with it. It

matters a lot. Um and it is definitely the case that um societies that want technological advancement have more technological advancement than societies

that that don't want change. Um so I think it is um actually important to think about it. One thing that's interesting about AI is that um I

believe it tends to be uh much more accepted when it is uh part of everyday life and then at that point people stop thinking about it as AI. it's just, oh,

this is the tool that I use. Like, do

you care whether it's AI that's helping you uh route your car when you ask the map application to help you drive somewhere? Like, I mean, it is there is

somewhere? Like, I mean, it is there is actually sophisticated AI that's going into that. Um, uh, uh, but you're not

into that. Um, uh, uh, but you're not really thinking about that, right?

You're just using a tool. And, um, so I feel like people's acceptance of AI um, you know, comes with experience, right?

um the more experience we have working with it, the more we learn how to work with it productively, um I think the more comfortable we we become with it.

Great, Brian. So, it's it's been a a fascinating conversation. Maybe as a as

fascinating conversation. Maybe as a as a very last question to make sure we cover it, I want to make sure that we talk about safety. What is the state of

safety currently? And where does open

safety currently? And where does open source and closed source sort of fit in the safety conversation today?

Safety is on everybody's minds right now. Um you know watching uh the Fable

now. Um you know watching uh the Fable release and the um the way that the government interacted with that um uh uh

I think uh is a consequence of um concerns about safety about these models. you know, they get stronger and

models. you know, they get stronger and stronger and then they could be uh misused. And um you know, there's

misused. And um you know, there's different approaches to thinking about safety and and trying to define safety.

Um I have maybe a a a slightly uh unorthodox opinion about this, which is that I think open technologies are generally safer because there's more

sunlight. you know, when more people are

sunlight. you know, when more people are thinking about um the safety of a technology and evaluating it and then contributing to making it safer, I think

that's inherently uh safer than um having a small group of people uh being in charge of safety for everyone else. I

also think with um artificial intelligence because it is really about ideas. It's really about um exploring

ideas. It's really about um exploring ideas in different ways. That diversity

is more safe than monoculture. Um and

what that means is that there's going to be different beliefs. Like diversity

isn't just about like the easy stuff.

Diversity is about the hard stuff. Like

when people have deeply felt disagreements, they really really totally disagree with each other.

um making it possible for people to explore uh their ideas in a diverse way I think is more safe than um trying to create a walled garden where um you know

certain ideas are considered safe and certain ideas are are considered unsafe and um you know this is uh controversial in today's AI um environment which I

think is interesting because we've had hundreds of years of tradition uh that speak directly to this. You know, in in um the United States, for example, we

have um laws about freedom of conscious conscience and um freedom of speech. And

you know, it's not because we didn't consider for thousands of years would it have been safer if we didn't have those, right? We tried that. We we tried

right? We tried that. We we tried actually having a monoculture about like these ideas are safe to talk about, these ideas are safe to believe. And we

found that to be much less safe than a pluralism where uh we officially don't take a position about what ideas are safe. We actually found that is much

safe. We actually found that is much safer as a society to in uh to um support diversity uh than it is uh to try to keep everybody safe um top down.

And so um I believe that open technologies for AI are inherently uh the safest way of building AI.

All right. Love it. uh controversial

take to close uh the conversation.

Brian, it's been fabulous. Thank you so much. Really appreciate your spending

much. Really appreciate your spending time with us today.

Thanks for inviting me.

Hi, it's Matt Turk again. Thanks for

listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very

Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This

really helps us build a podcast and get great guests. Thanks and see you on the

great guests. Thanks and see you on the next episode.