Introduction to Physical AI & Robotics at NVIDIA

Today I will talk about uh physical AI and robotics. Uh my name is Kalyan

and robotics. Uh my name is Kalyan Vadravu. I do product marketing here at

Vadravu. I do product marketing here at NVIDIA. Um so the agenda today is very

NVIDIA. Um so the agenda today is very straightforward simple. Uh we we'll talk

straightforward simple. Uh we we'll talk about what what is physical AI and what differentiates physical AI from

other modalities or other forms of AI.

what simulation first development is and what are the challenges and and progress we've made in real world deployment and finally we'll end with some resources so

very very straightforward um talk today um so before I jump in I just wanted to situate you guys about a little bit about myself

um like this this picture uh talks or or shares Um I had started my career in in

computer science uh with an ambition to someday you know live at a place where I can do the show and tell the the

metaphoric show and tell of of a technology. So I've always uh looked out

technology. So I've always uh looked out for roles where I'll be able to talk show and tell about what technology does

for you and and that has been my core goal. I started as a computer science

goal. I started as a computer science engineer. Our first job was in was in

engineer. Our first job was in was in telecom uh back in the days of you know CDMMA roll out here in the United

States. Um I transitioned from there to

States. Um I transitioned from there to uh a field application role um where I was able to interact with customers and

I liked the piece where I was bringing that technology to real world. Uh so

that made me think uh to go back and and to school and and do my MBA.

It was helpful in grounding myself in in what it takes to be a good marketeteer. Uh I

focused in B2B marketing. Um and over the years I've had an opportunity to work across various flavors of marketing at companies like Nokia, Microsoft,

Semens and now here at Nvidia even within Nvidia. Um I've I've had a great

within Nvidia. Um I've I've had a great luck and chance to explore various flavors of marketing. uh but but like

the picture tells you I enjoyed and happiest in in product marketing and um that that's that's where I feel home and

uh I want to stay infinitely curious about AI and robotics because it's it I feel it's it's an amazing opportunity and the

reason I'm sharing this is is uh to give a flavor of you know how my journey is and and every career is different but I

think one core piece is I I had one general direction and I just being curious at each role uh that was you know available to me that that that that

I took on right so I think just just wanted to get out of I live in the Bay Area I've been fortunate to live here over a over a decade uh so I get a chance to interact with the really sharp

and and great colleagues here at NV and outside Nvidia as well. So with that out of the way, u I hope this gives you a

little of who I am and let's let's get going.

So the the the first question as as as you guys come in you have is you know what is physical AI? What is what is so different from the rest of the forms of

AI? Right? So there have been various

AI? Right? So there have been various evolutions from the AlexNet moment uh to today. There has been generative AI,

today. There has been generative AI, agentic AI, physical AI. Physical AI is the manifestation or embodiment of AI in

our physical world. So imagine

a robot which in previous era was doing one pre-programmed task, right? It was

just welding at one particular uh x and y coordinates all the time on a car, right? Or picking one kind of object

right? Or picking one kind of object uh one kind of object hundreds of thousands of times. It was

pre-programmed to do just that, right?

But but when when you introduce AI into these workflows, these workflow each of these workflows becomes intelligent, right? So imagine a a vast mix of

right? So imagine a a vast mix of objects that now this you know industrial arm can pick various different things on the on on a

car car that the uh that the arm can do or if it's a if it's a robot that's navigating it can see and and work out

different scenarios right so that is physical AI and and we believe that is the next wave of AI right these are models models that take inputs,

instructions, uh, but they generate actions, right?

So, it's it's inputs to actions. And in

terms of what we think of robots, we think of it pretty broadly. A factory is a robot. If

pretty broadly. A factory is a robot. If

if you think about it, it has many intelligent systems and there are many moving things, non-moving things. those

all can benefit from AI cars. It it is probably the most

cars. It it is probably the most prevalent robot that's available to us and of course the real robot robots that

that we are seeing an exponential growth over the last you know 5 to 10 years right especially because

they we need these robots to do all the dull dangerous and dirty jobs right so the the these robots uh can go in these

intelligent robots bots can go in and solve for these situations where we don't have to do it anymore. Right? So

that's the core of what physical AI is and and and and I want you to you know uh ground yourself on on this concept.

But now what does it take to build physical AI, right? and and and what we have

theorized or what we are proposing is there are three key things that you need to build physical AI right first you

need to train the model right and and then and then it you need to simulate situations where it will it will work and then you have to deploy the model

right so this is a core concept that I want you to take away today, right?

Train, simulate, and deploy. And we have three different kinds of computers who have who are specialized in these three things at NVIDIA, right? For for

training, we have DGX. For simulation,

we have OBX and RTX Pro computers. And

for deployment, we have Jetson. Many of

you may have heard of Jetson. So these

these are the three computers and we call this the three computer framework to solve the challenges of physical AI, right? You need all these three

right? You need all these three computers um to build that AI model to build that physical AI um and and bring it bring it

to life, right?

So with these three computers we provide uh a platform called Nvidia is which is an open development platform which has three things each related to the three

topics that I just discussed.

For training we have synthetic data generation and data pipelines. For

simulation we have open simulation frameworks. And for deployment we have

frameworks. And for deployment we have the robot foundation models which you can customize for your robot

and and deploy it on on a jet. Right? So

this is the overall framework of how you go about building physical AI. Right? So you need these these three integral steps and

these three pieces of of the development platform right so let's start with the first piece

which is data right and and I I think many of you will appreciate um this this I I I'm not sure if you've

seen this but physical AI is very hard to develop Why is that? If you look at the the, you know, the chat bots and and the large

language model scientists and data scientists and stuff, they have no der of data, right? So there's the entire

internet's worth of data for them, right? And and the large language

right? And and the large language models, they have been trained on the internet scale data, right? So that's

easily available can be done. But with

physical AI that is just not enough because you actually need to understand how for example if you're manipulating

as simple thing as uh an apple you you need the actuation data the act the the data that the haptics feedback and all of

that that is not available there is no uh such easy source right and to to get to that

it it's very costly, right? So, you have to have what what we call the teley operation, right? So, you can collect

operation, right? So, you can collect you can teleoperate uh the robot and collect the data, but it is um it is limited by the number of hours you have,

right? So, it is a person who is

right? So, it is a person who is dedicated to collecting that data. So,

that's one piece, right? And and how do we solve that? we to to really build physical AI, we need the data because as

many of you know that these models are trained on huge corpasses of data so that they know what environment in

general they're going into. So how do we solve that? And and this is a little

solve that? And and this is a little more detailed view of what the robotics data gap is, right? So there is uh there are

three kinds of data that we need right.

So the data soup or the data recipe we call and each is important right. So you

need the internet data, synthetic data and real data. Internet data

real data. Internet data we all know it's very easily accessible right? So the egocentric data for

right? So the egocentric data for example somebody you know doing the dishes or somebody you know going up the stairs and stuff like the

videos that are available on on you know uh on the internet but there is not much data on for example manipulation what what

exactly is happening how is the object behaving that's not available in the video right um and then so just like if If you think of this as a pyramid at the

bottom you have quite a lot of it but it's lower quality. Um then you go one high one level higher which we call the

synthetic data. This can be generated.

synthetic data. This can be generated.

Um but it is comparatively lower quality than real world data but you can generate it but you just need algorithms pipelines

and workflows to generate synthetic data. Right? So, so that is the piece

data. Right? So, so that is the piece where we come in. Nvidia, we propose that to solve this whole robotic data

gap, uh, we propose that synthetic data can be used can be used to augment uh the data that's available to train these models. And of course, like I said, real

models. And of course, like I said, real data is is gold. Of course, it is gold.

It's it's high quality. You are

capturing it. you are capturing for your robot for your use case and and doing so that's that's valuable but it's it's hard to capture it's not scalable right

imagine u you you need a robot to go and encounter any situation that's that's that's very difficult right so this is also a fundamental

you know core challenge of of robotics versus uh phys uh any other type of AI right so you aligns and so on and so that's that's the difference and and and

that's where many of you it's it's a great area to specialize in you know I I I I heard that many of you are you know students so it's it's it's a great area

to specialize in get a deep understanding of why uh the data is is is important what data is important how

do you capture it uh how do you generate it um so so I think that's that's a little bit about

So broadly what we propose is we have two two important platforms. Nvidia Omniverse um it it it it is used to generate

realistic environments um and then in conjunction with Cosmos which is our world foundation model platform. These

two when combined they have the potential to generate uh the synthetic data that can help solve the robots robotics data

gap. So um I would urge you to go deeper

gap. So um I would urge you to go deeper into it. There is there are subsequent

into it. There is there are subsequent uh talks that that that will happen on on some of these technologies but I will leave it at at this

high level uh that Omniverse is our you know realistic simulation environment.

There are simulation frameworks that are built on top of it. And then cosmos which you know as a you know what world foundation

model is it it is able to generate new worlds reason about new worlds and predict about what what happens in the future. So world foundation models um

future. So world foundation models um are also used for data augmentation.

But what kinds of data is is is needed for robots? Predominantly two kinds, right?

robots? Predominantly two kinds, right?

One is for mobility. So the the the robot needs to move around, right? So

there are two main use cases of it. And

then the other thing that it has to do is it has to manipulate.

Interestingly, um the the mobility piece is is is there's there's lot of you know data available. There's there's scaling

data available. There's there's scaling that is is relatively easier but for manipulation uh believe it or not there is death of information. So there's lot

of progress lot of research that's happening in generating synthetic data for manipulation. So that once you have

for manipulation. So that once you have that data combined with your your real world data, internet data, you can train your model to manipulate new things,

right? First identify it, manipulate it

right? First identify it, manipulate it and and and be useful in multiple different scenarios, right? So so that's the uh two use cases that you should

think about when you know, hey, what is what are these um use cases where synthetic data is is useful for robotics, right? Um so that's a little

robotics, right? Um so that's a little bit of introduction. So going back to our our framework. So we we we said you know train simulate and deploy. We just

finished train and now let's talk about simulation right going back to this this funny

uh slide. So we we've talked about the

uh slide. So we we've talked about the data is you know costly to capture but if you think about it testing a robot in

in real world testing it is extremely dangerous and expensive right so you you lose parts everybody has seen those bloopers from from Boston Dynamics and

and and several other you know robots that are out there. So it it is extremely dangerous and expensive and and and

You need ways for for you to move fast, learn quickly, test in various environments and bring your product. If

you're a robotics roboticist, you would want your product to come to market as quickly as possible. So building on that physical

uh testing or physically doing everything on the robot is very expensive and it's not safe. So we

believe that every robot from now and forward is born in simulation. So you

you have the robot, you build the robot in simulation, give it all the the testing and the training and let it explore all

the scenarios. And once you've done

the scenarios. And once you've done that, once you've you know validated and tested the the model, then you're ready

to deploy. So we believe in simulation

to deploy. So we believe in simulation first development and and this is coming to pass, right? So we've been working on this problem for

several years now and and we do see in the industry that simulation as the first starting point

is is gaining traction. Right?

So we have a few open uh robot simulation framework. So the first one is Nvidia

framework. So the first one is Nvidia ISX sim. It's completely open source.

ISX sim. It's completely open source.

You can use it for you know software in the loop testing, hardware in the loop testing. Um, I think I mentioned about

testing. Um, I think I mentioned about synthetic data generation and you you're pretty much bringing your

robot into this virtual realistic virtual environment where it where it's almost behaving as if it were in the real world and you're going through

testing your you know validating your code in these scenarios, right? And and

if you're actually you can also connect your hardware. For example, there's

your hardware. For example, there's there's ways to connect um Jetson into ISX SIM. And imagine that that uh the

ISX SIM. And imagine that that uh the the robot is thinking that it's it it is getting real world out uh inputs and outputs, but it's actually inside

simulation, right? How cool is that?

simulation, right? How cool is that?

So, so I I urge you to, you know, dig deeper in ISXM. It's completely open source. It's available on GitHub. Um so

source. It's available on GitHub. Um so

you can you can go deeper into it very high level what are all the things that I sim will enable you to do right so you can get started right away we have these

assets called sim ready assets so warehouses you know things that go inside warehouses office settings there thousands of these assets so that your

robot can then interact or you can create an environment with these assets right so that's that's the sim ready assets.

Um we have uh we can we can um bring in various kinds of robots right um imagine like a kuka robot auk robot or any kind

of robot that's that's available you can bring it into your simulation environment uh into your specific simulation environment and and and test

it out. So we we support many of the

it out. So we we support many of the available robot models but this this is not a foundation model but the the types of robots right different companies

there's high high fidelity physics simulation so the key thing again in in physical AI is you need to be able to

simulate the physics right so a bouncing screw or a friction of of things interacting with each other the per the fact that objects are permanent, right?

So, there's quite a lot of physics- based simulation um that is sim provides and and given it's it's open source,

it's fully extensible um and and the one cool thing is if you don't have a laptop that has an

Nvidia GPU, we have something called Brev. I actually encourage you guys to

Brev. I actually encourage you guys to go check out uh this this uh portal of ours. It's called brev brv.anvidia.com

ours. It's called brev brv.anvidia.com

where you can get GPU access and you can try out issim in within a very short time. You don't

have to worry about hey I don't have my laptop doesn't have an Nvidia GPU. So

it's a very easy way for you to test and try out these these u open-source uh simulation frameworks.

So these are the key capabilities with ISIM.

ISC lab is is is a software that's built on top of sim and this is used for robot

learning and policy training right u the core concept here is there are multiple ways for a model to learn right so these are age-old you

know machine learning concept reinforcement learning and imitation learning right reinforcement learning is you give a reward hard for everything that it's doing right. Um and and and

and you punish it for anything that's that it's not doing right. Right. But

doing it at scale GP. So it's it's in not not just one instance of the robot.

There are thousands of instance of robots who are trying different things.

So they're learning at an exponential scale uh in in environments that are provided by lab. So again this is

this is available for you to go deeper in and and it's available on GitHub. So

I urge you to you know take take a look at that and then similar to ISC sim um this robot learning framework which we call ISAC lab it's basically a gym for

your robot right it it goes to the gym to train and and learn about these environments.

Uh so ISAC lab is is another thing that's also available uh through through prev urge you to you know go go learn a little more about it.

Now, so we've going back to our first diagram, we've talked about training, why why why we need synthetic

data. Two, we talked about uh simulation

data. Two, we talked about uh simulation why simulation first development is important because you you don't want to spend time breaking your robot. So we

talked about simulation and within simulation we talked about what uh is sim provides um it and then what is lab which is a robot learning framework what

that provides both of them are open source doing it in in in bre right so we've completed those two sections now the third one is is the interesting one

many of you can get started on this as well today is the deployment right so now you've you've trained your model, you've, you know, tested and validated

through simulation, right? Um and now it's ready it's ready to go into the real world and and what are what what manifestations what

ways can it can it go in and what are the things that we provide for that right uh so one one topic that I want to talk

about is the is 1.5 which is a robot foundation model we have done all the the work that I've I've told you right so we've we trained

it using synthetic data. We you know validated in simulation trained the policy in in simulation validated in ISAC lab and now the model

is ready for you. So it's available on hugging face uh for you to go explore.

It is a generalized um robot model. So

what does what does that mean? Um, so

there is there is this concept of a generalist robot being able to do a wide variety of tasks in a wide

variety of environments um, right out of the box, right? So as

soon as you're able and and being able to be customized for your specific robot with very little data, right? So that is the core concept of this robot

foundation model. It's it it's able to

foundation model. It's it it's able to do manipulation tasks right out of the box. It you can postrain it uh with with

box. It you can postrain it uh with with your own data and it's ready to execute

various tasks. And I'll I'll keep this a

various tasks. And I'll I'll keep this a little high level. Uh but I I think one thing that I want to uh hone in on is

most of these robot foundation models um or vision language action models. You

may have heard of it. Vision language

action models.

They have a dual architecture, dual system architecture. One that's

system architecture. One that's basically derived from our human brain, right? So we have supposedly two types

right? So we have supposedly two types of systems. one, the deep thinking brain which is called system two um and then

the the acting brain which is system one. I think this this concept is is

one. I think this this concept is is from a very famous book by Daniel Canman thinking fast and slow. Um but most of the robot foundation model architectures

they resemble this. So system two and system one and essentially it is u you know various kinds of inputs in

and actions out right so it's it's almost like the robot is is is seeing something and it is getting some prompts and it is able to do some actions right

so from these sensor inputs and text inputs to actions that's where the magic is and

the the truly generalizable models.

They're able to do a wide variety of tasks with very little new data for a custom environment, custom robot

embodiment or and a custom task. Right? So that's

the key. So start with a model that's available. This is available in hugging

available. This is available in hugging phase and then post train it. And I'll

give you some links later. uh in the presentation on how to post train this this model.

So that's that's one uh thing that you can deploy but you can also deploy libraries NVIDIA ISC um acceleration

libraries for manipulation and mobility right so um I don't know many of you may or may not be aware of Ross I believe many of you know it's it's it's a

middleware framework open-source middleware framework um that's widely adopted ed by you know several hundreds of thousands of developers worldwide.

What we have done to support those robot operating system ROSS developers is we have built

over 30 accelerated ROS2 packages.

Right? So imagine a common scenario where you are estimating the pose of an object. What pose is it? So to

object. What pose is it? So to

understand where to pick something you need to know how the object is pose is and then you can use your gripper. But

imagine the these algorithms have existed right pose estimation, motion planning, depth segmentation but these are now hardware accelerated. So these

now uh work on Nvidia GPUs at at a much more performant uh in in a in a much more performant

fashion. Um and and these are, you know,

fashion. Um and and these are, you know, 30 or so that are available. Uh they're

all on GitHub. You can go take a look.

And it's it's as simple as if you have a ROSS graph or a you know the the entire you know workflow you can you know drag and drop some of these ROS two packages

in there right um and and two broad classes I think I mentioned in the beginning of the the presentation that two main things that robot needs to do

really well they have to move around that's mobility and being able to manipulate things that's that's manipulation

Um so we we've been working on isacross uh now for a handful of years uh getting great feedback from the Ross

community. So do take a deeper dive into

community. So do take a deeper dive into isacross if you're a ROSS uh developer.

So these are two examples of of um models and libraries that we provide that that actually run on the robot,

right? Um so these are models and

right? Um so these are models and accelated libraries. But in order to run

accelated libraries. But in order to run these models and and these youated libraries which demand a lot from the

from the hardware, you need a platform that that can actually you know be performant while doing all this because a robot can't stop

while doing while thinking or else it might fall down or or the next package might go away. Right? So it

it needs to be real time and to do that we have Nvidia Jetson. Uh many of you may have heard of uh Nvidia Jetson

platform and this is our platform for deployment and and like like I just mentioned physical AI um demands very high

performance computing. You need to be

performance computing. You need to be real time. you you can't uh have a lot

real time. you you can't uh have a lot of latency or delay when when you're doing things. So for example, you know,

doing things. So for example, you know, humanoid robotics, visual agents, medical robotics, they're, you know, mission critical stuff, right? Um

across all these scenarios, you need a platform that that is responsive, that's real time, that's able to take in all these sensors. We call it sensor fusion,

sensors. We call it sensor fusion, right? There's there are

right? There's there are lidars cameras um various other sensors that are on board the robot, being able to

rationalize all that and decide, reason and and then act, right? So, we need a platform that that supports all of this

and and that is what Nvidia Jetson is, right? Um the ultimate platform for

right? Um the ultimate platform for physical AI right now is Jetson Ajax Thor which we announced you know I think

earlier this year. Um it

it is the platform if you're really looking for building humanoid robots. If

you have multi multiple models that are running along with accelated libraries that are running uh for mission critical use cases for use cases that that need

real-time response. So that's Jetson AGX

real-time response. So that's Jetson AGX star. But we also have Jetson Ajax Orin

star. But we also have Jetson Ajax Orin Jetson Orin Nano Super and I think Jetson Orin nano super is a great uh first

device for you to get started right um I've I tinker on my free time uh Jetson Orin has been amazing orin nano has been amazing for me to learn it's it's

affordable and accessible as well so um just like I mentioned brev as as a as a good starting point for you to try is

sim is labor nano super is a great way for you to get hands on and get started So there's quite a lot of resources available for you to get started on

Jetson. Uh there's community projects.

Jetson. Uh there's community projects.

There's there's a whole site called Jetson AI lab which I highly recommend for you. Go and and take a look at it. It's it it has a wide range

of uh projects and and ideas that are available for you to go uh you know discover more about physical AI.

Um and I want to just close out giving you a simple example of you know where where all of these you know comes together. So this is agriculture uh is

together. So this is agriculture uh is is uh one of an important use case that we have seen. Uh it it it you need to do

quite a lot of things. There is there's complete visual perception sensor fusion and and an autonomous let's say a tractor. Imagine an autonomous tractor.

tractor. Imagine an autonomous tractor.

It has to make several decisions in real time and and and be performant and do all these actions. So this is just an example.

actions. So this is just an example.

There are many other examples like like I mentioned earlier there are vision AI agents robots that are looking at other robots saying you know what what is this

robot doing right or not? What's the

environment uh like and and so on and so forth. So, so that's that's that's the

forth. So, so that's that's that's the use case. There's many use cases for for

use case. There's many use cases for for example, you know, uh in in healthcare, surgery and and so on and so forth. Um

but but so that encompasses what uh what I wanted to talk about at at a high level. So what what we we talked

high level. So what what we we talked about is three things. First is

how do we you know train our models?

What are the challenges? Why is data so important? Um what is simulation first

important? Um what is simulation first development? Why is that important? Why

development? Why is that important? Why

is that gaining traction? And then

finally we talked about deployment, right? So that's that's the arc that we

right? So that's that's the arc that we went through and now let's go into some fun things. Now it's holiday season. Um,

fun things. Now it's holiday season. Um,

so if you if you want to get started, um Jetson dev kits, developer kits, they're all actually 50% off. Uh, so it's a it's a great time for you to get started. I

think there's some of these is is available in US only, but but I, uh, do take a look. Um, I usually use, you

know, the December break to to try new things and and so it's it's it's it's a good time for you to, you know, put your get your hands dirty and put your uh to

dip your toes into AI and robotics. So

that's available one. My colleagues from the the team suggested that I share this with you.

Grab the QR code and and you should be good to And I think we'll we'll put the the links in the in the chat. Um let me see how we're doing on time here. I

think we have five more minutes. Okay.

So I want to uh so you've heard all this but I what how do you get started? What

are the things that you can get started with? Um I I strongly recommend that you

with? Um I I strongly recommend that you get started with the robotics learning path. the one on your left, the left pan

path. the one on your left, the left pan most panel. So, it's like cool set of

most panel. So, it's like cool set of self-paced and instructor-led courses. I

think that's a great way to get your, you know, get a little bit of understanding of what what things are and how to do it. And and I'm a big

believer in learning through doing. Uh

reading or is is okay, but it's doing is is is the way to learn. I've left you some links with for the developer forum and then uh Omniverse developer

community we do quite a lot. Our teams

do live streams um almost every week. So

I think you should you should get benefit from it. We have discord in various channels. The forum is very very

various channels. The forum is very very healthy. The community is happy to help

healthy. The community is happy to help you. We are happy to help you. So I

you. We are happy to help you. So I

would say you know these are a few things for you to think about this you know this holiday season if you have time you want to get get ahead get started and

just a bonus I really like this that we've been doing quite a lot of work with hugging face slay robot uh which is

um is is a great platform for folks to get started with robotics um I had mentioned group 101 1.5.

Um so this is basically the group n1.5 model running on layer robbo's arm right uh so 101 arm um there are quite a few

resources this cure code links to one we'll add few more lobo framework has also been integrated uh group 1.5 has been integrated into the layer framework

so that's that's another it's it's very affordable open source um so it's a great way to get get things going And um yeah with that I think u the team

has put together this this series. So I

think I would I would urge you to take join all of these uh the folks who are going to do is sim Jetson and research their they

are stalwarts in in their each fields they they they'll go deeper much deeper into each of these technologies that we touched upon today. So I'd encourage you

guys to join on December 9th, 12th and and 16th. So it's it's everything done

and 16th. So it's it's everything done before before Christmas. So you you have that time to start getting deeper in it.

Um so I want to leave you with this if you remember nothing else. Um we used to have an econ professor who would who would would say that uh if you remember nothing else

want you to remember this. um to develop physical AI there are three key things that you need to master you can specialize in any of these but given robotics is multiddisciplinary

more you know about these three things the better provides frameworks for synthetic data generation

simulation and then deployment so um with that I thank you for your time and I think we let's look at some questions Let me see.

Give me a minute while I scroll through.

Hopefully, it's quite large.

>> Calian, while you're checking things, would you be able to possibly show that QR code again for the 50% off? We have a few folks who'd like to see that QR code again. Yes, sure.

again. Yes, sure.

>> Thank you.

I think we can also drop the the link.

Actually, let me leave that screen like that.

Just going through one second.

Okay. So maybe what what I'll do is I'll combine a few questions, right? So

there's there's a few questions about roles and and transitions and being able to work in this in this field. Um

yeah, I think so maybe I'll combine all of that. Right. So I think

of that. Right. So I think the the the broad way to to think about it is I think the the the core concepts

of AI machine learning um being able to understand how a model is is built model architecture and how you know h how

these these uh key techniques are built that's that's that's you know table stakes. So once once you learn that uh I

stakes. So once once you learn that uh I would you know focus on building real life projects um doing it with the

open-source community doing it in the open I think that is probably the best way to do it. I have seen many engineers you know get their first break through the their their projects that they work

with open source for. I I I strongly recommend uh folks explore hugging face the models that are available.

try um Jetson AI lab which which actually helps you run all of these models and I think contribute to open source and and have a portfolio of

projects that that you would you would be proud to show off and I think that is the you know universal uh suggestion that I can give on all

most of these questions about transitions into the these kinds of roles and then in terms of courses like I said the robot learning path that that

I splashed earlier I think uh I think one of my colleagues can can share the links in in the chat uh that's a great place for you to start our uh DLI

courses and media deep learning institute courses are a great way for you to get started but I think the going back to the concept of

show and tell being able to learn through doing um is is extremely important and and working in open and

and contributing to open source be it through you know contributing data sets or models um or algorithms and so on and so forth and I've seen many of that that

work through and I think the other key thing that I want to mention is robotics inherently is multiddisiplinary so I would I would pick two or three of your

strong suites and and go deep but also also learn a little bit about you know what it takes to you know build a robot.

uh so I think just at a higher level uh so that you can work with your counterparts in a in a in a startup or a big organization so you know what the challenges and you know uh what happens

after your job ends right or or you get a deeper appreciation of it and and you can be of value u to a potential you know employer right so I think that's

that's you know broadly what I'd recommend I think um okay, let me bunch the rest of it. Quite

a lot of questions. So, give me a moment while I try to bucket these.

There are there are some questions on what kind of platforms are suitable for what kinds of tasks, right? So, I think

uh broadly there are three computers, right? that that that we propose for

right? that that that we propose for physical AI for for running it. So for

for training the models we have the DGX uh series of of computers. uh for for simulation we have this RTX Pro we we

broadly put it under OBX but RTX Pro uh 6000 series that's that's the computers that are that are specifically

built for uh physical um realistic physics simulation that's that's number two and number three the platform for deployment is Jetson right so there are

so there are these three hardware um platforms terms for each of these tasks. So that's one question.

Okay.

Where can we get find where can we find real-time projects to work on? There's

there's quite a lot of actually uh maybe you can start with Jetson community projects. So if you just Google Jetson

projects. So if you just Google Jetson community projects, there's quite a vibrant set of uh people who are building projects. So that'll give you

building projects. So that'll give you ideas on on what what folks are building. I mean it ranges from vision

building. I mean it ranges from vision AI, edge AI, um you know and and various types of robots, right? So so I think that's a good place to start.

>> Yeah, I think there's there's some questions on I6 SIM accuracy.

real world dynamics and and what are the main challenges I'm going to bucket all of that right um it it it is it the there is always the sim to real gap

right so there is there I would I would not be you know accurate if I say hey you know it's it's 100% so it it is it is not 100% simulation to reality there

is there is a gap but uh the one of the uh person has asked you know specifically for contactrich manipulation So what we are doing for

that it's a little bit of a deeper dive but um I believe it's Kiran I would recommend you to

dig deeper into Newton uh which is our open-source physics engine that that actually solves specifically for this it

works well with ISAC lab um so I think that is uh specific to contactrich manipulation and and simulation so that's That's um that's one thing you

can explore. And then in terms of

can explore. And then in terms of challenges and limitations, I think it it is it is evident that you know you you you have to

explore a few different uh situations, environments to get it to to work in the real world. So it's it there is there is

real world. So it's it there is there is still that sim to real gap but through ISC sim through ISC lab through Newton which is the new physics engine and and

through uh crowd uh provided community provided solvers we're able to bridge that gap

right so it's I it's still there but we are working on bridging that gap so I think and then I would also recommend there's a deep dive on I6 sim

uh in in the coming weeks. So I I would I would go deeper in in in during those um sessions.

Okay.

I think there's a few questions on on open source. Uh I I just could I just re

open source. Uh I I just could I just re uh reiterate some of the topics that I that I

mentioned. So in terms of open-source

mentioned. So in terms of open-source is sim and ISC lab they're you know open-source frameworks that that you can go extend you can you can build on top

of it uh you you can see what's under the hood um there's there's workflows that are built in I think I left links to the documentation so those are

available for you to go deeper and and learn more so I think there was one question about is and do you need an extra degree to support transition? I I don't I I I I

support transition? I I don't I I I I personally don't think you need an extra degree to support transition into these roles. I I think like I said, you know,

roles. I I think like I said, you know, hands-on activity working in open um building with the community are are the right ways to do it.

Okay, this is another related question with IS6. When you train a robot in

with IS6. When you train a robot in simulation, does it you know capabilities or application in the real world generalize to every environment

you deploy it into? Um it it it it depends on the specific use case, right?

It's a pretty broad question I think this this person has asked but but what what we believe is you you are making progress if if you're

if the net new data that's needed to train your your model is is is is lower and lower. You're able to generalize it

and lower. You're able to generalize it to new tasks. That's when you're winning. That's when you're you know

winning. That's when you're you know pre-training and and post-raining are are doing a good job. and and it is more of an art rather than a science. So as

for your specific embodiment, for your specific task um for your specific environment, you have to work through these three constraints to see what data

is is improving the success rate, right?

So it's a little bit of a trial and error for you know getting that last bit u to your question of every specific scenario. So you have to

do a little bit of you know digging deeper into these three things for this task for this embodiment this type of

robot. Um each robot has its own

robot. Um each robot has its own idiosyncrasies right so you you have to be able to you know customize to that.

Then of course the environment what environment are you placing your robot in. So um there's there's there's still

in. So um there's there's there's still work that needs to go in.

There's a question on Ross ROSS 2.

Um I think on on Ross 2 we we publish our bench performance benchmarks. Um so

I think uh that that would be the best thing is if you look for is Ross benchmarks on the GitHub you will see how um these um accelerated libraries

perform um what frame rate and and what success rate and so on and so forth. So

that those are available. Look for ISAC Ross benchmarks. Uh they're available on

Ross benchmarks. Uh they're available on GitHub. I think that's one of the

GitHub. I think that's one of the questions.

I think that's broadly the ones I've tried to club things together, but if I have not gotten to your uh things. So I

think I would say two things, right?

One, make sure you join the rest of the series, the the webinar that are going to happen in in the series. There are

three more. And then the forums right forums and discord are your friends please go in and and ask there that those are right places for you to ask

deeper questions and then uh another way to do it is our live stream. So with

that I think I I thank you for your time. Uh I hope some of this has sparked

time. Uh I hope some of this has sparked some um ideas in your mind and I hope to see you guys uh through the series and

uh I hope you have a great learning filled uh physical AI learning filled uh December. Uh thank you.

December. Uh thank you.