1: Introduction to Neural Networks and Deep Learning; Training Deep NNs

[SQUEAKING] [RUSTLING] [CLICKING] All right, so today's lecture. Introduction to neural networks and deep learning.

So we'll start with a very quick intro to these things, and then we'll switch and dive deep into neural networks. All right, so the field of AI originated in 1956.

Sadly, it didn't originate at MIT. It originated at Dartmouth because all these people got together at Dartmouth-- I guess it's got a nice quad or whatever-- they got together, they defined the field. But fortunately for us, MIT was very well represented.

So we have Marvin Minsky, who founded the MIT AI Lab-- John McCarthy, who invented LISP and then later defected to the West Coast-- and then Claude Shannon, who invented information theory, who was a professor at MIT.

So MIT was well represented. These folks founded the field.

And they were so bright, they thought that AI was going to be substantially solved, quote unquote, "by that fall." Now, obviously, it turned out a bit differently than what they expected. So it's been, whatever, 67, 68 years since its founding.

So it's gone through, essentially, in my opinion, three seminal breakthroughs, starting with the traditional approach, then machine learning, deep learning, and generative AI.

So let's take a very quick look at each of these breakthroughs and what motivated them.

So let's start with the traditional approach to AI. And so what is AI?

AI, informally, is the ability to imbue computers with the ability to do things that only humans can typically do-- cognitive tasks, thinking tasks, and things like that. And so the most common, sensical way to do that is to say, well, if I want the computer to do something complicated, like play chess, I'm just going to sit down with a few chess grandmasters, show them a whole bunch of board moves,

and ask them how they figure out how to respond, how to play the next move.

I'm going to sit down and talk to all these people, and then I'm going to write down a whole bunch of rules. If this is the board position, move this.

If this is the board position, move this, and so on and so forth.

Or I might sit down with a cardiologist and tell them, OK, how do you actually interpret an ECG?

They will give me all this-- similarly, a bunch of if-then rules. I will take all these rules, I'll put them into the computer, and boom, I have a system that can do what a human can do.

Now, this approach, even though it's commonsensical, it kind of makes sense.

It had success in only a few areas. And so the interesting question is, why was it not pervasively successful? Why was it not pervasively successful?

It seems like a pretty good idea to me.

And the people who came up with these things are smart people. They're not dumb people.

They know what they're doing. So why did it not work?

AUDIENCE: Because it's time-intensive, so in the case that you have to run through all these scenarios that can ever exist, and still some new scenarios can come up that you didn't cater for [INAUDIBLE. ] RAMA RAMAKRISHNAN: Right.

So there are two aspects to what you said, which is the first aspect is it's time-intensive.

That, as it turns out, is not a big deal because computers are getting faster and faster.

The second thing is actually the key thing, which is that it doesn't generalize to new situations very well. The problem is there are an infinite number of things that you're going to see when you

very well. The problem is there are an infinite number of things that you're going to see when you deploy these systems in the real world. By definition, what you're training it on is a small sample of rules. So these rules are very brittle.

But there's actually an even more interesting reason, and that reason is that we know more than we can tell. This is called "Polanyi's paradox."

So the idea is that if I come to you and say, hey, here's a picture, is it a dog or a cat?

You will tell me within-- I believe they measured it, like, 20 milliseconds or something.

You know if it's a dog or a cat. And then if I ask you to explain to me exactly how you figured that out, you'll come up with a bunch of reasons-- alleged reasons.

Oh, if it has whiskers, I think it's a cat, or whatever.

But the problem is that you actually, first of all, can't really articulate what's going on in your head, how you do these things.

And number 2, even if you articulate it, oftentimes your articulation has no correspondence with how your brain actually does it. So you're incomplete underlier. So this is Polanyi's paradox.

So if you can't even tell me how you do something, how the heck am I supposed to take it and put it into a computer? Doesn't work.

And second is the fact that we can't write down these rules for all possible situations-- edge cases, corner cases, cetera-- and the world is full of edge cases.

So for these reasons, this approach didn't work. And so a different approach was developed, and this approach was, well, basically said, hey, instead of explicitly telling the computer what to do, why don't we simply give it lots of examples of inputs and outputs?

Chess positions-- next move. ECG-- diagnosis-- inputs and outputs.

And then why don't we just use some statistical techniques to learn a mapping a function that can go from the input to the output?

That was the idea, and this idea is machine learning.

So machine learning is basically just a fancy way of saying learn from input-output examples using statistical techniques. Good. All right, so now there are numerous ways to create machine learning models. And if you have ever done linear regression, congratulations, you've been doing machine learning.

And only one of those methods happens to be something called "neural networks."

There are many other methods, and, in fact, you probably have done these other methods if you have done a course like The Analytics Edge or something similar.

So machine learning has got tremendous impact around the world.

It's like, at this point, it's widely accepted. It's a very, very successful technology.

And in fact, whenever people are actually talking about AI, chances are they're actually talking about machine learning. It's just that AI sounds cooler.

The only problem is for machine learning to work really well, the input data has to be structured.

And what I mean by that is data that can essentially be numericalized and stuck into the columns and rows of a spreadsheet. So for example, here, let's say I want to put together a dataset of patients, their symptoms, and their characteristics, and then in the following year, after they showed up at the doctor's office, whether they had a cardiac event or not.

I might create a dataset like this with age, smoking status, yes-no exercise, blah, blah, blah, blah blah. And so either these numbers are numbers--

blah blah. And so either these numbers are numbers-- they're numerical-- or if they are not numerical, they're categorical.

Yes-no smoking, yes-no-- things like that. Which means that if you have categorical variables, you can just numericalize them pretty easily. You folks have done some machine learning before, so you know things like one-hot encoding and stuff like that can be done to make them all numerical.

So the point is, you can just render the data into the columns and rows of a spreadsheet pretty easily. That's what I mean by structured data. But the situation is very different

pretty easily. That's what I mean by structured data. But the situation is very different if you have unstructured data. So if you have an image of a cute puppy-- this is my puppy, by the way, from many years ago. Sadly, he's no more, but his name was Google.

[LAUGHTER] So, yeah, anyway, my DMD alums know Google well. So this Google.

If you want to take Google-- this picture-- and figure out how to numericalize it, the first thing you want to-- need to understand is that if you actually look at how this picture is represented digitally in the computer, basically every picture like this is depicted using three tables of numbers.

And we'll get to what these numbers mean later on, but the point I'm making is that each number basically represents the amount of light-- on a scale of 0 to 255, the amount of light in that location, in that pixel.

That's all-- the amount of light. So basically, this table is the amount of-- sorry-- this table is amount of red light, amount of green light, amount of blue light.

Now, you will agree with me that if you, for example, look at something like this and say, OK, 251 at this location-- there is a lot of blue light because it's 251 out of a possible 255. Maybe a lot of blue light somewhere here. There's a lot of blue here.

Whether that area is blue because of a piece of sky, some water, or a bunch of blue paint-- could be anything-- it's going to say 251. So the underlying reality, the underlying object that's being described, has nothing to do with the 251, so that's the whole problem.

The raw form of the data has no intrinsic meaning with the underlying thing.

So given that there's no connection between the number and what it's describing, how the heck can any algorithm do anything with it? It can't.

So what you have to do is something called "feature engineering," or "feature extraction," where you have to manually take all these things and create, essentially, a spreadsheet from them.

So basically, let's say that you have a bunch of birds and you're trying to build a bird classifier to figure out what kind of bird species it is.

You might actually have to take this picture, and then you have to measure the beak length, the wingspan, the primary color, and so on and so forth.

So you basically structuring the unstructured data manually.

And this process of structuring unstructured data is basically called-- we use the word "representation." We take the raw data, and we represent the data in a different form. And the reason why I'm focusing on the use of the word "representation" is because it becomes really, really important a bit later on when we get to deep learning.

So we have to represent the data in a different way for it to work. That's the basic idea.

All right, so what that means is that historically, researchers would manually develop these representations.

And once you develop them, once you have the representations, you can just use traditional linear regression or logistic regression and get the job done.

So the whole name of the game is the representations.

So in fact, people doing PhDs, for example, in computer vision, would spend, like, four years developing amazing representations for solving one particular little problem. We have a bunch of, say, CAT scans, and we need to take the CAT scan and figure out whether a particular kind of stroke-- there is evidence for it in the CAT scan. They might actually sit and develop all kinds of representations and test it and so on,

and then they'll finally declare victory and say, yay, I'm done with my PhD.

Here is this amazing representation, and you can build a classifier with it to predict a particular kind of stroke with high accuracy. So that's where the world was.

Now, as you can imagine, developing representations-- because it's so manual-- is this massive human bottleneck, and this sharply limited the reach and applicability of machine learning as you would expect.

To address this problem, a different approach came about, and that's deep learning.

So deep learning sits inside machine learning. And deep learning can handle unstructured input data without upfront manual processing, meaning it will automatically learn the right representations from the raw input. "Automatically" is the keyword. Automatically learn representations,

which means that you could give it structured data, you can give it pictures, you can give it text, you can give it anything you want, and just learn it.

It can automatically extract these representations. And since it's being automatically extracted, you can imagine a pipeline where the raw data comes in, you have a bunch of stuff in the middle that's learning these representations automatically without your help, and then boom, you just attach a little linear regression or logistic regression at the end. Problem solved. That, in a nutshell, is deep learning--

input, a whole bunch of representations being learned, and then piped into a linear or logistic regression model. So the amazing thing is this simple idea is just incredibly powerful.

regression model. So the amazing thing is this simple idea is just incredibly powerful.

That idea has led to ChatGPT, has led to AlphaGo, AlphaFold, and so on and so forth.

And I kid you not, I've been doing deep learning for about 10 years now, and every time I look at it, I literally get goosebumps every so often-- that something so simple could be so powerful. It's really, like, boggles the mind.

I'm like, I'm just so lucky to be alive and working during this period.

And coming from people who have been in the industry a long time, this breathless exclamation is not very rare, particularly because I'm not in marketing.

I actually mean it, with all due apologies to various marketing folks. [LAUGHTER]

Just realized it's being taped, so. So this demolished the human bottleneck for using machine learning with unstructured data.

And so it comes from the confluence of three forces-- new algorithmic ideas, a lot of data, and then, very importantly, the fact that we have access to parallel computing hardware in the form of these things called GPUs-- graphics processing units. And these three forces came together, and they were applied to an old idea called "neural networks," and that's basically deep learning.

And I'll go through it very quickly, because obviously we're going to spend half the semester looking into this thing in detail.

So what's the immediate application of the ability to automatically handle unstructured data?

What is, like, the no-brainer application?

It's OK if it's obvious. Tell me. Sorry. AUDIENCE: Image classification. RAMA RAMAKRISHNAN: Right.

So image classification, yes. So you can take an image-- a good example of unstructured data-- you can do some classification on it. But more generally, what I'm getting at is that every sensor in the world can be given the ability to detect, recognize, and classify what it's sensing-- every sensor. Because remember, what does a sensor do?

A sensor is just a receptacle for unstructured data. A camera is a receptacle for unstructured video or unstructured still images. Microphone-- unstructured audio.

So every sensor-- you can imagine taking a sensor and sticking a little deep learning system behind it. And now, suddenly, what comes out of the sensor of the deep learning system, you can count, you can classify, you can detect, you can do all kinds of stuff. In short, you can analyze, and you can predict.

And the way I'm describing it right now, you'll be like, yeah, duh, obviously. But you know what?

This "obviously" thing is actually not at all obvious in terms of whether it will help you find interesting applications or not.

So here's something I literally saw last week. Actually, I have another slide before that, but we are coming to that. So for instance, every time you use Face ID, unlock your phone, this is the basic principle at work. The camera and the iPhone is the sensor, and they stuck a deep learning system behind it to do image classification-- drama, non-drama.

That's what it's classifying. And so here, you have a breast cancer-- this is a breast cancer detection system from a mammogram.

By the way, this picture is a very interesting picture.

So there's a professor in EECS, Regina Barzilay, who's a very well-known expert in this field.

And she actually has built a breast cancer detection system, which has been deployed at Mass General Hospital.

And it turns out, she's actually a breast cancer survivor, and she was-- --she's good now, all good-- but after she built her system, I heard that she actually ran that system against the mammograms from many years prior, when she went for a mammogram and was told that everything is fine.

She ran the system on that mammogram, and it came back and said, here's a problem.

So a very interesting example where a deep learning system picked up something that a radiologist could not. So these things can be quite powerful.

Obviously, any self-driving system has numerous deep learning algorithms running under the hood-- pedestrian detection, stoplight detection, zebra crossing detection, so on and so forth.

It's being very heavily used in visual inspection manufacturing. You have various cameras, and it'll show people looking at-- saying, OK, there is a dent or there's a scratch.

They have a little system, which is a dent detector, scratch detector, and so on.

That's going on right now. And now I come to the example I saw last week, which is-- so this is an example of you can create dramatically better products if you really internalize this idea of-- it's almost like you're looking at the world and saying, oh, there's a sensor-- can I attach a DL thing behind it? That's the way you should be looking at the world for startup ideas. So here's an example.

These apparently are the world's first smart binoculars.

This is the binocular two weeks ago, where you look at the bird, and now it tells you what kind of bird it is right there.

It's a simple idea, but imagine you are the first out of the gate with this feature.

You will have a little bit of an edge till everybody catches up, like, three months later.

Let's be very clear-- there are no long-term monopoly windows in the world.

There are only short-term windows, so the hunt is always on for a little monopoly window.

So here's an example of that. So I encourage you to always think about the world as-- where are the sensors here? And can I attach something behind the sensor to do something useful with it? All right.

Now let's turn our attention to the output. We have been talking about in structured data, unstructured data, and how deep learning has unlocked the ability to work with unstructured data, but you've been neglecting the output side of the equation.

So traditionally, we could predict single numbers or a few numbers pretty easily.

So you've all done the canonical-- should this person be given a loan application in machine learning? So it just predicts the probability that a borrower will repay a loan based on a whole bunch of data or a supply chain. You predict the demand for a product next week, or it can predict a bunch of numbers. So given a picture, you can say, OK, which one of 10 kinds of furniture is it?

You can predict 10 numbers, 10 probabilities that add up to 1.

You can predict a whole bunch of numbers that don't have to add up to 1, such as the GPS coordinates of an Uber. So these are all simple, unstructured-- sorry, simple structured output-- just a few numbers.

What we could not do very easily was to actually generate pictures like this.

We could not generate unstructured data. We could only consume unstructured data.

You can generate text, you can generate pictures, and so on-- and audio, and so on and so forth.

So with generative AI, that problem is gone. So generative AI is the ability to actually create unstructured data, and therefore, it sits within deep learning.

It still runs on deep learning, but it's just one kind of deep learning.

There's plenty of stuff going on in deep learning that's got nothing to do with generative AI.

Nowadays, of course, if you're a self-respecting entrepreneur who wants to ride this craze, you'll probably declare whatever you're doing as generative AI, and some VCs may actually be ready to fund you. Who knows?

But the point is, there's plenty of stuff going on in deep learning that's got nothing to do with generative AI, but this is the overall picture.

Now, here we can produce unstructured outputs like pictures. You can take this thing, and then you can actually come up with a nice picture-- description of it.

This actually is a very famous picture, by the way, in the world of computer vision.

So we are actually going to be analyzing this picture a little later on in the semester.

You can obviously go from a very complicated caption to an image. You can go from text to music.

Can people hear it? OK. AUDIENCE: Yeah. RAMA RAMAKRISHNAN: Yeah, yeah.

All right, and, of course, we go from text to text, i.e., ChatGPT.

And then as of a few months ago, things have gotten even more interesting where you can actually go-- you can send text and an image in, and you can get text out.

And in fact, as of a few weeks ago, you can send text, image, text, image, text, image in an arbitrary sequence into the system, and it can actually come back to you with text and image. So things are becoming multimodal.

And I just want to share with you a really fun example I saw recently.

So this person sends this picture. Can folks see this?

This is very complicated parking sign, apparently in San Francisco.

And they are like, it's Wednesday at 4:00 PM. Can I park here? [LAUGHTER]

Tell me in one line-- because you really didn't want GPT-4 to be giving you a big essay about this.

You literally want to park. So GPT-4 comes back and says, yes, you can park here for up to 1 hour, starting at 4:00 PM. And folks, I double-checked this thing.

It's correct. We all know these things hallucinate. Can you imagine getting a parking ticket and telling the judge, I'm sorry, I didn't realize it was hallucinating?

So you have to double-check it. So, yeah, so things are getting multimodal very quickly.

And so the picture here is that within GenAI, we used to have these separate circles-- text to text, text to image, text to music, text to this, text to that, and so on and so forth.

Those are all beginning to merge now inside GenAI because multimodal models are going to become the norm this year. We already have really good closed models.

We actually already have very good open-source multimodal models, and so my feeling is that by the end of the year, the idea of using a text-only model is going to be like, really?

You do that still? It's going to become like a quaint, old-fashioned thing.

I think multimodality is going to become the norm.

So that's where the world is, and this is the landscape. So any questions on the landscape before we actually start doing some math?

OK. Yeah? AUDIENCE: It's not going to be the landscape, but it's more related to the breast cancer brush. So how did she know that it was evaluating, like, the training that you do, and you were [INAUDIBLE], which is based on the existing breast cancer [INAUDIBLE]. But here, she was taking the mammograms

from six weeks past, [INAUDIBLE], which is probably from a smaller version which was not there earlier, right? And [INAUDIBLE]-- RAMA RAMAKRISHNAN: You mean the evidence of there being a problem would have been smaller? AUDIENCE: Yeah. RAMA RAMAKRISHNAN: Yeah.

AUDIENCE: So how do know your model is trained enough that it done [INAUDIBLE]?

RAMA RAMAKRISHNAN: Yeah. So the question is that, in general, how do you train your model so that it gives you the right answers, given that over the passage of time, the amount of evidence in this data could be very highly variable? So in this particular case of the professor I talked about, everything at that point was going through an expert radiologist. So five years ago, this mammogram

was seen by a radiologist, and that person concluded there is no problem.

So that was the training label-- the wrong training label.

So typically what happens is that training labels could be wrong some small fraction of the time, so you need to have systems that are robust. So your data needs to be complete, it needs to be comprehensive, it needs to be have correct labels.

If these ideas are not met, your systems are not going to be that good.

But as it turns out, with neural networks, even with some amount of noise in the labels, they still do a pretty good job. So that's the general idea.

AUDIENCE: But there has to be some verification that [INAUDIBLE].

RAMA RAMAKRISHNAN: The verification comes from the human.

So remember, when we look at radiology data, the data we are working with is-- the input is, let's say, an image, like a mammogram or something.

And then a human radiologist, or a set of radiologists, have said, this has a problem or does not have a problem. So that is called "ground truth."

So it is this ground truth image and label-- this combination that's being used to train these models.

Yeah? AUDIENCE: Will you be covering, like, embodiment [INAUDIBLE]? RAMA RAMAKRISHNAN: Embodiment?

AUDIENCE: Yeah. RAMA RAMAKRISHNAN: So are we going to cover embodiment?

So the embodiment here refers to the fact that if you have robots, they need to actually operate in the real world. And so robots are an example of what's called "embodied intelligence."

So unfortunately, due to the constraints of time, we're not going to get into robotics at all.

But I will say that a lot of the deep learning stuff we're going to talk about, those are all fundamental building blocks in modern robotic systems. All right, so in summary, X and Y can be anything, and it can be multimodal.

I literally could not have put up this slide maybe two years ago.

So it's very simple in how it looks, but it's very profound. You can learn a mapping from anything to anything at this point very easily, as long as you have enough data.

So now, note that all this excitement that we see around us is everything stems from deep learning.

Everything depends on deep learning, and so if you understand deep learning, a lot of interesting things become possible. So let's get going, all right?

So we'll start with the very basics-- what's a neural network?

Now, recall logistic regression from back in the day. So what is logistic regression?

You send in a bunch of numbers-- a vector of numbers-- and you usually get a probability out-- between 0 and 1, what is the probability of something or the other?

And so this logistic regression model is also represented in this form, if you will recall.

So basically, what we do is we take all these numbers, we run it through a linear function, you get a number, and then we take that thing and run it through 1 divided by 1 plus e raised to minus that.

And that's guaranteed to give you a number between 0 and 1, which can be interpreted as a probability, and that's logistic regression.

And the canonical loan approvals, things like that, all fall under this convenient bucket.

So this should be super familiar.

All right, now we're going to actually look at this simple, modest, humble little operation using the lens of a network of mathematical operations, and the reason why we do it will become clear a bit later.

So we'll take this very simple example where we have, let's say, two variables-- GPA and experience.

This is the GPA of some graduates, a number of years of work experience.

And then this is the dependent variable, which is either 0 or 1.

And 0 If they don't get called for an interview, 1 if they get called for an interview.

It's a two-input-variable, one-output-variable problem, and it's a classification problem because we are classifying people into-- will they get called for an interview? Yes or no.

And so that's the setup for this problem. And let's say that we actually run it through any-- and we actually try to fit a logistic regression model to it.

So if you're familiar with R, for example, you would use something like GLM to fit this model.

If you use something like statsmodels in Python, there's a similar function for it.

Scikit-learn-- there's another function for it. You get the idea.

You can use whatever favorite methods you have for logistic regression modeling to get this job done.

And if you do that with this little dataset, you're going to get these coefficients.

The 0.4 is the intercept, 0.2 is a coefficient of a GPA, 0.5 for experience, and that is the resulting sigmoid function. All right, cool.

So now let's actually rewrite this formula as a network in the following way.

So first, what we'll do is we'll take GP and experience and stick it here on the left side, and we'll put little circles next to them, and we'll call them the input nodes.

And so imagine that somebody writes a GPA into the circle-- 3.5, or years' experience, 2.0-- and then it flows through this arrow. And as it flows through, it gets multiplied by its coefficient, 0.2. The 0.2 is coming from here. Similarly, experience gets multiplied by 0.5.

It comes in here. And this node, as the plus indicates, is adding everything that's coming into it.

So it's adding 0.2 times GPA, 0.5 times experience, plus the intercept, which is the green arrow coming from-- on its own. It comes through here.

And what comes out of-- this is just a single number. And that number goes into this little circle, and then out pops a probability. So I've sort of done this ridiculously long-winded way of writing a simple function, and the reason why I'm doing it is will become clear in a second.

So this is a little network of operations for the simple function.

And so, for instance, how you would use it is, to make a prediction, you will-- let's say someone has a 3.8 GPA and 1.2 years of experience.

You just plug it in here, do the math, you get 0.76. Same thing here.

Comes in here, add them all up, you get 1.76. You'd run 1.76 through the sigmoid, you get 0.85, and that is the probability that that particular individual may get called for an interview.

At this point, we're just doing logistic regression. Nothing more complicated.

So now, if you have many variables, not two variables, like x1 through xk, the same logic applies.

Each one has some coefficient, and then there's an intercept.

They all get added up here, run through a sigmoid, and out pops this number.

Notice how the data flows from left to right.

All right, any questions on this?

All right, good. So now a terminology. So you will actually-- you'll discover that the world of neural networks and deep learning has its own terminology.

They have their own ways of referring to things that the rest of the world has been referring using something else for the longest time.

It's kind of annoying sometimes, but it's the way it is.

So remember, in regression, we used to call those numbers next to each variable as "coefficients" and the constant thing as an "intercept." Well, guess what? In this world, those coefficients are actually called weights, and the intercept are called biases.

So in the neural network world, these are called weights and biases.

And sometimes if you're a little lazy, you may just call the whole thing as weights.

So when you see in the newspaper that, oh my god, this amazing model's weights have been leaked on the internet or on BitTorrent or something, that's what's going on. All these coefficients have been leaked because once you know what the coefficients are and what the architecture is, you can just reconstruct the model. All right, so that's what's going on here.

Now, why did we do this network business? Why did we write it as a network? Yeah, what is advantage?

Any guesses?

AUDIENCE: When you have multiple functions flowing through the network, it's just easier to see it that way. RAMA RAMAKRISHNAN: Right.

If you have lots of things going on, it's easier to see it if you actually write it in graphical form. Yes correct. But so is it only a usability advantage?

AUDIENCE: The other thing is you want different functions for different layers of the [INAUDIBLE].

RAMA RAMAKRISHNAN: Ah, OK, so maybe we want to use different functions in different layers.

But I think there's actually even a larger, more basic point, which is that the moment you write it down, you suddenly realize that I could have lots of things in the middle. I don't have to go from the input to the output directly.

I could do lots of things in the middle. That's some of the key idea. So what you do is-- so remember the notion of learning representations of unstructured data, where you take a picture and say beak length and things like that. And remember, I said deep learning actually automatically learns these things. Where is that automatic learning coming from?

Well, this is where it's coming from. So what we do is we take this thing-- this just a logistic regression model-- inputs get added up as a linear function or run through a sigmoid.

And then we are like, hmm, if we want to learn representations of the raw input, we better be doing something in the middle here because the output is the output.

That's not going to change. It's either a dog or a cat. You don't have any choice as to what it is.

The only agency you have at this point is you can take the raw input and do things in the middle with it. You can do a lot of stuff in the middle and then run it through something to get the output. So in any mathematical discipline, if someone comes to you and says, here's a bunch of data-- I want you to do something with it--

what is the most basic first thing you should do?

Run it through a linear function. The most basic thing in math is a linear function.

So given anything, just run it through linear function, see what happens.

So that's exactly what we can do. So the simplest thing we can do here-- we can insert a bunch of linear functions.

So what we do is we take all this input and we just run it-- we do a linear function on it.

So think of this as x1 times 2, plus x3 times 4, and all the way to xk times 9, plus some intercept, and boom, it goes to the other end.

So this little circle here with a plus in it is just-- thank you-- this is just a linear-- it's a shorthand for a linear function. So whenever you see a circle with a plus, it's just a shorthand for a linear function. So you can take this whole thing and run through a linear function, and when you do it, you'll get some number right there.

You'll get some number. So you've taken these k numbers and you've compressed them in some way into one number. But you don't have to stop at one number. You can do more.

So we can have a stack of linear functions in the middle.

There's a linear function here, another one here, another one here.

At this point, the k numbers you have-- k could be, for example, 1,000.

It's just the size of your input data. You have taken these k things, and you have compressed them into three numbers at this point.

So maybe 4 is the right number, maybe 10 is the right number. We don't know, and we'll get to how do we know what the right number is later on. So we can stack as many linear functions we want.

So we have transformed this k thing into a three-dimensional vector.

k numbers become three numbers, and now we can flow these three numbers through some other little function.

And as you will see in a few minutes, that function is called an activation function, and it's chosen to be a nonlinear function because if you don't choose it to be a nonlinear function, all the effort we're doing is going to be a total waste of time. For now, just take it on faith that you need to have nonlinear functions here, but note that the three numbers here are still three numbers.

They are three different numbers. We did three numbers.

And once we do this, we'll be like, you know what? This was fun. Let's do it again.

So you can do it again, and you can keep on doing it. You can keep it 100 times if you want.

And the key thing is that every time you do it, you're giving this network some ability, some capacity to learn something interesting from the data, to learn an interesting representation.

Now, of course, you're thinking, well, how do we know it's interesting?

How do we know it's a useful thing? And we'll come to all that later on.

We're just giving it the capacity, the potential to learn interesting things from the data.

Whether it actually lives up to its potential, we don't know yet.

We'll give it the potential because the more transformations of the input data you make, the more opportunity you have to do interesting things with it.

If I don't even give you the opportunity to transform it once, you don't have any opportunity.

If I give you 10 chances to transform things, you have 10 shots at doing something useful.

So you can do this repeatedly. And once we are done doing these transformations, we just pipe it through to our good old logistic regression sigmoid here, and we are done.

So this is the basic idea. And so just to contrast it, this was good old logistic regression, where we take the input, we run it through a linear function and pop out a number-- a probability number.

But after we do all this stuff, the input stays the same, the output stays the same, but in the middle, we just run it through a whole bunch of these functions, these layers-- boop, boop, boop, boop-- and then we get the output.

That's all we have done, and this is a neural network.

A neural network is nothing more than repeatedly transformed inputs which are finally fed to a linear or logistic regression model.

Any questions? Could you use the thing so that everyone can hear? Yeah.

AUDIENCE: There are two questions. Firstly, so can we say that there is a challenge of explainability? Is it that we don't know which arrow it went through? That's one.

Second, who's controlling the number of iterations or the number of functions? That's up to us?

RAMA RAMAKRISHNAN: Yeah. AUDIENCE: Or, how does that work?

RAMA RAMAKRISHNAN: So, yeah-- so the first question-- explainability-- we actually know exactly for any given input data point-- we know exactly how it flows through the network, so there is no problem there.

The problem is in ascribing, OK, we think this person is going to be-- repay the loan because of this particular attribute. We don't know that because those attributes all get enmeshed together and goes to this complicated thing. So we know exactly what happens-- we just can't give credit to any one thing very easily. And again, I'm just standing on the brink

of this vast ocean of something called explainability and interpretability, which I'll get to a bit later on in the semester, but that's the quick kind of right-ish, kind of wrong answer.

Number two, we decide the number of layers. We decide a whole bunch of things.

And as we'll see in a few minutes, there is something that's given to us and something we get to design, and I'll make it very clear which is which. Yeah?

AUDIENCE: Yeah, so this-- RAMA RAMAKRISHNAN: Did I say your name right? AUDIENCE: Yeah.

So which functions have to be linear? And also, why does it have to be linear?

RAMA RAMAKRISHNAN: Yeah, so these functions-- the f of x here-- they have to be nonlinear.

As to why they have to be nonlinear, we'll get to that in a few minutes.

So these are called "neurons." These things where you-- basically, there's a linear function followed by a little nonlinear function.

Each one of these things is called a "neuron." By the way, this is loosely inspired by the way how neurons work in human-- in mammalian brains.

But the connections between neuroscience and deep learning are very heavily argued, so I'm going to stay away from it.

Suffice it to say, I just think for building practical deep learning systems in industry, you're not worry about this. All right, let's move on.

Terminology-- this vertical stack of linear functions or neurons-- this vertical stack is called a "layer." This a layer, that's a layer.

And these little nonlinear functions, which we haven't got into yet, are called "activation functions," and we'll get to why they are called that in just a second.

And the input is called an "input layer," and I have the word "layer" in double quotes because it's not really doing anything.

It's just the input, but we call it the input layer.

And the very final thing that produce a output is called the "output layer," obviously, and everything in the middle is called a "hidden layer."

So the final piece of terminology is that when you have a layer like this in which, say, three numbers are coming out and there's another layer, if every neuron in this layer is connected to every neuron in this layer, it's called a "fully connected" or "dense layer."

So for instance, here, this arrow that's-- whatever number is coming out-- let's say the number 3 is coming out of this thing here-- that number 3 flows on this arrow to this thing, flows on this arrow to this neuron, and flows on this third arrow to this neuron. That's what I mean.

So every neuron-- its output is being sent to every neuron in the following layer.

We call it fully connected or dense. And then if you look at logistic regression-- this just logistic regression-- you can see, basically, logistic regression is a neural network with no hidden layers.

So in some sense, logistic regression is almost the simplest possible network you can think of-- like, barely a neural network. It's got no hidden layers. That's what makes it logistic regression.

And so as you might have guessed by now, deep learning is just neural networks with lots and lots of-- of what? AUDIENCE: Layers.

RAMA RAMAKRISHNAN: Yes, layers. So here are a few.

And by the way, these are not even considered all that impressive these days, but I put them up because this thing here is called "ResNet," and it's famous because the ResNet neural network was, I think, the first network to surpass human-level performance in image classification.

It's sort of like the Skynet [CHUCKLES] of image classification.

It surpassed human-level performance. And I'm putting it up here because we will actually work with ResNet on next Wednesday. And we'll actually take ResNet, we'll fine-tune it, and solve a real problem in class. All right, so it's got lots and lots of layers.

Now let's turn to these activation functions. We've been ignoring these little guys so far.

So the activation function at a node is, first of all, it's a function that receives a single number and outputs a single number. It's not very complicated. It receives, basically, this-- this is a linear function, which receives all these inputs-- it could be 10 inputs, 1,000 inputs-- runs it through a linear function, outputs a number. And that single number-- a scalar--

goes in here, and it comes out as another single number. Just remember that.

And so these are some of the most common activation functions.

In fact, the sigmoid we saw, which is actually be used for the output, is actually a kind of activation function where a single number comes in and it gets mapped into this curve because of this thing. So the single number that comes in is a, and it gets transformed as 1 divided by 1 plus e raised to minus a.

And you get a shape like this, and it's called the "sigmoid activation function."

And as you can see here, for very small values, for very negative values, it's going to be pretty close to 0, meaning it won't get activated.

And for very, very large values, it's going to be pretty close to 1.

All the action happens in the middle. When your values are somewhere in this range, there's a dramatic increases in what comes out, so that little thing in the middle is the sweet spot for these functions. And this-- I'm almost embarrassed to call it an activation function because it's literally not doing anything.

It's sort of gettinig a nice label for free. Basically, it says you just get a number, you just pass it straight along. It's a linear activation function, but just for completeness, I want to put it here.

And then we come to the hero of deep learning, which is the rectified linear unit.

Rectified linear unit-- it's called ReLU. And ReLU is going to become part of your vocabulary very, very quickly. And so ReLU is actually a very interesting function.

very quickly. And so ReLU is actually a very interesting function.

So you write it as maximum of whatever number and 0, which is another way of saying if the number is positive, just send it along unchanged.

If the number is negative, send a 0 instead-- squash it to 0.

So which means if the number is negative, nothing happens. If the number is positive, it breaks up.

So what happens is that you could have a very complicated linear function with millions of variables, and then it puts a single number, and that number, unfortunately, happens to be negative-- the ReLU is not impressed.

It's going to send a 0 out. It's a very simple function.

And many, many folks who have been in deep learning for a long, long time believe that the use of the ReLUs is one of the key factors that led to the amazing success of deep learning because it's got some very interesting properties, which

we'll get to, hopefully, on Wednesday. So the shorthand here is that whenever you see this thing, it's just a linear activation-- linear function followed by just sending it straight out.

If I put a ReLU in here, I'm going to denote it like that, which mimics the graph-- how it looks.

And if I put a sigmoid, I'm just going to use this thing here. Just a visual shorthand.

There are many other activation functions, by the way.

There's something called the tanh function, the leaky ReLU, the GELU, the swish.

I mean, it's like a menagerie of activation functions because very often researchers will be like, well, I don't like this activation function.

Here's a little modified version of the function, which is going to be better for certain things.

So people's research creativity is, at this point, has gone unhinged, so lots of options.

But if you just stick to the ReLU for your hidden layers, you can basically get anything done, practically. You don't have to worry about anything else.

So we'll only focus on ReLUs for all the intermediate stuff. Yeah?

AUDIENCE: Yeah, how do you gauge which activation function is more suited for your use case?

RAMA RAMAKRISHNAN: Yeah, so the rule of thumb here is that for your hidden layers, use ReLUs because empirically, we have seen that they do an amazing job. For your output layer-- your very final thing-- you actually don't have a choice because what you have to use depends on what kind of output you have to work with. If it's an output which is a probability number between 0 and 1, you have to use a sigmoid.

If it is, say, 10 numbers, all of which have to be probabilities and they have to add up to 1, you got to use something called the "softmax," which we'll get to on Wednesday.

So it really depends on the output, and the nature of the output dictates what you use in the output layer.

So coming back to this-- so if you want to design a deep neural network, the input is the input, the output is the output, and so you get to choose everything else.

You get to choose the number of hidden layers, the number of neurons in each layer, the activation functions you're going to use, and-- for the hidden layers-- and then you have to make sure that what you choose for the output layer matches the kind of output you want to generate. So this is all in your hands.

You decide what happens, but you will-- there's a lot of guidance for how to do these things, which we'll cover as we go along. Did you have a question? AUDIENCE: Kind of, but I guess I'll do it.

Is there also exploration in dynamic setting of layers so that our user is determining the layers that the system itself [INAUDIBLE] layers. RAMA RAMAKRISHNAN: Yeah, so there is a whole field called neural architecture search-- NAS-- where we can actually try a whole bunch of different architectures and then use some optimization-- and, in fact,

reinforcement learning, which we won't get to in this class-- as a way to figure out really good architectures for any particular problem.

But the question of, OK, when I'm training a model with a particular kind of data, the first pass through the training data, I'm going to use two layers-- the second pass, I'm going to do 7 layers-- that is not done.

And the reason it's not done is because of certain other constraints we have and how we can do the optimization and the gradient descent and stuff like that.

But what you can do-- and we will look at this thing called "dropout"-- for certain layers, you can actually, for each time you run it through the network, you can decide on this layer, I'm not going to use all the nodes.

I'm going to drop out a few other nodes randomly. And it's a very effective technique to prevent overfitting, and we'll come to that a little later on. Yeah?

AUDIENCE: One question regarding neural networks is about the coefficients.

If it's a small size, is it fine on its own? Or do we have as the coefficients [INAUDIBLE].

RAMA RAMAKRISHNAN: No, the whole trick here-- the whole name of the game-- is we use the training data and something called a "loss function," which I'll get to on Wednesday, along with an optimization algorithm so that the network figures out by itself what the weights need to be, what the coefficients need to be so as to minimize prediction error, and that's the whole thing. The magic here is that we don't have to do anything.

We only have to set it up, sit back-- often for many hours-- and watch it do its thing. Yep.

AUDIENCE: Just one last question. You mentioned nodes just now when you were answering Rolando's question. Can you just confirm exactly what a node is?

I have an idea that it's basically any circle, but you just added a lot more detail.

RAMA RAMAKRISHNAN: Sure. No, when I'm referring to a node, I'm literally referring to something like this, which think of it as a linear function followed by a nonlinear activation.

So it receives a bunch of inputs, runs it through a linear function, and pass it through, like, a ReLU or a sigmoid or something, and out pops a number.

So in general, a node will have many numbers potentially coming in but only one number going out.

Now, that one number may get copied to every node in the next layer, but what comes out of that particular node is just a single number.

All right, so let's use a DNN for our interview example. So in this problem, we had two inputs-- GP and experience. The output variable has to be between 0 and 1 because you have to predict the probability that someone will get called for an interview.

So the output size is fixed-- sorry-- the input size is fixed, the output is fixed.

So since it's really only the very first network we're actually playing with, let's just start simple. We'll just have one hidden layer, and we'll have three neurons. And as I mentioned to Tomaso's question from before, if you are choosing activation functions in the hidden layers, just go with the ReLU as a default.

It usually works really well out of the box, so we'll just use a ReLU.

And since the output has to be between 0 and 1, we don't have a choice.

We have to use a sigmoid for the output layer. That's it. So those are our design choices.

And when we do that, this how it's looked like. We have two inputs-- x1 and x2, GP and experience-- and then it goes through these three ReLUs, and then out comes these three numbers, and they pass through a sigmoid, and we get a probability y at the end.

All right, quick question-- concept check-- how many weights-- how many parameters, both weights and biases, does the network have? Let's take a moment to count.

All right, any guesses?

Yeah? AUDIENCE: 12. RAMA RAMAKRISHNAN: I think you're almost there.

Are folks going to be doing a non-binary search on this now? OK. AUDIENCE: Is it 9?

RAMA RAMAKRISHNAN: No. Yes? AUDIENCE: 13. RAMA RAMAKRISHNAN: Yes, very good. So that's 13.

And my guess is that the reason you came up with 12-- and I made the same mistake-- that's why I know it-- is so you probably forgot this green thing here.

So what folks often forget is the bias. We all count the things.

And the easy way to do it is say, OK, two things here, three things here, 2 times 3 is 6, 3 times 1 is 3, another 9.

And then you have to add up all the intercepts, so you get 13.

And so when we get to very complicated networks-- the first two or three times you work with very complex networks-- and we'll do it starting very soon-- just get into the habit of hand-calculating the number of parameters just to make sure you understand what's going on.

Once you get it right a couple of times, you don't have to do it anymore.

The first couple of times, hand-calculate to make sure you get it. So yeah.

So let's say that we have trained this network using techniques which we'll cover on Wednesday, and it comes back to you after training and says, OK, these are the optimal-- the best values for the weights and the biases that I have found.

So now your network is ready for action. It's ready to be used. And so what you can do is-- let's say that you want to predict with this network. If you have x1 and x2, what comes out of-- so what comes out of this top neuron? Let's call it a1. It's basically this.

That's what's coming out of this thing. For any x1 and x2, this is what's coming out, similarly for a2 and a3. And then what comes out at the very end is basically a1 times that, plus a2 times that, plus a3 times that, plus 0.05.

And the whole thing gets run through the sigmoid, and that's what you get.

So this slide and the one before, just make sure you look at it afterwards and-- to make sure you totally understand the mechanics of it because this is really important.

And if you don't fully understand, like, internalize the mechanics, when we get to things like transformers, it's going to get hard.

So just make sure it's, like, automatic at this point. It should be reflexive.

So, yeah, and so when you want to predict anything, just run some numbers through it.

You get all these things, and boom, you calculate it. It turns out to be 22.6. That's the answer.

All right, so I just want to say that-- let's say that you built this network and now you're like, hey, given any x1 and x2, I can come up with a y, but I'm feeling a little mathy. Can you actually write down the function?

Yeah, you can write down the function. This is what it looks like.

Super interpretable, right? So this goes to the comment that, you made earlier on, where the act of depicting something using this graphical layout makes it so much easier to reason with and to think about compared to trying to figure out what this function is doing.

The other point I want to make is that this contrast-- what we just saw with the logistic regression thing we saw earlier, which was this little function-- and so here, even this simple network with just three hidden layers-- sorry, three nodes in that single hidden layer-- it's so much more complicated than the logistic regression model-- so much more complicated.

And it is from this complexity springs the ability of these networks to do, basically, magical things.

And that's where the complexity comes from. That's where the magic comes from.

And here, in this case, the number of variables hasn't even changed. It's still only two.

But we can go from the two inputs to the one output in very complicated ways as long as we know how to train these networks the right way.

That's the secret sauce, which we'll spend a lot of time on.

So, yeah, to summarize, this is what we have-- it's a deep neural network.

By the way, this kind of network where things just flow from left to right is called a "feedforward neural network," in contrast to some other kinds of networks called "recurrent networks," which we won't get to in this class because transformers have actually proven to be much more capable than recurrent networks.

And those have become the norm, so we'll just focus on those instead.

And so this arrangement of neurons into layers and activation functions and all that stuff-- this called the "architecture" of the neural network.

And as you will see later on, the transformer-- the famous transformer network-- is just an example of a particular neural network architecture, much like convolutional neural networks, which we'll get to next week for computer vision, or another example of a particular network of the architecture. So we will focus on transformers. They are a particular kind of architecture.

All right, so in summary, that's what we have.

You get to choose the hidden layers and neuron's activation functions and stuff like that.

The inputs and the outputs are what you have to work with. And so we will actually take this idea and then use it to actually solve a problem from start to finish on Wednesday. So I think I'm done.

I give you three minutes back of your day. Thank you. [APPLAUSE]

I give you three minutes back of your day. Thank you. [APPLAUSE]