YouTube Video

okay hi everyone I'll be started okay so it's Tuesday of week two so hopefully that means everyone has done the

assignment one everyone done assignment one um you know uh I'm if I'm saying this I'm probably saying it to the wrong

people but it it seems like every year some people blow some of their um late days on assignment one and it's really

just the wrong place to use them um so yeah hopefully you've all done assignment one um and did not that this is meant to be the easy onramp um and

then we go straight on from that so that out today we have assignment two so assignment two um has two purposes um

purpose one is to make you do some math um to um some understanding of what newal networks really compute and how they compute it and that's what I'm

going to talk about today is also going through that math um but then simultaneously maybe it does three things in assignment two um so we're going to be learning something about

dependency paing which will be actually something about um language structure and Linguistics but then thirdly um for assignment two we're going to start

using pytorch so pytorch is is one of the leading software Frameworks for deep learning and the one that we're going to

um use for this class um so for p i mean the assignment 3 pytorch is exceedingly um scaffolded so it's sort

of you know here's this thing and you have to write these two lines use these two functions um but nevertheless um for help people get up to speed and get

started using pytorch on Friday at 3:30 in Gates bo1 or it will again be recorded um we have a tutorial on pytorch and so that's a great way to get

more of a sense of py torch and how it works before doing assignment two um yeah uh the other things yeah so

for nearly all the lectures we've got um further reading of places that you can look um of all the classes in the attire

um this um for many people might be a really good one to look at the suggested readings we have several readings which are sort of shorter tutorials and

reviews of the kind of um Matrix calculus um and linear algebra that we need for this class um so really encourage you um to look at those um if

you decide that one is your favorite you can tell us on Ed which one you think is the best one to choose between between them I kind of like the one that's first on the list but maybe you'll feel

differently um yeah um conversely um yeah so today will be sort of all math and then Thursday will be kind of all language and Linguistics some people

find the language and Linguistics hard as well um so I guess different kinds of people um okay so getting straight into

it um so where we started last time um I'd sort of shown these baby neural networks and sort of said well we can think of each of those orange things as

basically like a little logistic regression unit and the crucial difference from then the kind of um statistics machine learning you see in a

stats class 109 or wherever is that in those you have one logistic regression and you're defining the input features to it and you've got some decision

variable that you want to un have at the output here you're sort of building these Cascades of little logistic regressions and so the idea is right at

the end we're going to Define what we want we're going to capture that by our objective function or loss function but for the stuff in the middle that that

stuff in the middle is going to be a chance for the neural network to learn by itself what would be useful inputs to

further Downstream neurons what kind of functions should I come up with in terms of my inputs that will help me provide

um useful outputs to help the final computation down the track um and you know if you haven't sort of seen and um

thought about this much before I mean I think you know it's worth uh sitting with that idea for a moment CU this is really a super powerful idea which is

what's made neural networks more powerful in most circumstances than other forms of machine learning the fact that you have this

self-organization of intermediate levels of representation that you use to compute things that will be useful Downstream for what you eventually want to

do um the other reason I was bring back up this picture is I've sort of wanted to go straight from here um to matric

um so while you could sort of wire together neurons however you wanted to and arguably if you look at human brains they look more like neurons wired

together however you wanted to but for what's done with neural networks basically there's always this kind of regular structure of layers so once we

have this regular structure of layers we are taking um the output of one of our

neurons at one layer and we're feeding them together with weights um to produce um the inputs to the next layer so we're

taking the X1 X2 X3 outputs we're multiplying them all by weights we're adding a bias term um and then we're going to put it through a nonlinearity and that will

give us the value at the next layer so if we then kind of collapse that to a vector and this to a vector that then collapses into a computation

that first of all we're doing a matrix multiplication we're calculating WX of the inputs and then we're adding on the

biases as a a vector of biases which gives us this intermediate value Z and then we have this nonlinearity or

activation function which is applied um to that which gives us the values in the next layer of the new Network and the activation function is applied to a

vector and produces a vector but it's operating on each of the individual components of that Vector one at a time so we've got some Scala function that we're just applying to each element of

the vector um and so that's the kind of picture um we saw when I did this example and I'm going to continue to use

this example in today's class remember we were going to decide whether the word in the middle of the input window was a location or not and so we were doing the

matrix multiplication putting it through the nonlinearity we're then just doing a DOT product here and then we're going

that got stuck into um uh sigmoid um to predict yes or no um and the final thing I wanted to say a little bit about is

these FS the nonlinearity or the activation function and where did they come in well the starting point of where they came in in

the history of newal networks is when people came up with this idea that well you could represent um the operation of

a basic neuron by doing a matrix multiplication of the inputs and then having a bias term or here represents a

threshold term to see whether the neurons should fire or not that was actually in the very first implementation which B dates back to the

1940s as done as a threshold right so that if um the activation was greater than Theta you output one otherwise you

output zero and well if you have a threshold um the the two lines are flat right so there is no slope there is no

gradient um so that's actually makes learning much harder so the whole secret of what we build with newal networks and

in an alternative name that's um popular in some circles these days is gradient-based learning and the entire idea of gradient based learning is if we

actually have some slopes um then it's like going skiing during spring break you can work out where it's steeper and you can head down where it's steeper um

and that will allow us to op imiz our function to learn much more quickly and so that's one reason that we don't just

want to have threshold units we want to have things with slopes so we um have gradient so in subsequent work people

started using activation functions with slopes um and so the first popular one um was this sigmoidal logistic that

we've seen for mapping to probabilities but you know it's sort of imperfect it seemed because you know the output was always non- negative so that sort of

tends to push things towards bigger numbers so um there was quite a bit of use then of this tan H function um and

you'll actually see tan H when we do assignment three we'll be using tan H's in our current new networks um and so um

I've written there um the formula usually give for Tan H in terms of exponentials um yeah if your math is Rusty it's not obvious that tan H and

logistic have much to do with each other um but if you want to treat this as a math problem um that a 10 H is literally just a rescaled logistic you're

stretching it by two and moving it down by one it's the same function um okay um but you know so that's nice

but if you're calculating 10 hes you have to do all of these exponentials and you know exponentials are kind of slow on your computer and things like that you might wonder whether you couldn't

get away with something much cheaper and so people thought about that and thought oh maybe we could just use a so-called hard tan H um where it has a slope of

one in the middle and is then just flat outside that area and you know that seemed to work in many cases and so that then led to the popularity of the

rectified linear unit um so the rectified linear unit is simply zero on the negative region and then is yal x in

the positive region now this seems kind of wonky and goes against what I was saying about gradient based learning because once you're in the negative

region there's no gradient um you're just dead um but in the positive region there is gradient and the gradient is

particularly simple right the slope is always one um and so you know this still feels slightly perverse to me but you

know this really became the norm of what people use for a number of years because people found that although for an individual neuron it was dead half the

time anytime it went negative that overall for your newal network some things would be alive so it kind of gave sort of a form of specializ ation and

the fact that the slope was always one meant that you got um really easy productive backward flow of gradients in

a way we'll talk about later and so um learning with realu turned to be out to be very effective and people started using the realu nonlinearity everywhere

and it sort of became the default in the norm and you'll see us using it in the assignments in particular we use it in assignment to and so get to see that it

works but nevertheless at some point people sort of had second thoughts and decided you know having a dead over half of its range maybe isn't such a good

idea after all even though it seemed to work great for a few years and so a lot of what's happened since then is then to come up with other functions which are

in some sense Ru like but not actually dead um so um um okay I don't really yeah uh here we go

enough so one one version of that is the socalled Leaky value so for the Leaky value you make the negative half a straight line as well with a very minor

slope but still it's got a little bit of slope um there is then a variant of that called the parametric value where you have one extra parameter which is actually what the slope of the ne the

negative part is and people showed some positive result with that um more recently again and this is what you often see in recent Transformer models

um is you see um nonlinearities like Swiss swis and Jello so both of these are sort of fancy functions but kind of

what they both look like is basically this is yal X to all intense and purposes not quite but approximately and then you got sort of some funky bit of curve down here which again gives you a

bit of slope um it's sort of the curve is going the opposite way that's sort of a bit funny but they seem to work well commonly used um in recent Transformer

models um so you know that's a bit of a dump of all the nonlinearities people use I mean the details of that aren't super important right now but the

important thing to um have in your head is why do we need nonlinearities and the way to think about about that is that

what we're doing with neural networks is function approximation there's some very complex function that we want to learn you know like maybe we want to go from a piece of text to its meaning or we want

to be interpreting visual scenes or something like that um and so we want to build really good function approximators and well if you're just

doing Matrix multiplies a matrix multiply of a vector is a linear transform so that doesn't let you multiply complex functions I guess strictly if you put a bias on the end

it's then an aine transform but let's keep it simple linear transforms right so if you're doing multiple if you're doing multiple Matrix multiplies you're doing multiple linear transforms but

they compose so you could have just multiplied these two matrices together and you'd have a single linear transform so you get no power in terms of

representation by having multi-layer networks that are just Matrix multiplies you know as in a little aside in terms of representational power having

multi-layer Matrix multiplies gives you no power but if you think about in terms of learning actually it does give you some power so in the theoretical Community looking at newal networks

there actually quite a few papers that look at linear newal networks meaning that they're just sequences of the multiplies with no nonlinearities because they have interesting learning

properties even though they give you no representational power um okay but we'd like to be able to learn functions like this not only functions like this and to

be able to learn functions like this we need more than linear transforms and we achieve those by having something that makes us be calculating a nonlinear

function and it's these activation functions that give us nonlinear functions okay cool

um okay so then getting on to today so the whole thing we want to do now is gradient based learning right this is our stochastic gradient thecenter

equation where here you know that upside down triangle symbol right that's our gradient we're wanting to work out the slope of our objective function and so

this is how we're going to learn by calculating gradients so what we want to know is how do we calculate the gradients for an arbitrary function and

so what I want to do today is first of all do this by hand um for math um and then discuss you know how do we do it

computationally um which is effectively um the famous thing that's taken as powering um underpowering all of neuron Nets which is the back propagation algorithm but the back propagation

algorithm is just automating the math okay and so for the math it's Matrix calculus and at this point then there's a huge Spectrum um between people who

know much more math than me and people who barely ever learned this um but you know uh I hope to explain the essentials um or remind people of them

um enough that you're at least at a starting point um for reading some other stuff and doing homework too so um let's get into that and so I'm going to spend

about half the time on those two halves um and you know the hope is that after this you'll feel like oh I actually understand how new networks work under the hood fingers crossed

okay so here we go so if you're a Stanford student um you maybe did math 51 um or else you could have done math

51 which teaches linear algebra multivariate calculus and modern applications um math 51 covers every everything I'm going to talk about and

way more stuff so if you actually know that and remember it you can um look at Instagram for the next 35 minutes um but I think the problem is um that you know

quite apart from the fact a lot of people do it as Frost um you know this is a lot to get through in 10 weeks and I think that a lot of the people who do this class sort of by two years later

don't really have much ability to use any of it um but you know if you actually looked at this book really Harden for a very really long time you would have discovered that actually

right towards the end of the book in appendix G um there's actually an appendix on newal networks and the multivariable chain roll which is

precisely what we're going to be using um for um doing our neural networks um but there are only two problems one problem is that this is Page 697 of the

book and I'm not sure anyone ever gets that far and the problem is even if you do get that far you know I don't know I find these pages that they're really

dense texty Pages it's not even easy to understand them if you have gone there so here's my attempt on that um so the Mantra to have in your head is G if I

can remember basic single variable calculus you know that I've got 3x squ and the derivative of that is 6X that's all you sort of need to know right the

ENT is multivariable calculus is just like single variable calculus except you're using matrices okay so that's our article of

faith and we're going to do that and so what we're wanting to do is to do Matrix calculus or the generalization of that tensor calculus sort of using um vectors

matrices and higher order tensors because if we can do things and what's referred to as vectorized gradients in the neural network world that that will

be sort of the fast efficient way to do our operations you know so if you want to think it all through you can do it single variable at a time and check that you're doing the right thing and I sort

of tried to indicate that in the first lecture but if we want to have our networks go room we want to be doing Matrix

calculus okay so let's work up to doing that okay so this is the part that I I trust everyone can remember right so we

have f ofx = x cubed and we can do single variable um derivative and the derivative is

3x² everyone remember that one okay that's something we can all start from and remember this derivative is saying the slope of things right so the slope

of things lets us work out where is something steep so we'll be able to go skiing right that's our goal right and

so you can think of the slope of things as how much the output will change if we change the input a bit right that's our

measure of um steepness right so um that so since the derivative is 3x^2 if we're at x = 1 that means the slope is about 3

* 1 S 3 so if I work out the value of the function for 1.01 it's gone up by about three times point I move the X by

0.01 and the output moved by 0.03 where if I go to x = 4 the derivative is 3 * 4^2 is 48 and so if I work out the value

of the function at 4.01 I get approximately 64.4 versus 64 right that small difference from 4 to 4.01 has been

magnified 48 times in the output okay okay so now we just uh remember the Mantra it's going to be exactly the same

single value calculus um but with more stuff so if we have a function with n inputs we're then going to work out its

gradient um which is its partial derivative with respect to each input so its gradient will now be a vector of the

same size as the number of inputs um and there's this funky symb um which people pronounce various ways I mean you know this kind of originated some kind of someone's weird way of

drawing a calligraphic d right so it is really a D um so I think I'll mainly just call it D but sometimes people call it partial or funy D or some some other

name right so you have DF dx1 DF dx2 for each of the variables okay so if we go

beyond that um and then have a function with um n inputs and M outputs what we

then get for um the gradient is what's referred to as the Jacobian now actually um the dude this is named after was a

German Jew so it should really be yakobi um but no one says that in this country um Jacobian um okay so the Jacobian is

then a matrix of partial derivatives um where you're working out for each output and each input the

partial derivative between the component of the input and the output so this looks like the kind of thing that we're going to have when we have a neural

network layer because we're going to have um n inputs and M outputs for the two layers of our neural networks so we'll be using these kind of

jacobians okay um so then you know the whole idea of newal networks is we've got these multi-level computations and they're going to

correspond to composition of functions so we need to know how to compose things both for calculating functions and for

calculating their gradients so if we have a one variable function and we want to um work out um its derivative in

terms of a composition of two functions what we're doing is multiplying um our computations okay so um if you compose

together um Z of Y um that's the function that we did at the beginning that gives you oh was it no it's not sorry it's different okay Z of Y gives you

3x2 right and so we know that the derivative of that is um 6X okay if we do it in terms of the

pieces we can work out DZ Dy um which is just going to be three and um Dy DX is 2X and we can work out

the total derivative by multiplying these two pieces and we get 6X the same answer right um so um Matrix calculus is

exactly like single variable calculus except we're using um tensors of different um so the word tensor is used to mean as you go up that Spectrum in in

its size so from sort of Scala to Vector to Matrix to then you know what in computer science is normally still uh is

multidimensional arrays um that spectrum is then tensors of different um Dimensions okay so um when we have

multiple variable functions we're going to multiply jacobians so here we have a function WX + B and then we um compose

um the nonlinearity F to get H and so we're going to be able to compute that in the same way as a product of these

partial derivatives which are jacobians okay so let's start looking at a few examples of what we get so let's

count with start with an element wise activation function so when we have a a vector that's being calculated as the

activation function of a previously computed quantity well we're Computing that component wise as I explained

before so hi equals F of Z and where the sort of f is our activation function that actually applies to a scalar but you know overall this layer is a

function with n outputs and N inputs and so it's going to have an N by n Jacobian and well what that's going to so this is

our definition of the Jacobian um but in this case this is sort of a special case

because if I equals J um then we're going to have um the output um

J the HJ depending on Z and otherwise it's going to be zero CU for the off diagonal entries it doesn't matter how you change the value it's not changing

the output because the output only depends on the corresponding index and so what we're going to get for this Jacobian of activation functions is a

matrix where everything is zero apart from the diagonal terms um that correspond to where we're calculating the activation function and for those

ones we're going to have to work out how to compute the derivative of our activation function that was on assignment one one of the questions on

assignment one I do believe or or was it on assignment two no no it's assignment two one of the questions on assignment two I got that wrong one of the ones on the new assignment is say hey um can you

work out um the derivative of a logistic function well then we'd be a to plug that straight into um fpre so I'm not going to give that answer away today um

okay so um other things that we want to do um with uh Jacobian is well we have this um layer of our neural network

where we're um calculating WX + B and we can want to work out the partial derivative of that with respect to X um you know this is the kind of place where

it actually works to remember the Mantra and say Matrix calculus is just like single value variable calculus but with matrices so if you just don't use your

bra too hard and think oh it's just like single variable calculus so what should the answer be it's obviously going to be W right and indeed it is um similarly if

we want to do the same thing for wxb and work out the partial derivative with respect to B um well that would be one in terms of single variable calculus and

so in Matrix calculus that becomes an identity Matrix okay slightly different the same idea um but that's reflecting the fact that b is actually Vector so we

need we need it to be coming out um as an identity Matrix um okay so um higher

up in my example picture I did this sort of vector um do product of UT um and well what happens if we work

out the um the um to C in of that what we end up with strict ly is we come out with

HT um and you know this is sort of like when you're working out um well we did this in the first class right when we did a a DOT product calculation that you

kind of get for each individual element you get the opposite term and so you get um the other Vector coming out um these are sort of good ones to comput it at

home for practice to make sure um you really do know the answers and why they work out the way they do okay um so let's go back to our

little neural net um this was most of our neural net up above our neural net um there was the nonlinearity now I'm going to uh leave

that out of this time oh see I got it wrong it's on assignment two um but you know normally you'd be C calculating the partials of the output the loss function

with respect to the inputs but since the loss function is On Assignment two I'm going to leave that out and I'm just going to calculate derivatives with

respect to this score that feeds into the loss function so we first we got um the newal network layer the nonlinearity and then we're doing this dot product to

work out a score for each position which feeds into the logistic function so if you want to work out ddb um so that's with respect to the

bias for first so the way we do it is you know we break up our equations into our individual pieces that are composed together and so that means we break this

up so we first calculate the zal WX + B then we apply the activation function to the different components okay then after that

um what we remember to do is okay to work out um our um partial derivatives

of B of s with respect to B that what we're going to be doing is doing the product of um the partial derivatives of

the component pieces so we're applying um The Matrix calculus version of the chain rule so dsdb equals D sdh * DHD Z

* dzb um and which corresponds to these three layers that are composed together and so at that point um we remember our

useful jacobians from the previous slide and we can just apply them so the top

one um DSD is um the you transpose or else or maybe it's you let's come back to that there's a fine point on that that I will explain more

about later um um okay um then for the DH DZ that was the activation function where we got the

diagonal of um the derivative of f of Z and then for dzdb that's where we got the identity function okay um so we can

simplify that down and so what that's going to end up as is the UT transpose that funny symbol there times the um the

vector element y's um derivative of f um this symbol which doesn't normally turn up um

in your regular math course but turns up all the time in New networks is referred to as the Hadad product and the Hadad product is meaning element wise

multiplication so it's not like a cross product where you put two vectors together and you get out one number of Scala you put two vectors together you element wise multiply them all and

you're left with another Vector of the same type okay so that so now this gave us are working out of the partials of dstb

and for a neural network um we want to work out all the other partials as well

so overall here in the picture right we had the X the W the B um and the U and

we'd like to work out partials with respect to all of those variables so we can change their values and learn so

that our model predicts better um so um so suppose we now want to um calculate ddw so again we can split it up with the

same chain Rule and say ddw equals the product of these three things and the important thing to notice is that two of those three things were

exactly the same ones that we calculated before the only bit that's different is that at the end we're now doing DZ DW rather than

dzdb and so the first central idea that we'll come back to when we do computation graphs is oh we really want to avoid doing repeated work so we want

to realize that those two parts of things are the same and since we're just sort of multiplying these um partial derivatives together right we can just

compute what that part is and reuse it and so if we want to um wait yeah okay

so if we're wanting to calculate ddw the part that's the same this part here we can refer to as Delta so Delta

is sort of the Upstream gradient or the error signal the part that you've got from sort of starting at the beginning DS DH DH DZ this sort of shared Upstream

part we can calculate that once and then we can use it um to calculate both of these two things and for dsdb because

the dzdb just comes out as the identity Matrix um the answer is just Delta but for D

sdw we need to work out the DZ DW before we're finished okay so what um does what do we

get for that last piece so one question you might start off with and is normally a good thing to think about when you're doing assignment problems on this and other things is the first thing to think

about is you know what do things look like like am I should the answer be a vector a matrix uh what size should it

be and things like that so for ddw um W is an N bym Matrix

um and S is a scalar so therefore since we have one output and M * m inputs the

answer according to math should be that we've got a one by n * m Jacobian I a

big long row Vector um but here's where um things get a teeny bit tricky um and

there's sort of we end up with this weird mess of um math and Engineering convenience because you know immediately

what we're wanting to do is we're wanting to take our old parameters which will be in stored in the form of matrices vectors and so on that we're

using as coefficients and we're going to want to subtract from them um our you know a fraction of our calculated

gradient so what we'd like to do is have our um calculated gradients in the same shapes as our parameters because then we can just do subtraction whereas if

they've turned into a God Almighty row Vector um that's not quite so convenient um so it turns out that what we end up

doing um is using something that gets referred to as the shape convention that um we uh uh

reshape our um jacobians so they fit into things that are of the same shape as the parameters that we are using so

we're going to represent um DSD W as an N bym Matrix laid out as follows and that's a place that one people can get

confused okay so that's what we want to calculate that kind of Matrix but um and so that Matrix is going to be Delta * DZ

DW so Delta is going to be part of the answer and then we want to know um what DZ DW is um and the answer is going to

be it's going to come out like this so D sdw is going to be um delta T * XT so it's going to be the product of the Upstream gradient which was the same

thing we calculated before for the other two quantities and then a local inputs symbol which um is input signal which is

here coming out to XT okay um and you know so we're taking the transposes of those two vectors which it means that we end up

calculating an outer product of those two vectors um which gives us our gradient um and so why is that the right answer well you know it kind of looks

convenient CU that's giving us something of the right shape um for what I was arguing we want to find out and we have the right number of terms um now I'm

going to rush through this so I encourage you to read um the uh lecture notes and do this more carefully but um let me at least a little bit explain why

it makes sense right so um if you think of one weight um in so all of these connections are our Matrix right The Matrix is being represented by all these

lines and in your network so if you think of one number in The Matrix so here is w23 so it's connecting from input 3 or

it's multiplying input three to give part of the answer of H2 right so it's this line here um so

for this line here um this weight is being used only in the calculation of H2 and the only thing it's dependent on

is X3 um so that if you're then wanting to work out the partial

of um h two um or Z2 sorry yeah um sorry yeah

sorry Z2 the partial of Z2 with respect to X3 it's sort of depending on these two pieces only and that's what you're

achieving um by working out um the sort of outer product like that okay um yeah so let me just come

back um one more time to um this the sort of question of the shape of derivatives

um you know so I already sort of fudged it um when I was sort of um talking about oh should I put the the transpose

there or should I nod and get a row vector is a column Vector um so there's sort of this disagreement between

whether you kind of have the Jacobian form which is what actually makes the chain roll work right in terms of doing multiplication um versus the shape

convention which is how we store everything for our computations and Mak doing stochastic gradient descent where

you're um subtracting um whatever kind of tensor you have easy um so um you know this can be a source of

confusion um since we're doing a computer science course for the answers in the assignment we expect you to follow the shape convention so you know

if you're working out the derivatives with respect to some Matrix it should be shaped like a matrix with the same parameters um but you know you may well

want to think about Jacobian forms and Computing your answers I mean there are sort of two ways to go about do doing this one way of doing it is to sort of

work out all the math using Jacobian Al math 51 and at the end just to reshape it so it fits into the same shape as the parameters according to our shape

convention I mean the other way is to sort of do each stage following the shape convention but then you sort of have to be game to sort of reshape

things as needed by sort of doing um transposing to have things work out at the different stages okay that was my attempt to quickly review the

math most people are still here um I will now go on to the second half and go on to the um How We Do the computation

right so you know so most of um yeah so the famous thing that powers new networks is the back propagation algorithm so the back propagation

algorithm is really only two things you know its invention made people famous because it gave an effective learning algorithm but you know at a fundamental

level the back propagation algorithm is only two things thing one is you use the chain rule you do calculus of complex

functions and Thing Two is um you store intermediate results so you never recompute the same stuff again that's all there is to the um back

propagation algorithm and so let's just go through that so if we're computationally wanting to deal um with

you know functions and doing back propagation we can think of them as being represented as a graph and in some

way or another um this kind of graph is being used inside your newal network framework so here is here is a re- representation of my little neural

network for finding whether the word at the center is a location so I'm taking the X Vector input I'm multiplying it by W I'm adding B to it I'm putting it

through the nonlinearity and then I'm um doing the dot product with my Vector U right so that was my computation and so the The

Source nodes are the inputs in this graph the interior nodes then the operations I do um and so then the edges

that connect those together then pass along the result of each operation so I passed along WX to the addition function

with B then I that gives me Z that I pass through the nonlinearity which gives me H which I then um dot product with the U to get S okay so I do

precisely this computation and this is referred to as Ford propagation or the forward pass of a neural network so um

the forward pass just calculates functions okay but then once we've done that what we want to

do is then work out gradients so we can do gradient based learning and so that part is then referred to as back

propagation or the backward pass and then we run things backward so for running things backward we're going to use the same graph and we're going to

backwards pass along at gradients and so we start at the right hand side and we have dsds so dsds is just one because

you know um if you change S you've changed s and then what we want to do is sort of then work further back so we can

work out DSD ddz dsdb ddw dsdx as we work back um and so this is um the what

we want to work out with gradients um and so how are we going to do that well if we look at a single node so for

example our um our nonlinearity node but any node where H equal F ofx what we can

have is an upstream gradient um DS DH and what we want to do is calculate the downstream gradient of the next variable down the D

sdz and the way that we're going to do that is we're going to say well let's look at F what is F's gradient and

that's going to be our local gradient and then this is immediately what gives us the chain rule that D sdz is going to be the product of

our Upstream gradient D sdh times the DHD Z the local um gradient that we calculate at that node so Downstream

gradient equals um Upstream gradient times local gradient

um oh yeah that's what that's what it says when I press um that again okay so this is the sort of the um single the

single input single output case though those inputs might be vectors or matrices or something like that um we then have sort of more complex graph

cases um so I think I should have retitled this SL oh yeah so still so sorry so the next case is for our node it might have

multiple inputs so this is where we're calculating WX so in that case we still have an up we have a single Upstream gradient and

then what we're going to do is we want to calculate the downstream gradient with respect to each input and the way we're going to do that is we're going to

work out the local gradient with respect to each input and then we're going to do the same kind of multiplication of

Upstream gradient times local gradient with respect to each input again um chain

rule okay um so here's a little example of this so I'm this isn't really uh the kind of thing you normally see in a

neural network but it's an easy example so F of XYZ is going to be x + y * the

max of y z and we've got current values of X Y and Z of 1 2 and z respectively so here's our little computation graph

um and so for forward propagation you know we're going to do this addition we're going to do this Max function and then we're going to multiply the two and

that gives us the value of f um so we can run that with the current values of X Y and Z and this is what we get so the

max of two and 0 is two addition is three the answer is six okay so then after having done that we run the

backward propagation and yeah so this procedure you know is not actually special to new networks right you can use it for any piece of math if you want to just run your math um on pytorch

rather than um working it out in your head or with Mathematica um okay so now we work out um backwards so we want to

know the local gradient so d a DZ is going to be one sorry I said that wrong d a DX is going

to be 1 so a = x + y d a d y = 1 um for the max function that's going to depend on which of the two is larger because

it's going to have a slope of one for the one that's the biggest and zero for the one that's the smallest um and then for the product that's like what we saw

with vectors that dfda is going to be B and D FDB is going to be a um so those are all our local gradients and so then

we can use those to calculate out the derivatives so dfdf is one um we then multiply that by the two um local

gradients um that are calculated um for A and B so that gives us um two

and three where you're swapping over the numbers then for the max um that we're

having the one that is biggest um we're taking the Upstream times one so it gets three the other one gets zero and then

for the plus we're just sending the gradient down in both directions and so both of them come out as two um and so

that gives us DF DX so the final function value is two D fdy we're taking the three and adding the two I'll mention that again in a

minute which gives us five and then DFD Z is zero um and we should be able to again be able to quickly check that

we've got this right right so um if we consider you know the slope around um Z

as you change z a little so Z is Z if we make Z 0.1 that makes absolutely no difference to what the computed function value is

um so the gradient there is zero that's correct um so if I change up the Top If I change x a little bit right if I

change X to 1.1 then I'll be calculating um 1.1 + 2 is

3.1 um and then I'll be taking the max which is two and I'll be calculating

5.1 um and so wait no I did that wrong oh times two wait I didn't do the multiplication

right um sorry yeah so we get the 3.1 that's multiplied by two that gives us 6.2 so a change of 0.1 in the X has

moved things up by 2 so that corresponds to the gradient being two and so then the final case is well what if we change

y to um so y started off as two and made it 2.1 then we're going to get 2.1 multiplied by 1 is

2.1 61 6.5 and right and then we've got the 2.1 here the oh sorry I keep doing this

wrong 2.1 + 1 = 3.1 and then we've got 2.1 as the max so we've got 2.1 *

3.1 and that comes out to be 6.51 so it's approximately gone up by so our 0.1 difference has gone up to

approximately 0.5 this is just an estimate um and so that correspond to the gradient being five right we get

this five times multiplication of our changes okay um and so that that fact that illustrates the fact that the right

thing to do is when you have outward branches in your um computation graph and you're running the um back

propagation that what you do is you sum the gradients right um so that for this

case we had y being um the Y is sort of going into these two different things in our previous chart so once we've worked out the Upstream gradients we sum them

to get the total gradient and so that's what we did back here we had two outward things and we sort of took these calculated Upstream gradients of two and

three and we just summ them to get five and that gave the right answer okay um and so you can think about that

um for the sort of just generally how the sort of things to think about as sort of gradients move around in these

pictures so that when we have a plus operation that um plush just sort of distributes gradient so the same gradient that's the Upstream gradient

goes to each input um when you have a Max it's kind of like a router of gradient so the max is going to send the gradient to one of the

inputs and send nothing at all to the other inputs um and when you have a multiplication it's a little bit funky because you're sort of um doing this

sort of switching of the forward coefficient so you're taking the Upstream gradient multiplied by the opposite um for coefficient gives you

your um Downstream gradient okay um so we kind of have this systematic way of being able to sort of

forward pass calculate the values of functions then run this backward to work out um the um gradients heading down the

network and so the main other thing of the back propagation algorithm is just that we want to do this efficiently so the wrong way to do it

would be to say well gee I want to calculate dstb ddw dsdx ddu so let me start doing those one at a time and when I've done them

all I will stop because that means if you first calculated dstb you do all of the part that's in blue um but then if

you went on to DS DW um you'd be calculating all the part in red and well just as we saw in the

math part when we were doing it as math um these parts are exactly the same you're doing exactly the same computations so you only want to do

those that part once and work out this Upstream gradient or error signal that is being then sort of calculated and is then being shared so the picture that we

want to have is you're doing together the Shar part and then you're only sort of doing separately the little bits um that you need to

do okay um boy I seem to have been rushing through today and I'm going to actually end early unless anyone is going to slow me down but I did have uh

just a few more slides um to go through um yeah so the sort of generalization of this as an algorithm

is you know in the general case you know so we normally have these sort of neural network layers and matrices which you can represent as vectors and

matrices um and you know it's sort of nice and clean and it looks like um doing that in Calculus class I mean strictly speaking that isn't necessary

so the algorithm for forward propagation and backward propagation that I've outlined that you can have it work in a completely arbitrary comput ation graph

providing it's a dag that doesn't have Cycles in it um so the general algorithm is well you've got a whole bunch of variables that depend on other variables

there's some way in which we can sort them so that each variable only depends on variables to the left of it so that's referred to as a topological sort of the

outputs and so that means there's a way we can do a forward pass where we're calculating um variables in terms of ones that have already been calculated

but you know if we want to have some extra wonky AR so it's not like nice Matrix multiplies or anything we're totally allowed to um do that or we can

have things not fully connected right so there's no connections across here right we can have an arbitrary computation graph um and so that gives us our

forward propagation and then once we've done the forward propagation we can initialize the output gradient as as one

and then we're going to visit the nodes in reverse order and at for each node we're going to compute a gradient by using the Upstream gradient and the

local gradient to compute the downstream gradient and so then we can head back down the computation graph and work out all of the downstream gradients and so

the crucial thing to notice is that if you do it correctly um that working out

um the the gradients has the same bigo complexity as working out the forward calculation right so that if you're

doing more you know in if if in terms of Big O terms right you might have different functions depending on what the derivatives are but in bigo terms if you're doing more work in the backward

path than you're doing in the forward paths that means that you're somehow failing to do this um efficient computation and that you're recomputing

some of your work okay um so because we have such a good algorithm here you should be able

to just work out the backward path automatically and that that gets referred to as automatic differentiation

so if you had the symbolic form of what you're calculating with your forward pass um you should just be able to say

yo computer um can you work out the backward pass for me and you know kind of mathematical like it could look at the symbolic form of all of your

functions um work out their derivatives and do the entire thing for you um so early on there was a

pioneering um deep learning framework theano principally from the um university of Montreal which attempted to do precisely that that you had the

entire forward path computation started in symbolic form and it just did the entire thing for you and worked out the backward pass

automatically um but you know somehow that sort of proved to be um too heavyweight or um hard to deal with

different things or people just like to write their own python or whatever it is um so that idea did not fully succeed

and so what in practice all of the current main Frameworks have fallen back on is something that's actually less automated than that so it's sort of like

we've gone backwards in time but the software's g a lot better really it's a lot staer and faster um so all of the modern deep learning Frameworks sort of

say look I will manage the computation graph for you and I can run the forward propagation path and the backward propagation path but you're going to

have to work out the local um derivatives yourself um so if you're if you're putting in a layer or putting in

um you know a function like an activation function in the in a neural network your class your python class that represents that you're going to

have to tell me what the forward um computation is and what the local gradient is and I'm just going to call your local gradient and assume it's

correct um so there's a bit more that has to be done manually so so the part that's automated then um is that you

know when you know precisely this code obviously but roughly you know inside the Deep learning software um it's Computing with a computation graph and

it's got a forward and a backward and it's doing what I presented on the pictures before so for the forward um pass it's topologically sorting all the

nodes of the graph and then it's going through them and for each node in the graph it's calling its forward function which will be able to compute its local

value in terms of its inputs which have already been calculated because it's topologically sorted and then it's um running the backward pass and in the

backward pass you're reversing your topological sort and then you're working out um the gradient which is going to be the multiplication of the Upstream error

signal times your local gradient and so what a human being has to implement um is that for anything whether it's a

single gate here's a multiply gate or a newal network layer you have to implement a forward pass and a backward pass so here for my baby example since

we're just doing multiplication my forward pass is that I just multiply the two numbers and um return it so I'm specifying that for the local node and

then the other part is that I have to work out those gradients and well we sort of know how to do that because that's the examples that we've been

doing here um but notice that there's sort of a trick right for what I've got Now you kind of can't write down what

the gradients are cuz you know what these cuz you know backward is just taking as an input the Upstream gradient and you can't work out what the

downstream gradient are going to be unless you know what function values you're calculating it at um so the standard trick that all which is how

everyone writes this code is you're relying on the fact that the forward is being calculated before the backward and so your forward method um shoves into

some local variables of the class what the values of the inputs are and then you have them available um so when you get to the back backward pass you can do

what we did before um that um the DX is going to be the Upstream error signal times the opposite input and um and

similarly for Dy and that's going to give us the answer okay um just two last things then

to mention yeah so doing this um your you need to write um you need to get the math right for what's the

derivative of your function so you get the right backward calculation so the standard way to check that you've got

the right backward calculation is to do manual gradient checking um with numeric gradients so the way you do that um is

you sort of like for the couple of examples I did when I said oh let's check it by for going from 1 to 1.1 what should the slope be approximately we're going to do

that in an automated way and so we're going to say at the value X let's estimate what the gradient should be and the way to do that is to pick a small H

there isn't a magical number because it depends on the function but typically you know for neural networks around 10us 4 is good um a small H and work out the

function value I forward part at x + H and x - H divided by the run which is 2 H and that should give you an

estimate of the slope what the backward pass is calculating and you want those two numbers to be approximately equal you know within some 10us 2 of each

other and then probably you're calculating the gradient right and if they aren't equal um that um you

probably have made a mistake um yeah so um note that this formul for the version I did for my examples I just compared to

x with x + H right I did a one-sided estimate which is normally what you get taught in a math class if you're doing this to check your gradients numerically

you're far far better off doing this two-sided estimate because it's much more accurate and stable when you're doing it equally around both sides of

your H um yeah so this looks easy to do um if if this was just so good why doesn't everyone do this all the time and forget about calculus um you know

the reason you don't want to do this is that doing this is incredibly slow right because you have to repeat this computation for every parameter of your

model that you're not getting the kind of speed UPS your getting from the um back propagation algorithm but you know it's useful for checking your implementation is correct you know in

the old days before Frameworks like py torch um you know we used to write everything by hand and people often got things wrong um but nowadays you know it's less needed but it's good to check

that if you've implemented your own new layer that it's doing the right thing okay um yeah so that's everything that we need to know about new Nets

propagation is the chain rule applied efficiently forward pass is just function application backward pass is

chain rule applied inefficiently um so you know uh we're going to inflict pain on our students by making them do some

math and calculate some of these things and um do the homework and I know that'll be harder for some of you than

others um you know that in some sense you don't actually need to know how to do this the beauty of these modern deep learning Frameworks is they'll do it all for you they predefine common layer

types and you can just plug them together like pieces of Lego and they'll be computed right and this is precisely the reason that high school students across the country and the world can now

do deep learning projects for their science fairs because you don't actually have to understand any of this math um you can just use what's given to you um

but you know um we kind of uh want to hope that you actually do understand something about what's going on under the hood and how new networks work so

therefore we make you suffer a little bit and of course you know if you sort of wanting to look at and understand more complex things you need to have

some sense of what's going on so later on when we get on to a current new networks we'll talk a bit about things like exploding and Vanishing great and if you want to have some understanding about well why things

aren't working and things are going wrong um then you sort of want to know what it's actually calculating rather than just thinking it's all a black box

magic and so that's why we hope to have uh taught something about that okay I think I'm done if the audience is

sufficiently stunned um and we can stop for today okay thank you