Section 1.1 of Pattern Recognition and Machine Learning - Polynomial curve fitting

okay all right let's get started so um we're going to be talking about section 1.1 which is on polom curve fitting um so this is just a setting where we have

some um scalers some input scalers some output scalers and we're going to be trying to fit um a curve to that data and the reason why we're talking about this really simple example is actually

highlights one of the great aspects of this book where um which is kind of not about just introducing kind of a list of you know cookbook list of approaches ways of fitting data you know ways of um

testing this testing that but actually trying to say um trying to kind of fit them all into kind of a a holistic approach of how to approach you know regression problems here or classification problems and why we're

doing the things we're doing so I think uh this chapter is really useful because it highlights in kind of a simple setting many of the issues that we're going to be seeing throughout um the book and dealing with in more

sophisticated ways later so it's it's cool for that um so let's get started so uh the setting is you know we have a real valued input real value Target um

and we're going to try to predict the function that goes from the inputs to the targets now um one aspect that um he kind of highlights in passing talks about in passing but I think it's really

important um is this notion that in this example but I think also in real life it's very important to this we're going to be working with synthetically generated data first so the idea is that you know we're going to have some

approach some curve fitting approach some classification approach and many people when um they're faced with problems like that they'll just take their you know technique that they want to apply and they'll just apply it to

the real data and we'll get some answer um but you will have no idea whether that answer was um correct and you won't and maybe you know maybe your package is right but you haven't set up your data

correctly um or vice versa or you've done both incorrectly some number will come out and it's it's it's often not clear that your proce procedure you've executed the procedure incorrectly and

so it's really really important whenever you're you know doing kind of data analysis um technique when you're applying data analysis techniques or you know running any new procedure to run it

on synthetic data first where you know what the answer is and to just make sure that every aspect of your of your pipeline you know data preparation all the way through to the

you know data analysis data reporting all of that is working properly and the way to do that is to generate ground truth data and that's you know he calls it synthetically generated data uh in our case it's going to be the data is

going to be generated from sin 2 pi of X so we know you know what the what the true function is that generates the data and we're going to try to see you know how well we do uh in comparison to that so it's really important this this

notion of synthe synthetically generated data is really important really useful and should always be used um when we're when you're doing that when you're doing data analysis you know as much as possible

it's just essenti it's essentially how you debug kind of dat data analysis pipelines okay um so let let's let's get going okay so now we've got a training Set uh so we've we've observed these

scalers you know at X different at n different values so here we are packed into um a vector and now we observe the target values um and we're we're trying to determine the fun we're going to pretend we don't know that it's S 2 pi

of X we're going to try to go from you know fit some function that takes from the input to the output so let's kind of have a look I think there's you know here's an example

of you know of um some x's and and and some y's and some targets okay um so now and so how did we actually get these Target values well we

it said we first computed the corresponding value of the function sin 2 pi of X that was our ground truth and then we added some small amount of noise and this is this is kind of capturing the common situation um where you know

our data for you know for regression or or kind of learning to be possible we you have to kind of assume there is some underlying structure so we're going to

assume the data has this underlying regularity um but we don't observe the regularity directly we observe it after it's corrupted by some noise now this noise sometimes this noise is actually

due to a physical process or maybe there's quantum mechanical noise um that you can't you can't actually avoid but usually this noise is coming from and it's great that he highlights this

variability that are themselves unobserved so for example let's assume that you're trying to fit like uh the heights of people as a function of their gender and their

age now obviously there's going to be some Trend there's going to be some fun you know you're going to be able to fit some function because certainly height does vary with gender and age um but there's going to be other factors you're

not taking into account so for example maybe people's diet or you know their genetic background things like this that are going to determine their height um their environmental experience that that's that can be an important factor

as well you know how their environment growing up um but your model isn't going to capture that and that's just going to show up as noise so it's going to be some kind of some variability around the

predictions you can make using the factors um that you've included so in this case height and age so the the main point is that you know when we say noise it doesn't mean that there's actually like some you know underlying physical

noise processes added it's just taking into account the factors that um we haven't included the important factors that we haven't included we can't include we just don't know about um in

addition to those unavoidable let's say you know quantum mechanical noise or whatever it is so I think this is an important point you know like the where this noise comes from it's not just you Randomness in the environment it's just

actually often reflects our our lack of knowledge about the other factors that influence the um the variable we're trying to predict so that now we come come to an

important point where when we've got a training set the point isn't to fit the data on the training set um um the point is always to make predictions for some

new value so you know we use a training set to kind of get a good function that fits that data but then the point is always to like pick to predict a new value um you give given a new value

predict the target for for that value um and you know for us in in our particular case it's going to be you know implicitly trying to discover the underlying function sin 2 pi ofx and this is a tough problem right because

this is this function is in some sense you know infinite dimensional and we're trying to um we have kind a finite number of samples of it so we're trying to kind of generalize from a finite

number of samples to this um to a you know potentially much larger object and the question is you know how to do this and you know you know and that's you know that process is going to kind

of have some uncertainty associated with it um and there's going to be some um often some cost with getting the answer is wrong and so we have some tools that we're going to be discussing um later in the chapter and throughout the book that

allow us to um come up with estimates of for example you know the the functions that were fitting um that leverage um that leverage our our knowledge of the cost

of errors and also allow us to kind of Reason about our uncertainty in kind of a principled way so that's probability Theory and decision Theory which we're going to talk again touch on later in this chapter and then we'll um we'll

discuss more uh later on in the book in depth all right so now let's get on to actually trying to fit this data so again this is often you're going

to have um to make a choice so you have some data set you have to make a choice of um how you're going to fit that data one of the ways of doing so is with what's called with these parametric

models and one of one of those parametric models is this uh polinomial model so it says this polinomial model

models the targets as a polinomial function of the input value so we've got a constant term um a linear term a squared term all the way up to an order

M term so there is an MTH order polinomial so we say here um and we're going to try to fit the unknowns for us are going to be these these coefficients W so we're going to use a training data

to hopefully get us good values of w and good values are going to be judged um by not only how they fit the training data but somehow and we'll see how um how

well they allow us to generalize to unseen data so that's what this is about one important Point again that he that he mentions in passing but it's um again

a very um very useful concept and a very useful technique is that if you look like if you looked at this model this model is nonlinear in X so we have

powers of X but it's linear in the parameters we care about so you know if I give you the idea is that I can give you um an input value X and now you can create a new feature which is that value

squared and you can create a third feature which is that value cubed so so essentially you can um bring in the power of a nonlinear model by creating these nonlinear features which are you

know once you're given the input those features are determined and then you try to find a linear combination of those features so in a sense you're you're um you're able to use linear techniques

while leveraging the power of nonlinear nonlinear features and that's I think a really powerful idea um believe it's called like feature augmentation something like that and you know obviously you're not restricted to

using you know powers of your input you can do you know if if you somehow had a you know brainstorm you could put in s of X or you know sign you know whatever a trigonometric function then you would find that you could actually fit your

this particular data set really well usually you won't know that what features you have to use and so you know polinomial are you know sometimes not a bad guess but the main point is you can

bring in nonlinear features um and then find a linear waiting of those and since the only unknown for you will be those linear we weightings you effectively have a linear model which has which are

easier to learn easier to talk about etc etc so this is the notion of linear models and we're going to be talking about them later um in chapters three and four okay so now how are we going to

determine these weights right we need to figure out these weights um the question is how do we determine those and let me actually move this to the side so it's yeah there we go how are we going to determine those weights okay so one

reasonable way and in actually even later in this chapter we'll see how we can kind of justify this but it doesn't seem unreasonable to start is to determine the weights by how well a

given function OB um the the function we have this polinomial function um actually predicts the training data that we have so what we're doing here is we're looking at for every Target value

we're looking at its predicted value and now we're going to be squaring that difference and this Square difference is really useful because it doesn't care what the sign of the difference is it'll be zero only if only if the our function

matches the data point exactly otherwise it'll be non zero and the larger um the difference gets um the bigger this error term and then we're just summing those up for each one of our data

points and then finally we're doing this one over half um this one over half term that'll be that's just for convenience and basically it's because when we're trying to in our particular case when

we're trying to figure out the optimal W um what we'll do is we'll take a derivative of this with respect to W and this power of two will cancel this one over two so we won't have these

um scaler coefficients hanging around that we just normally don't care about so that that's basically why we do it that's what you mean by the later convenience okay so that's our that's our error function here is again the

picture of what that would look like so for every data point that you have you look at this deviation you square it you add up all the contributions and then you divide that by two and that is your um your error and you trying to minimize

that the point is to find um you're trying to find a w and every time you choose a different W this red this red Curve will change maybe it'll go from this to this to I don't know like to something like that um and you just

measure its differences uh at the data points and some of those differences so now the question is well how do we find um how do we find this

this this Del view um you know as as I kind of alluded to earlier what we can do is we can just take this is a this ends up being a linear function so this is a quadratic function of w so you can

see this W's are here they're going to factor into Y and then we're going to square them so we're going to get something that's quadratic in um in the W's so when we take the derivative we'll get uh something that's linear in the

W's and it'll have a unique solution and that's how we pick it so we essentially just minimize the the error function and in for our particular error function that minimization we can do in closed form gives us a unique solution very

convenient one of the nice properties and all often motivations of using quadratic errors um and so once we've got that W often we'll we'll denote this the you

know the the our estimate of the parameters of the star or you know the unique solution W star now for every we look at what

this object is so now for every value of the input X we can actually compute our prediction y ofx w star this is this is the this is a function that we will have

learned okay so and now you give me any ex and I can you know I can plot essentially continuous curve um for any you know given all the values of X all right um

now the next the next aspect is remember we said okay look we have a polinomial of order M the question is how do you pick M right if m is zero well we just have a constant if m is one we have a linear

you know quadratic cubic how do we pick that this is a very this is a very common problem um and it's also um not it doesn't have a perfect answer there are kind of methods that you can use to

kind of come up with with an answer we're going to talk about one here and the the method we'll talk about here is essentially one of the most common ways of dealing with this problem by essentially looking at how well

different orders generalize to unseen data but the interesting thing of this next section is showing what happens for different ORD for different orders of M

so we're going to try you know m is zero all the way to you know um m is 9 just for illustration purposes and see what that does and how we can deal with it so I think that the easiest thing to do at

this point is to just look at the picture uh so here on the left sorry it's cut off a little bit but um this is the linear fit so the data points are

the are these blue dots and m equal Z is just a is a constant term so it's you know it's just going to be taking the the average value

so for a squared loss um the constant fit will just take the we we'll find the line that that is the average of of the data points that that's kind of that's exactly what you see here now here is a

linear fit and so it you know it's it seems to have found some kind of a slope in the data so tries to capture that slope we increase the order further to m equals to 3 that seems to kind of capture the shape of the data and this

is kind of this is you know J with our understanding of the sign function which if you take the domain to be wide enough has kind of a cubic shape to it um and and uh there is that now

interestingly you can see for all of these cases the the data the are estimates the the red curves don't go through the data points they're very close and they kind of minimize this but

they don't go through it if you make the order large enough in this case equal to the number of data points um one minus the data points minus one you can

actually fit all of the data points uh exactly so you can see it at m equals 9 you're going through every single data point exactly and so you you've actually reduced your error on the training set

to zero but you can tell that this is in some ways it's kind of it's it's it's too Wiggly right it's kind of it seems to have fit um kind of missed the trend to some extent and just really focus on

fitting everything exactly and thereby become um very volatile and very um you know very bouncy in between the points so it's kind of unlikely this will generalize to you know a new data point

you know for example if you put a new data point here it's going to get a huge error there because um it's you know it spent all of its flexibility trying to go through every single point exactly which is meant that it has to squish

itself a little bit and now it's going to have a huge error for um the points there so it's kind of missed uh maybe it's kind of missed miss the forest for the trees to some extent and this is an example of

overfitting so that's what we're going to talk about here so indeed so the m equals 3 polinomial gives the best fit um we obtain an excellent fit to the training data so this is always this is

always the case so with training data the more complex your model is the better you'll fit your training data you just have more parameters those parameters you can tweak to fit you know all little Wiggles in your data but the

problem is is that you will start experiencing overfitting and the source of over fitting is that you're essentially Now using those additional wiggle you know parameters that your model has you start fitting them to the to the noise to the noise variability to

the noise rather than the underlying structure that you're actually interested in so this is called overfitting very important topic you've probably heard about it and one that um you just need to learn to deal with and

we'll talk about how to deal with that in kind of a schematic form here and we'll talk about it in more detail um later in the section and later on um in the

book okay once again it's kind of worth emphasizing so what what do we care about we don't care about the training data right like we already I already know what the target values are on the training points what I care about is

generalizing to new data so that's always the point and so how can we um you know how can we try to measure our performance on our generalization

performance so one way to do that is by creating by creating or having left aside some part of our data now bringing that piece of data in and essentially

using as a separate test set so here because we know how the data was produced we can just generate another set of data points so here we go we're going to consider a separate test set of 100 data

points using exactly the same procedure and then what we can do is we can then try um different values of M train them on our our training set and then evaluate them on the test set so that's

kind of like U measuring somehow our generalization performance how well do we do for a given order um when we train on you know whatever our data set is well we can measure that by just having

a set that we haven't used in training and evaluating performance on that and so let's have a look what that does and here is here is that we'll tell what we'll talk about what erms is in a

second but just think of this as kind of our our measure of error and with look there's two things to notice so the the blue here is the training results of training and M is the order of the model

and you can see that as for low you know for low orders it you get pretty poor pretty poor um performance but as the order increases the error drops until at

actually nine the error is basically completely um is reduced to zero so that's the point where we our our curve goes exactly through every single point but now what we care about is

generalization performance so first you can see that it's always higher than um always higher than the the training error just because you haven't you haven't seen that data but more more

importantly it actually will start to increase at some point so at you know at the value of m equals 9 you can see this this test there has actually gone way up so in some sense you get a signal that

okay you know I don't know you know I can't quite tell whether what how these guys relate to each other but it really seems like nine is too high of an order and if I see something like this I'm like well if if their performance is all

basically the same why don't we just stick with m equals 3 because that seems like you know um GES the same performance the simpler model simpler models tend to be better so so yeah but

the main Point here is that using this test set we're able to gauge a little bit what the right order of the model is um in this setting for our data so that's kind of a

That's essential that's the the kind The Meta approach of having kind of data that you've left out and evaluating the performance of your model on that data so training on the training data testing

on this left out um training set okay um yeah so that's just going to describ what we just went through yeah he's going to Define he going to Define errors in this case but this

erms um why are we doing erms um so there's a few things going on here first there's a square root term going on that allows us to measure the errors on the same scale as the data points themselves

so you can imagine like let's say the data one way to see this is you know let's say your data was had units of meters well your error term is going to be because of the squared it's going to be in units of meters squared but you

want to it's it's useful to be able to actually compare those errors in the same units as your data points to kind of get a sense of how do they comp compare in terms of the size of my data how how big are they and so that's what the square root does um dividing by you

know multiplying by two and dividing by n that's so you can kind of um You can compare data sets of different sizes so you know if we have if one data set had twice as many data points well those

errors um would tend to be twice as large just because you have you're you're adding up um the results over twice as many um input examples so this allows you to just allows you

essentially normalize this um for the number of data data points you have so that you can compare different data sets on the same footing so that's what that's about if you actually look at the form of this you'll see if we kind of go

back to what the error function looks like you'll see so we multiply by two so that gets rid of this um and we divide by n so it'll be a divide by n and there'll be a square root and that's

basically like it's almost like a kind of a standard deviation so you you have your data points you try to predict um you try to predict uh give kind of a mean function for them and you measure the standard deviation of the of the

error um around around that mean so just some intuition for what this uh what this erms is and we and we just saw okay that essentially but either way you

know it doesn't change the point that um at if m is very large the training error will go to zero you can measure this in

RMS or in the e squ um but the the test error will go up so this kind of gives you a hint as to the order of the model um that you should pick for your data

um now the question is you know why is it that this you know this you know why why does high order model um do worse and we kind of touched on it a little bit which is you know it's kind of

fitting the noise but it is kind of I like his little discussion here um about that um because you can kind of in you can if you if you view it from you know a certain perspective it can kind of be

a little bit odd you know I'm telling you well you know the order three model is seems like the right one well the order nine polom contains an order three Pol polinomial right so for example you could just set

some of the higher order coefficients to zero and now you've got your order 9 polinomial is actually effectively in order three polinomial um another argument you might have against why the order three uh does

so well is well the sin 2 pi of X um function if you do the tailor series on it it will contain terms of all orders actually it's all odd orders because it's an odd function but either way

it'll have polinomial going all the way up to you know basically as as as high in odd order as you want so why is it that this order nine um when we have an

order n polinomial which should you know which will be able to include order seven three five three and one Pol within it why is it why is it that it you know it does so

poorly and so we can kind of so those are kind of valid questions and we can kind of look at to get an insight into why that's happening we can actually look at the coefficients that we're learning so here is the coefficients we

learn when m is equal to zero it's just you know a small small number m equals 1 it seems reasonable at m equal 6 we start seeing that we're getting you know some quite High coefficients m equals 9 we've got these enormous coefficients um

and essentially what's happening is you're kind of getting very large coefficients kind of battling very large positive coefficients battling very large negative ones to come and kind of exactly balance out and and meet the

data points that you have so essentially this is kind of a signature of of overfitting that you're having these coefficients being fine-tuned as you call I like that term line tune to the data but having large positive and

negative values that they they'll just exactly squeeze in um and and uh fit the the data PLS that you have and but again you know we're not interested in fitting the the training

data we're interested in generalizing to new data um and when that's the case this kind of fine-tuning is something we want to avoid so the question is you know how we do that so that's kind of exactly what it's

highlighting here um another interesting point that he discusses is that these issues this is these issues of overfitting will tend to

become um less will be will tend to diminish when we have more data so if we kind of stick with you know a maximum order of nine well here is um here's

kind of what it looks like with 15 data points so you can see it still has these kind of weird W Wiggles but because we essentially constraining it with more points um it it doesn't have the opportunity to kind of develop these

enormous crazy Wiggles in between the data points because essentially these data points are are constraining it saying well you know you can't swing enormously out because well there is a point here that's kind of constraining

you and by the time you have a 100 data points this order 9 model seems to actually capture the data really well in some sense you've kind of constrained it um so much that it just doesn't have the

opportunity to to wiggle you know wiggle although you know if you then made an order 100 model you would still find that maybe you could you could get these crazy Wiggles but for an order n model um this this over problem is much less

so it seems like there's kind of a balance between and doesn't just seem like it is the case there's a balance between the size of your data and the model complexity so the larger the data you have um the more complex the model

you can use without running into overfitting yeahi gives a her istic here um which yeah I mean you you just have to see if it holds for your particular

data set but um the number of data points should be no less than some multiple five or 10 of the number of adaptive parameters so if You' got you know let say you've got 10 adaptive parameters which we do for the m equals

9 model well you need 100 data points and indeed it seems like we're not getting much overfitting in that case um but again you'd have to measure the overfitting the way we've described by having a test set and you know

holding it out but again it looks like in this particular setting that you you just constrained the model so much that I don't imagine this result is going to be very different if you get another 100 data points so just you can kind of

eyeball it sometimes now um the question is kind of you know what aspect of the fitting procedure produces is kind of overfitting and uh as we'll discuss in subsequent subsequent sections what

we've done here is an example of Maximum likelihood estimations that energy model corresponds to the negative log likelihood in and we'll discuss what that means later um and maximum likelihood approaches will just tend to

overfit and there are other uh there's the other approach which going to be going to be the focus of the book kind of basan approaches which um avoid avoid this problem by essentially having a

prior by saying that um you know the weights for example can't be so large and if you do that properly then you can you can often avoid some of these overfitting issues now they they bring with them their own issues of you know

how do you pick prior Etc but at least in principle they provide a way of um in kind of a principled way of avoiding overfitting if you can kind of state your model correctly and integrate out

over the parameters that you don't care about um you can avoid overfitting um with uh with beian approaches and we'll see more of this um as we go

okay um so we saw that okay so we saw that the problem was you know our our our model was getting really complicated um and getting these kind of very large values of the

coefficients so you know we what if what if we try to kind of reduce the kind of penalize the size of those coefficients so instead of kind of picking let's say the order of the model which can be kind of you know maybe it's going to be a

little bit annoying having to go you know pick exactly what order it is will allow the model to be let's say up order nine but we'll actually put a a penalty on having large coefficients and this is an example of

what's called regularization we're trying to kind of regularize um you penalize the size of the coefficients to control the complexity of our model um and here's you know here is one

of the simplest the simplest kind of way of doing that is this um what's called you the L2 Norm where they you know the the sum of squares and which is which is this particular term here and so the

idea there is that you're going to be you're going to try to fit the data so this is your goodness of fit term but then you're going to also try to minimize the size of those coefficients and the hope is and this this parameter Lambda is now controlling the complexity

of your model so if you set your Lambda to zero well you're back in the case that you were before and you're going to have your order nine model that's going to you know have these wild swings in it but if you set the Lambda very very high

well the coefficients are going to go to zero because it's essentially not worth it to fit anything because you'll you'll gain um you you you'll be penalized too much but in between there'll be kind of

a sweet spot where um the coefficients will be kind of just large enough uh that they can fit the data um but also you know hopefully not overfit so not swing around too much so this kind of

Lambda parameter now gives you kind of a continuous control over the um the model complexity and yeah here's kind of a minor technical point that sometimes the constant term is is is is omitted from

regularization or it's uh regularized independently because it's just it just deps the offset of the data and and and often that um like the size of that offset uh is not something you NE want to penal penalize directly it just has

to do with the nature of your data set um so you might not want to include it into the into the regularization a couple of just terminology points here so the effect of you know as as we discussed these these

um um regularization will it's going to reduce the size of your parameters and actually if you work out what it does in many of these mods you'll find that um it is indeed a shrinkage of the Valu you

will you will take like your let's say your maximum likelihood um estimate and it will actually apply it will scale it down by some amount so um that's why they're often called shrinkage meth methods and this

particular form of regularization is often called Ridge regression um so you'll you'll you'll hear these all the time they they all just mean this exact same thing so you're just regularizing your model by W Square term it's very

simple um a very simple procedure neural networks this will also be called weight Decay again um because you're you're you're shrinking those weights these will be the weights

of the strength of connections the weights between different neurons and when you regularize when you uh shrink those weights it's kind of like um you

know decaying those values from their Maxim estimates also there are um when you do this dynamically there will be you know when you look at the the Dynamics of synapses

of those strengths when you have regularization like this that can actually manifest um in those Dynamics as a Decay term on the weights so it'll be you know the weights will will be um

driven by their inputs and the targets are trying to match um but there'll be kind of a background decay in their values that's trying to avoid having very large weights and kind of essentially making regularization happen

so that's anyway just some terminology these are all it's all it's all just this so not to get confused by it so now let's look at um what regularization actually does um so here is here is kind

what we talked about so we're we're going to talk about in terms of the lawn of the natur natural logarithm of Lambda now when there's no regularization Lambda is zero lawn Lambda is minus

infinity and we're exactly in the situation we were before so these are the same weights that we saw when we just fit an M9 model but you know as we increase the regularization so now it's minus 18 now

the weights start you know we have an order N9 model but these weights are not the weights aren't enormous so it's kind of a different way of actually controlling the model complexity so before we controlled it by just having

okay Max topping out the model at at three coefficients whereas now we're allowing all these coefficients but we're saying look the coefficients themselves can't get can't get too large

so it's a different approach to um to controlling the complexity of the model here we've really cranked up the regularization these weights are essentially zero so I think there's a picture of that

um here we go yeah here's a picture of that right here so this is an intermediate value of regularization so again here's the true function here is our fit and again this is remember this

is an order 9 model but with some regularization and now it's actually you know tended to fit it pretty well and if we crank up the regularization even further well now there's no wiggle room in the weights so the weights are

essentially around zero and you just you fit the data like that so you kind of now are now approaching that constant model um although again technically it's not actually constant because you have nonzero coefficients for for all the

orders so you're getting the same effect um but in a different way rather than limiting the order of the of of your of your model limiting the size the

coefficients so that's pretty cool um and we can then you know just like just like we did before with the model order we can just look at the training and test sets and now you get these smooth

curves which is really nice um telling you okay what what value of Lambda to use so and it looks like you know we would pick the one where you know the the test error is lowest and looks like it's about here so probably even um um

less regularization than than what we have in this particular figure but that's that's essentially how you you would often do it in practice so you would you would try different values of Lambda compute the test eror and then

often then if you actually want to report because you've used this to determine your value of your regularization parameter you want to report the performance using another data set and another part of the data that You' that you've held out um kind

of a it's called a validation set or vice versa I always get these names mixed up yeah so I'm just kind of mentioned um out here so yeah so this kind of this is this is I think in this

part particular section which I think we're at the end of the section now um I think we've touched on some of the um the main aspects of of uh you know regression problems that come up which also will come up in

classification as well so you know how you pick how to pick a model um how to measure the deviation of the model from um from the data uh that having training sets test sets regularizing your model

the way of you know controlling model complexity and these are all kind of very important issues that we're going to discuss kind of and using more sophisticated more principled approaches

um in future sections of the book uh so I hope that was helpful and um I will see you next time for the next section thank you