Section 1.0 of Pattern Recognition and Machine Learning - Introduction

great so let's start with uh chapter one where he's going to lay out some of the core ideas of what pattern recognition machine learning this book we're reading about is all about uh some of the core

terminology topics um that we're going to be covering in the rest of the book so at the beginning he just talks about um you know searching for batter

patterns and data why that's important and there's a cool example of this that happened in the 16th century where this very wealthy uh Danish guy who was a big astronomy Enthusiast he uh he had an

observatory and he took very or had people take very um precise measurements of the orbits of the planets and you know astronomical bodies um and those very precise measurements later then

helped Kepler uh come up with Kepler's laws of uh of motion of planetary motion upon which then um Newton built his laws of universal gravitation and this kind of led to the development of mechanics

calculus um much of the math that we're actually going to be using in this book so Gathering data but also then finding regularities in data is uh really important topic um that's led to lots of

developments and that's kind of what we're what we're going to be thinking about in in this book now the key aspect here is that um you know we're going to be looking for looking at the automatic

Discovery regularities so we're going to be designing algorithms um that will find patterns and data for us so Kepler for example he had to kind of look at this data and think about you know the

regularities that could be there but we're going to be designing algorithms that try to find these things for us now interestingly I really like that uh you know Bishop emphasizes this so it's you know we're going to be interested not

just in discovering these regularities but also taking actions you know once we have these regularities so once once once you discover these patterns and data what to do with them should you you

know should you classify this as you know as a planet should you say that uh you know the the the planet is going to be in a particular place um and there's going to be a really interesting

interplay between you know probabilistic Concepts that uh that help you to characterize uncertainty and and and and your knowledge about a situation with um Concepts from decision Theory so how

much it costs to get a decision decision wrong and that interplay can tell you what information is worth Gathering and how to best apply it so that's kind of a lot of the stuff that's going to be coming up um in the

book okay now a very important example and if you know if you know anything about machine learning or you're interested in it at all there's idea of recognizing handwritten digits and so I

think you know back I don't know I guess in the 70s 80s maybe even before um there was the idea of applying computers to the task of recognizing handwritten digits and why do you want to do that

well it's because people were you know sending mail and you want it would be really useful to have a system that could just read you know the backs of envelopes and tell where mail was

going so this problem handwritten you know um handwritten digit recognition was became really important and is very practically important and that's kind of the the basic Benchmark that people use

if you've heard of this data set called mist of you know of handwritten digits that's what it was all developed about so having lots of examples of digits that then we could train um you know

algorithms to help us recognize so yeah so let's get back to the problem so we've got you know these handwritten digits and they're they're 28 by 28 pixel images um and the goal is we want

to build this machine that will take one of those images and give a answer 0 to9 and as he points out this is a non-trivial problem because you know people aren't robots so everyone writes

their zeros and ones and twos in a different way so it's important to be able to um deal with that um variability so you know how do we deal with that

variability so one possibility so and before we get onto that here is kind of an example of you know what some of those digits look like and okay you know if everyone wrote their zeros in exactly this way it would be no problem you

could just match for this exact pattern but of course people write their zeros all differently their ones differently so that's that's why it's important to be able to handle that variability now back in the early days

of AI you know one way people might have um approached this problem is um through setting up us just systems of rules so the kind of rule-based AI was um you know was very big in the early days and

it's still around and I think there probably are applications where it's really you know the perfect thing to do but for applications like this we're kind of we're dealing with let's say more like kind of organic data um it it

seems intuitive that a rule-based system wouldn't necessarily work because how would you be able to capture all possible variations and zeros like even in this particular example right we have kind of a zero but like you know maybe

like is it a zero maybe it's actually kind of a six and you know there other things you know what is this is that is that a five like it looks like a five but it could be something else or look at even these two examples that's a three but maybe it's like an eight and

the person didn't fill it in right and trying to capture that with you know large systems of rules doesn't seem appropriate for this particular kind of problem so you know as as you know Bishop says um you know it's going to

lead to a proliferation of rules and exceptions to the rules and that's just going to become you know um really really difficult to build and difficult to um you know debug in a kind of in a

reasonable way so for problems like this it's much better and you can get much better results and uh in fact this is now become a benchmark this amness data set recognizing these digits is now a

benchmark of modern AI systems to adopt the machine learning approach so we just instead of trying to build in the rules like going in and okay telling it well if you have a line

that's kind of going in this direction it's probably a one um you just prepare first a training set so a training set of examples of you know lots of zeros

and lots of ones and lots of you know all the digits along and with each example training set you also have a kind of Target vector or you know just some label that tells you

which which kind of Digit it is and the idea is to essentially be able to build an algorithm that will take these TR this training set and these targets and figure out the regularities that are there without you having to tell it so

Building Systems that can figure out these regularities so that we can its output will then tend to be correct and we'll we'll give it an important we'll give it a new digit and it will tend to

classify correctly the way that you know maybe a human um would do that so you know the result of running the machine learning algorithm can be expressed as a function y ofx which

takes a new image and outputs a vector telling you you know what what the digit supposed to be and so how do we you know figure this out so this this happens during a training phase which is also

known as learning um where U the system somehow and the whole big chunk of the book is about exactly how this happens but it will go from those examples that you've shown it um and and the labels

and try to figure out how you know how to match you know what the regularities are that match um those training examples to the labels you've given um and you know once you've trained this model how do you evaluate

it well these models can be very sophisticated so in some senses sometimes you can even memorize your entire training set so it doesn't necessar it's not very useful to train

to test the models on the same data so what we often do is we hold out some of the data from training um into what's called a test set and then we evaluate the performance on this heldout data so

so that the model doesn't have a chance of just just memorizing its uh its examples and this you know this this is a really important property that we want which is kind of generalization and why

do we you know generalizing from the training set to held out data to to to data in the real world because we don't really care actually we we already have the labels for the training set so we don't really care about classifying that

data the point is you know when new people write their you know their their digits um how um how are those going to be um are those going to be classified

correctly so that's kind of what the whole uh what the whole game is about so generalization is kind of a central goal of pattern recognition which is uh which is the case Okay so

the next step so once you're trying to build one of these systems um there's often a step of pre-processing the data so you know those digits that that we um that we had they don't arrive they just

arrive as you know 28 x 28 pixels um but often it's not that's not the most useful uh format of dealing with um with that data so you might for example um

you know take these images and say okay well it's going to be really useful to take these pixels and maybe figure out where all the if how many edges like that they contain or you know do they have any Loops or even something more

basic just taking these images and kind of centering them so that this seven uh appears the center of the image rather than on the side so reducing some of the variability um and that's really important because and and often you know when we're looking at these problems a

really a difficulty that we have is our brain does so much pattern recognition and pre-processing already that we're like well you know obviously this is kind of a really simple problem to solve to the extent that actually you know in

1956 um you know one of these very first AI conferences they decided that they were going to give the problem of vision to a grad student to solve over the summer so you know it's not just us you know us novices that we're looking at

this thinking it's an easy problem even like you know the great great scientists of the time were thinking oh yeah like Visions are you know simple like you just you know find these patterns it's easy enough a grad student will solve in

the summer and here we are you know more than you know 60 70 years later and okay you know now we now have systems that are solving Vision um at reasonable ways and we still don't really understand how

they work we just have systems that can actually do the problem so it took a long time and it um and a lot of the techniques in this book kind of give you a sense of what's what's required but

yeah so a very the very first step of a you know machine learning algorithms often is kind of pre-processing Step of taking your data and putting it in a format that's more useful or in which uh

the the pattern recognition problem will be um easier to solve so that's kind of what we're we're talking about here which is exactly so transform them into into oops transform them into um some

new space of variables where it's hoped that the problem will be uh easier to solve so that's the whole point and this this is kind of often called you know feature extraction so as we said you know with the digit after your Center

them maybe you want to start looking for Loops that'll that might be use might be useful features for determining whether something is an eight or you know a six or or

not um another aspect of this pre-processing is indeed to speed up the computation so you might not you know you might not want to work with the entire 784 set of pixels it might be um

simpler to reduce to reduce that somehow um so essentially the game the game is going to be to find these us useful features that are fast to compute um but are still preserving the information you need for discrimination because of

course we can just you know zero out all the features and uh that's fast to compute but obviously all the information is gone and so and and part of this process you know uh part of this

process will essentially can result in having fewer features to work with than the raw pixel that you um that you started with so that can be kind of a case of dimensionality reduction going

from your you know large pixel space all the way down to um a a limited set of useful small but useful features with which you can you can work um so that's

that's uh often the first step okay so now we're going to talk about kind of some of the different types of kind of learning problems that that we're going

to come across um the type we've been talking about so far is supervised learning and it's called supervised learning because um for each one of your training examples you actually have a

label and the whole game is uh to figure out the regularities to take you from the training examples to the labels so it's supervised in that you had someone a supervisor has given you a label for

every one of your the elements in your data set now the type of problem we've talked about where you have kind of a discrete number of categories you choose from that's typically called classification so you're classifying you know uh those

digits those picture you know those handwritten digits into a number from you know Zer Z to nine might be a pictures you might pictures of birds you want to tell which kind of bird it is um so those are all classification problems

we're trying to classify um um an input into one of the number of categories another really important class is regression where the output is actually kind of more continuous so for example

let's say you want to know what is a particular price of a stock tomorrow so that's like a that's not a number of you know one to nine it's it's basically has a continuous value so that might that's an example what's called a regression

problem but again um what's you know what makes most supervised learning is that in that stock example you will have let's say historical data that has said that well when all these important

economic parameters are moving a certain way the stock price was so and so the next day and you have like a history of that um of the of that data and then you want to now know tomorrow what is going to be that stock price so the

supervision is in the fact that you have historical data with labels of some kind the other very important um class of you know machine learning patent

recognition problems is unsupervised learning I don't know why I keep switching to these uh to the Eraser but unsupervised learning um and that's where the data doesn't come with labels

so you have to kind of learn you that's more about finding regularities in the structure of the data itself so for example you might be looking for clusters in the data you know the data

is often not randomly distributed in the feature space actually um it you know it is bulking together in certain parts and that's going to be really useful information maybe that's going you could

use that later on to performer classification uh you might actually want to just model the distribution of data in that space that's cond density estimation how you know I have a you know you you pretend that your data is

actually samples from some distribution you're interested in and now you're trying to estimate that distribution and you know as dimensionality gets higher that can become very difficult so it's kind of an interesting problem how to do

that um in you know thousands of dimensions and the final kind of uh problem that is we deal with machine learning which is really important is a reinforcement learning and I don't know if you've heard about you know things

like Alpha go uh that beat uh the best human players just a few years ago um you know Alpha zero you know or you know deep mind's um other amazing rein

reinforcement learning system which uh was learn to play Atari games at human levels these are all examples of reinforcement learning and what and what makes them different from um supervised learning is that in reinforcement

learning although you do get a reward uh you don't get a reward for every everything you do so the the a there will be an agent there'll be something some kind of um algorithm that's acting in an environment and every once in a

while it will get a reward so most of the time maybe nothing will happen but then it'll occasionally get a reward so for example if you're playing chess you might not get a reward until you either win or lose the game and the question is

how do you build a system that with that just that sparse information whether you won lost the game can figure out you know how to play chess and um amazingly this can be done and it was originally

done for you know back in the 90s um and now it's been recently been done with these amazing um uh these amazing reinforcement learning algorithms from uh Deep Mind for um other games like go

these Atari Games Etc so this is a really important um really important um aspect of learning um but it's not one that he's going to talk about um much in the book so we're going to be focusing

much more on supervised learning unsupervised learning so yeah so those were kind of the big categories um and yeah so then he kind of finishes off the section by just saying that the whole the whole point of

this this whole introductory chapter is to introduce kind of some of these basic concepts that are going to come up over and over again um in the book and then also to have introductions to these three tools the probability Theory

information Theory and decision Theory um and the kind of the interplay of these is what's going to help us solve these um you know machine learning pattern recognition problems um in in a

principled way um and kind of tackle these lots of these really interesting um really interesting tasks in kind of a principled way um that will hopefully uh you know give good results great so

that's uh that's kind of what this introductory section was all about