Afrikaans
Akan
Albanian
Amharic
Armenian
Azerbaijani
Basque
Belarusian
Bemba
Bengali
Bihari
Bosnian
Breton
Bulgarian
Cambodian
Catalan
Cebuano
Cherokee
Chichewa
Chinese (Simplified)
Chinese (Traditional)
Corsican
Croatian
Czech
Danish
Dutch
English
Esperanto
Estonian
Ewe
Faroese
Filipino
Finnish
French
Frisian
Ga
Galician
Georgian
German
Greek
Guarani
Gujarati
Haitian Creole
Hausa
Hawaiian
Hebrew
Hindi
Hmong
Hungarian
Icelandic
Igbo
Indonesian
Interlingua
Irish
Italian
Japanese
Javanese
Kannada
Kazakh
Kinyarwanda
Kirundi
Kongo
Korean
Krio (Sierra Leone)
Kurdish
Kurdish (Soranรฎ)
Kyrgyz
Laothian
Latin
Latvian
Lingala
Lithuanian
Lozi
Luganda
Luo
Luxembourgish
Macedonian
Malagasy
Malay
Malayalam
Maltese
Maori
Marathi
Mauritian Creole
Moldavian
Mongolian
Myanmar (Burmese)
Montenegrin
Nepali
Nigerian Pidgin
Northern Sotho
Norwegian
Norwegian (Nynorsk)
Occitan
Oriya
Oromo
Pashto
Persian
Polish
Portuguese (Brazil)
Portuguese (Portugal)
Punjabi
Quechua
Romanian
Romansh
Runyakitara
Russian
Samoan
Scots Gaelic
Serbian
Serbo-Croatian
Sesotho
Setswana
Seychellois Creole
Shona
Sindhi
Sinhalese
Slovak
Slovenian
Somali
Spanish
Spanish (Latin American)
Sundanese
Swahili
Swedish
Tajik
Tamil
Tatar
Telugu
Thai
Tigrinya
Tonga
Tshiluba
Tumbuka
Turkish
Turkmen
Twi
Uighur
Ukrainian
Urdu
Uzbek
Vietnamese
Welsh
Wolof
Xhosa
Yiddish
Yoruba
Zulu
Kylie Ying has worked at many interesting places such as MIT,
She's a physicist, engineer, and basically a genius. And now she's
about machine learning in a way that is accessible to absolute
What's up you guys? So welcome to Machine Learning for Everyone.
is interested in machine learning and you think you are considered
is for you. In this video, we'll talk about supervised and
we'll go through maybe a little bit of the logic or math behind
we can program it on Google CoLab. If there are certain things
you're somebody with more experience than me, please feel free to
and we can all as a community learn from this together. So with
Without wasting any time, let's just dive straight into the code
concepts as we go. So this here is the UCI machine learning
they just have a ton of data sets that we can access. And I found
the magic gamma telescope data set. So in this data set, if you
to summarize what I what I think is going on, is there's this
these high energy particles hitting the telescope. Now there's a
actually records certain patterns of you know, how this light hits
properties of those patterns in order to predict what type of
whether it was a gamma particle, or some other head, like hadron.
the attributes of those patterns that we collect in the camera. So
know, some length, width, size, asymmetry, etc. Now we're going to
help us discriminate the patterns and whether or not they came
So in order to do this, we're going to come up here, go to the
to click this magic zero for data, and we're going to download
notebook open. So you go to colab dot research dot google.com, you
I'm just going to call this the magic data set. So actually, I'm
magic example. Okay. So with that, I'm going to first start with
you know, I always import NumPy, I always import pandas. And I
And then we'll import other things as we go. So yeah
we run that in order to run the cell, you can either click this
on my computer, it's just shift enter and that that will run the
to order I'm just going to, you know, let you guys know, okay,
So I've copied and pasted this actually, but this is just where I
And in order to import that downloaded file that we we got from
over here to this folder thing. And I am literally just going to
Okay. So in order to take a look at, you know, what does this file
do we have the labels? Do we not? I mean, we could open it on our
pandas read CSV. And we can pass in the name of this file.
And let's see what it returns. So it doesn't seem like we have the
I'm just going to make the columns, the column labels, all of
So I'm just going to take these values and make that the column
All right, how do I do that? So basically, I will come back here,
calls. And I will type in all of those things. With f size, f
We have f symmetry, f m three long, f m three trans, f alpha.
Okay, great. Now in order to label those as these columns down
So basically, this command here just reads some CSV file that you
separated values, and turns that into a pandas data frame object.
then it basically assigns these labels to the columns of this data
this data frame equal to DF. And then if we call the head is just
give me the first five things. Now you'll see that we have labels
All right, great. So one thing that you might notice is that over
we have G and H. So if I actually go down here, and I do data
you'll see that I have either G's or H's, and these stand for
And our computer is not so good at understanding letters, right?
understanding numbers. So what we're going to do is we're going to
one for H. So here, I'm going to set this equal to this, whether
I'm just going to say as type int. So what this should do is
if it equals G, then this is true. So I guess that would be one.
be false. So that would be zero, but I'm just converting G and H
really matter. Like, if G is one and H is zero or vice versa. Let
now and talk about this data set. So here I have some data frame,
values for each entry. Now this is a you know, each of these is
it's one item in our data set, it's one data point, all of these
thing when I mentioned, oh, this is one example, or this is one
these samples, they have, you know, one quality for each or one
up here, and then it has the class. Now what we're going to do in
predict for future, you know, samples, whether the class is G for
that is something known as classification. Now, all of these up
and features are just things that we're going to pass into our
the label, which in this case is the class column. So for you
10 different features. So I have 10 different values that I can
And I can spit out, you know, the class the label, and I know the
this is actually supervised learning. All right. So before I move
little crash course on what I just said. This is machine learning
question is, what is machine learning? Well, machine learning is a
that focuses on certain algorithms, which might help a computer
programmer being there telling the computer exactly what to do.
programming. So you might have heard of AI and ML and data
all of these. So AI is artificial intelligence. And that's an area
goal is to enable computers and machines to perform human like
Now machine learning is a subset of AI that tries to solve one
using certain data. And data science is a field that attempts to
from data. And that might mean we're using machine learning. So
and all of them might use machine learning. So there are a few
The first one is supervised learning. And in supervised learning,
So this means whatever input we get, we have a corresponding
models and to learn outputs of different new inputs that we might
I might have these pictures, okay, to a computer, all these
with a certain color. Now in supervised learning, all of these
them, this is the output that we might want the computer to be
over here, this picture is a cat, this picture is a dog, and this
Now there's also unsupervised learning. And in unsupervised
to learn about patterns in the data. So here are here are my input
images, they're just pixels. Well, okay, let's say I have a bunch
And what I can do is I can feed all these to my computer. And I
my computer is not going to be able to say, Oh, this is a cat, dog
you know, the output. But it might be able to cluster all these
Hey, all of these have something in common. All of these have
down here have something in common, that's finding some sort of
And finally, we have reinforcement learning. And reinforcement
there's an agent that is learning in some sort of interactive
penalties. So let's think of a dog, we can train our dog, but
any wrong or right output at any given moment, right? Well, let's
Essentially, what we're doing is we're giving rewards to our
Hey, this is probably something good that you want to keep doing.
But in this class today, we'll be focusing on supervised learning
and learning different models for each of those. Alright, so let's
first. So this is kind of what a machine learning model looks like
that are going into some model. And then the model is spitting out
So all these inputs, this is what we call the feature vector. Now
of features that we can have, we might have qualitative features.
categorical data, there's either a finite number of categories or
qualitative feature might be gender. And in this case, there's
the example, I know this might be a little bit outdated. Here we
two genders, there are two different categories. That's a piece of
example might be okay, we have, you know, a bunch of different
a nation or a location, that might also be an example of
these, there's no inherent order. It's not like, you know, we can
three, etc. Right? There's not really any inherent order built
data sets. That's why we call this nominal data. Now, for nominal
to feed it into our computer is using something called one hot
know, I have a data set, some of the items in our data, some of
some might be from India, then Canada, then France. Now, how do we
we have to do something called one hot encoding. And basically,
well, if it matches some category, make that a one. And if it
So for example, if your input were from the US, you would you
0100. Canada, okay, well, the item representing Canada is one and
France is one. And then you can see that the rest are zeros,
Now, there are also a different type of qualitative feature. So
there are different age groups, there's babies, toddlers,
adults, and so on, right. And on the right hand side, we might
bad, not so good, mediocre, good, and then like, great. Now, these
data, because they have some sort of inherent order, right? Like,
being a baby than being an elderly person, right? Or good is
bad. So these have some sort of inherent ordering system. And so
we can actually just mark them from, you know, one to five, or we
let's give it a number. And this makes sense. Because, like, for
just said, how good is closer to great, then good is close to not
to five, then four is close to one. So this actually kind of makes
computer as well. Alright, there are also quantitative pieces of
pieces of data are numerical valued pieces of data. So this could
you know, they might be integers, or it could be continuous, which
So for example, the length of something is a quantitative piece of
feature, the temperature of something is a quantitative feature.
Easter eggs I collected in my basket, this Easter egg hunt, that
feature. Okay, so these are continuous. And this over here is the
that go into our feature vector, those are our features that we're
our computers are really, really good at understanding math, right
they're not so good at understanding things that humans might be
Well, what are the types of predictions that our model can output?
there are some different tasks, there's one classification, and
just saying, okay, predict discrete classes. And that might mean,
this is a pizza, and this is ice cream. Okay, so there are three
pictures of hot dogs, pizza or ice cream, I can put under these
Hot dog, pizza, ice cream. This is something known as multi class
binary classification. And binary classification, you might have
only two categories that you're working with something that is
isn't binary classification. Okay, so yeah, other examples. So if
sentiment, that's binary classification. Maybe you're predicting
dogs. That's binary classification. Maybe, you know, you are
trying to figure out if an email spam or not spam. So that's also
Now for multi class classification, you might have, you know, cat,
rabbit, etc. We might have different types of fruits like orange,
maybe different plant species. But multi class classification just
and binary means we're predicting between two things. There's also
when we talk about supervised learning. And this just means we're
values. So instead of just trying to predict different categories,
with a number that you know, is on some sort of scale. So some
be the price of aetherium tomorrow, or it might be okay, what is
Or it might be what is the price of this house? Right? So these
discrete classes. We're trying to predict a number that's as close
using different features of our data set. So that's exactly what
supervised learning. Now let's talk about the model itself. How do
Or how can we tell whether or not it's even learning? So before we
let's talk about how can we actually like evaluate these models?
whether something is a good model or bad model? So let's take a
set has this is from a diabetes, a Pima Indian diabetes data set.
number of pregnancies, different glucose levels, blood pressure,
age, and then the outcome whether or not they have diabetes one
So here, all of these are quantitative features, right, because
So each row is a different sample in the data. So it's a different
and each row represents one person in this data set. Now this
different feature. So this one here is some measure of blood
over here, as we mentioned is the output label. So this one is
And as I mentioned, this is what we would call a feature vector,
features in one sample. And this is what's known as the target, or
vector. That's what we're trying to predict. And all of these
And over here, this is our labels or targets vector y. So I've
bar to kind of talk about some of the other concepts in machine
we have our x, our features matrix, and over here, this is our
will be fed into our model, right. And our model will make some
is we compare that prediction to the actual value of y that we
that's the whole point of supervised learning is we can compare
oh, what is the truth, actually, and then we can go back and we
iteration, we get closer to what the true value is. So that whole
okay, what's the difference? Where did we go wrong? That's what's
Alright, so take this whole, you know, chunk right here, do we
chocolate bar into the model to train our model? Not really,
how do we know that our model can do well on new data that we
create a model to predict whether or not someone has diabetes,
data, and I see that all my training data does well, I go to some
model. I think you can use this to predict if somebody has
be effective or not? Probably not, right? Because we haven't
generalize. Okay, it might do well after you know, our model has
over again. But what about new data? Can our model handle new
model to assess that? So we actually break up our whole data set
types of data sets, we call it the training data set, the
set. And you know, you might have 60% here 20% and 20% or 80 10
many statistics you have, I think either of those would be
the training data set into our model, we come up with, you know,
corresponding with each sample that we put into our model, we
between our prediction and the true values, this is something
what's the difference here, in some numerical quantity, of course.
and that's what we call training. Okay. So then, once you know,
we can put our validation set through this model. And the
check during or after training to ensure that the model can handle
single time after we train one iteration, we might stick the
the loss there. And then after our training is over, we can assess
hey, what's the loss there. But one key difference here is that we
this loss never gets fed back into the model, right, that feedback
Alright, so let's talk about loss really quickly. So here, I have
I have some sort of data that's being fed into the model, and then
here is pretty far from you know, this truth that we want. And so
model B, again, this is pretty far from what we want. So this loss
let's give it 1.5. Now this one here, it's pretty close, I mean,
to this one. So that might have a loss of 0.5. And then this one
but still better than these two. So that loss might be 0.9. Okay,
performs the best? Well, model C has a smallest loss, so it's
take model C. After you know, we've come up with these, all these
C is probably the best model. We take model C, and we run our test
test set is used as a final check to see how generalizable that
you know, finish training my diabetes data set, then I could run
data and I can say, oh, like, this is how we perform on data that
any point during the training process. Okay. And that loss, that's
of my test set, or this would be the final reported performance of
So let's talk about this thing called loss, because I think I kind
right? So loss is the difference between your prediction and the
So this would give a slightly higher loss than this. And this
because it's even more off. In computer science, we like formulas,
of describing things. So here are some examples of loss functions
up with numbers. This here is known as L one loss. And basically,
absolute value of whatever your you know, real value is, whatever
subtracts the predicted value, and takes the absolute value of
value is a function that looks something like this. So the further
right in either direction. So if your real value is off from your
then your loss for that point would be 10. And then this sum here
we're taking all the points in our data set. And we're trying to
everything is. Now, we also have something called L two loss. So
which means that if it's close, the penalty is very minimal. And
then the penalty is much, much higher. Okay. And this instead of
the the difference between the two. Now, there's also something
It looks something like this. And this is for binary
loss that we use. So this loss, you know, I'm not going to really
But you just need to know that loss decreases as the performance
other measures of accurate or performance as well. So for example,
So let's say that these are pictures that I'm feeding my model,
might be apple, orange, orange, apple, okay, but the actual is
three of them were correct. And one of them was incorrect. So the
three quarters or 75%. Alright, coming back to our colab notebook,
bit. Again, we've imported stuff up here. And we've already
this is this is all of our data. This is what we're going to use
again, if we now take a look at our data set, you'll see that our
So now this is all numerical, which is good, because our computer
Okay. And you know, it would probably be a good idea to maybe kind
have anything to do with the class. So here, I'm going to go
in the columns of this data frame. So this just gets me the list.
right? It's called so let's just use that might be less confusing
thing, which is the class. So I'm going to take all these 10
to plot them as a histogram. So and now I'm going to plot them as
take that data frame, and I say, okay, for everything where the
of our gammas, remember, now, for that portion of the data frame,
these, okay, what this part here is saying is, inside the data
the class is equal to one. So that's all all of these would fit
And now let's just look at the label column. So the first label
be this column. So this command here is getting me all the
for this specific label. And that's exactly what I'm going to put
just going to tell you know, matplotlib make the color blue, make
set alpha, why do I keep doing that, alpha equal to 0.7. So that's
And then I'm going to set density equal to true, so that when we
the hadrons here, we'll have a baseline for comparing them. Okay,
just basically normalizes these distributions. So you know, if you
and then 50 of another type, well, if you drew the histograms, it
one of them would be a lot bigger than the other, right. But by
distributing them over how many samples there are. Alright, and
on here and make that the label, the y label. So because it's
And the x label is just going to be the label.
What is going on. And I'm going to include a legend and PLT dot
the plot. So if I run that, just be up to the last item. So we
item. And now we can see that we're plotting all of these. So here
made this gamma. So this should be hadron. Okay, so the gammas in
here we can already see that, you know, maybe if the length is
to be gamma, right. And we can kind of you know, these all look
clearly, if there's more asymmetry, or if you know, this asymmetry
probably hadron. Okay, oh, this one's a good one. So f alpha seems
distributed. Whereas if this is smaller, it looks like there's
Okay, so this is kind of what the data that we're working with, we
Okay, so the next thing that we're going to do here is we are
our validation, and our test data sets. I'm going to set train
this. So NumPy dot split, I'm just splitting up the data frame.
where I'm sampling everything, this will basically shuffle my
exactly I'm splitting my data set, so the first split is going to
to say 0.6 times the length of this data frame. So and then cast
to be the first place where you know, I cut it off, and that'll be
then go to 0.8, this basically means everything between 60% and
set will go towards validation. And then, like everything from 80
my test data. So I can run that. And now, if we go up here, and we
these columns seem to have values in like the 100s, whereas this
all these numbers is way off. And sometimes that will affect our
is way off. And sometimes that will affect our results. So one
is scale these so that they are, you know, so that it's now
standard deviation of that specific column. I'm going to create a
And I'm going to pass in the data frame. And that's what I'll do
going to be, you know, I take the data frame. And let's assume
you know, that the label will always be the last thing in the data
data frame, dot columns all the way up to the last item, and get
well, it's the last column. So I can just do this, I can just
and then get those values. Now, in, so I'm actually going to
the standard scalar from sk learn. So if I come up here, I can go
And I'm going to import standard scalar, I have to run that cell,
And now I'm going to create a scalar and use that skip or so
And with the scalar, what I can do is actually just fit and
is equal to scalar dot fit, fit, transform x. So what that's doing
fit the standard scalar to x, and then transform all those values.
going to be our new x. Alright. And then I'm also going to just
one huge 2d NumPy array. And in order to do that, I'm going to
okay, take an array, and another array and horizontally stack them
the H stands for. So by horizontally stacked them together, just
okay, not on top of each other. So what am I stacking? Well, I
so that it can stack x and y. And now, okay, so NumPy is very
right? So in this specific case, our x is a two dimensional
thing, it's only a vector of values. So in order to now reshape it
NumPy dot reshape. And we can pass in the dimensions of its
one comma one, that just means okay, make this a 2d array, where
what what this dimension value would be, which ends up being the
same as literally doing this. But the negative one is easier
do the hard work. So if I stack that, I'm going to then return the
thing is that if we go into our training data set, okay, again,
And we get the length of the training data set. But where the
so remember that this is the gammas. And then if we print that,
we'll see that, you know, there's around 7000 of the gammas, but
So that might actually become an issue. And instead, what we want
our our training data set. So that means that we want to increase
so that these kind of match better. And surprise, surprise, there
that will help us do that. It's so I'm going to go to from in the
going to import this random oversampler, run that cell, and come
add in this parameter called oversample, and set that to false for
oversample, then what I'm going to do, and by oversample, so if I
then I'm going to create this ROS and set it equal to this random
I'm just going to say, okay, just fit and resample x and y. And
take more of the less class. So take take the less class and keep
the size of our data set of that smaller class so that they now
data set, and I pass in the training data set where oversample is
is train and then x train, y train. Oops, what's going on? These
what I'm doing now is I'm just saying, okay, what is the length of
14,800, whatever. And now let's take a look at how many of these
we can just sum that up. And then we'll also see that if we
many of them are the other type, it's the same value. So now these
rebalanced. Okay, well, okay. So here, I'm just going to make this
then the next one, I'm going to make this the test data set.
switch oversample here to false. Now, the reason why I'm switching
validation and my test sets are for the purpose of you know, if I
how does my sample perform on those? And I don't want to
I don't care about balancing those I'm, I want to know if I have a
unlabeled, can I trust my model, right? So that's why I'm not
what is going on? Oh, it's because we already have this train. So
that data frame again. And now let's run these. Okay. So now we
And we're going to move on to different models now. And I'm going
about each of these models. And then I'm going to show you how we
first model that we're going to learn about is KNN or K nearest
already drawn a plot on the y axis, I have the number of kids that
on the x axis, I have their income in terms of 1000s per year. So,
making 40,000 a year, that's where this would be. And if somebody
would be somebody has zero kids, it'd be somewhere along this
somewhere over here. Okay. And now I have these plus signs and
I'm going to represent here is the plus sign means that they own a
to represent no car. Okay. So your initial thought should be okay,
classification because all of our points all of our samples have
the plus label. And this here is another sample with the minus
width that I'll use. Alright, so we have this entire data set. And
own a car and maybe around half the people don't own a car. Okay,
point, let me use choose a different color, I'll use this nice
point over here? So let's say that somebody makes 40,000 a year
that would be? Well, just logically looking at this plot, you
they wouldn't have a car, right? Because that kind of matches the
them. So that's a whole concept of this nearest neighbors is you
And then you're basically like, okay, I'm going to take the label
So the first thing that we have to do is we have to define a
in, you know, 2d plots like this, our distance function is
And Euclidean distance is basically just this straight line
would be the Euclidean distance, it seems like there's this point,
that point, etc. So the length of this line, this green line that
as Euclidean distance. If we want to get technical with that, this
let me zoom in. The distance is equal to the square root of one
squared plus extend that square root, the same thing for y. So y
other squared. Okay, so we're basically trying to find the length,
between x and y, and then square each of those sum it up and take
going to erase this so it doesn't clutter my drawing. But anyways,
so here in the nearest neighbor algorithm, we see that there is a
telling us, okay, how many neighbors do we use in order to judge
we use a K of maybe, you know, three or five, depends on how big
I would say, maybe a logical number would be three or five. So
to three. Okay, well, of this data point that I drew over here,
Okay, so of this data point that I drew over here, it looks like
this one, this one. And then this one has a length of four. And
bit further than four. So actually, this would be these would be
points are blue. So chances are, my prediction for this point is
probably don't have a car. All right, now what if my point is
somewhere over here, let's say that a couple has four kids, and
well, now my closest points are this one, probably a little bit
right? Okay, still all pluses. Well, this one is more than likely
let me get rid of some of these just so that it looks a little bit
let's go through one more. What about a point that might be right
definitely this is the closest, right? This one's also closest.
the two of these. But if we actually do the mathematics, it seems
this one is right here. And this one is in between these two. So
than this one. And that means that that top one is the one that
what is the majority of the points that are close by? Well, we
here, and we have one minus here, which means that the pluses are
that this label is probably somebody with a car. Okay. So this is
work. It's that simple. And this can be extrapolated to further
know, if you have here, we have two different features, we have
the number of kids. But let's say we have 10 different features,
function so that it includes all 10 of those dimensions, we take
and then we figure out which one is the closest to the point that
that's K nearest neighbors. So now we've learned about K nearest
be able to do that within our code. So here, I'm going to label
And we're actually going to use a package from SK learn. So the
packages and so that we don't have to manually code all these
be really difficult. And chances are the way that we would code
or it'd be really slow, or I don't know a whole bunch of issues.
hand it off to the pros. From here, I can say, okay, from SK
neighbors, I'm going to import K neighbors classifier, because
so I run that. And our KNN model is going to be this K neighbors
a parameter of how many neighbors, you know, we want to use. So
we just use one. So now if I do K, and then model dot fit, I can
weight y train data. Okay. So that effectively fits this model.
why can and I guess yeah, let's do y predictions. And my y
dot predict. So let's use the test set x test. Okay. Alright, so
that we have those. But if I get my truth values for that test
we actually do. So just looking at this, we got five out of six of
actually take a look at something called the classification report
So if I go to from SK learn dot metrics, import classification
say, hey, print out this classification report for me. And let's
y test and the y prediction. We run this and we see we get this
to tell you guys a few things on this chart. Alright, this
pretty good. That's just saying, hey, if we just look at, you
what it's closest to, then we actually get an 82% accuracy, which
versus how many total are there. Now, precision is saying, okay,
for class one, or class zero and class one. What precision is
diagram over here, because I actually kind of like this diagram.
And on the left over here, we have everything that we know is
actually truly positive, that we've labeled positive in our
this is everything that's truly negative. Now in the circle, we
were labeled positive by our model. On the left here, we have
because you know, this side is the positive side and the side is
truly positive. Whereas all these ones out here, well, they should
are labeled as negative. And in here, these are the ones that
actually negative. And out here, these are truly negative. So
the ones we've labeled as positive, how many of them are true
okay, out of all the ones that we know are truly positive, how
so going back to this over here, our precision score, so again,
that we've labeled as the specific class, how many of them are
recall how out of all the ones that are actually this class, how
is 68% and 89%. Alright, so not too shabby, we can clearly see
like this, the class zero is worse than class one. Right? So that
hadrons and for our gammas. This f1 score over here is kind of a
recall score. So we're actually going to mostly look at this one
test data set. So here we have a measure of 72 and 87 or point
which is not too shabby. All right. Well, what if we, you know,
that, okay, so what was it originally with one? We see that our f1
point seven two and then point eight seven. And then our accuracy
three. Alright, so we've kind of increased zero at the cost of one
is 81. So let's actually just make this five. Alright, so you
we have 82% accuracy, which is pretty decent for a model that's
the next type of model that we're going to talk about is something
in order to understand the concepts behind naive Bayes, we have to
conditional probability and Bayes rule. So let's say I have some
this table right here. People who have COVID are over here in this
have COVID are down here in this green row. Now, what about the
tested positive are over here in this column. And people who have
this column. Okay. Yeah, so basically, our categories are people
people who don't have COVID, but test positive, so a false false
and test negative, which is a false negative, and people who don't
which good means you don't have COVID. Okay, so let's make this
in the margins, I've written down the sums of whatever it's
sum of this entire row. And this here might be the sum of this
question that I have is, what is the probability of having COVID
test? And in probability, we write that out like this. So the
line, that vertical line means given that, you know, some
okay, so what is the probability of having COVID given a positive
saying, okay, let's go into this condition. So the condition of
slice of the data, right? That means if you're in this slice of
given that we have a positive test, given in this condition, in
test. So what's the probability that we have COVID? Well, if we're
of people that have COVID is 531. So I'm gonna say that there's
now we divide that by the total number of people that have a
so that's the probability and doing a quick division, we get that
96.4%. So according to this data set, which is data that I made up
not actually real COVID data. But according to this data, the
that you tested positive is 96.4%. Alright, now with that, let's
this section here. Let's ignore this bottom part for now. So Bayes
the probability of some event A happening, given that B happened.
happened. This is our condition, right? Well, what if we don't
if we don't know what the probability of A given B is? Well, Bayes
can actually go and calculate it, as long as you have a
of A and the probability of B. Okay. And this is just a
so here we have Bayes rule. And let's actually see Bayes rule in
So here, let's say that we have some disease statistics, okay. So
And we know that the probability of obtaining a false positive is
false negative is 0.01. And the probability of the disease is 0.1.
the disease given that we got a positive test? Hmm, how do we even
what what do I mean by false positive? What's a different way to
is when you test positive, but you don't actually have the
that you have a positive test given no disease, right? And
it's a probability that you test negative given that you actually
that into a chart, for example, and this might be my positive and
be my diseases, disease and no disease. Well, the probability that
have no disease, okay, that's 0.05 over here. And then the false
testing negative, but I don't actually have the disease. This so
positive, and you don't have the disease, plus a probability that
don't have the disease, that should sum up to one. Okay, because
then you should have some probability that you're testing positive
testing negative. But that probability, in total should be one. So
negative and no disease, this should be the reciprocal, this
should be 0.95 because it's one minus whatever this probability
up here, this should be 0.99 because the probability that we, you
test negative and have the disease plus the probability that we
disease should equal one. So this is our probability chart. And
being point 0.1 just means I have 10% probability of actually of
in the general population, the probability that I have the disease
probability that I have the disease given that I got a positive
can write this out in terms of Bayes rule, right? So if I use this
probability of a positive test given that I have the disease times
divided by the probability of the evidence, which is my positive
Alright, now let's plug in some numbers for that. The probability
that I have the disease is 0.99. And then the probability that I
over here 0.1. Okay. And then the probability that I have a
what is the probability that I have a positive test given that I
and then having having the disease. And then the other case, where
negative test given or sorry, positive test giving no disease
having a disease. Okay, so I can expand that probability of having
these two different cases, I have a disease, and then I don't. And
having positive tests in either one of those cases. So that
plus 0.05. So that's the probability that I'm testing positive,
And the times the probability that I don't actually have the
0.1 probability that the population doesn't have the disease is
multiplication. And I get an answer of 0.6875 or 68.75%. Okay. All
that we can expand Bayes rule and apply it to classification. And
base. So first, a little terminology. So the posterior is this
Hey, what is the probability of some class CK? So by CK, I just
categories, so C for category or class or whatever. So category
dogs, category three, lizards, all the way, we have k categories,
So what is the probability of having of this specific sample x, so
of this one sample. What is the probability of x fitting into
so that that's what this is asking, what is the probability that,
this class, given all this evidence that we see the x's. So the
here, it's saying, Okay, well, given that, you know, assume,
class is class CK, okay, assume that this is a category. Well,
actually seeing x, all these different features from that
prior. So like in the entire population of things, what are the
probability of this class in general? Like if I have, you know, in
percentage? What is the chance that this image is a cat? How many
down here is called the evidence because what we're trying to do
we're creating this new posterior probability built upon the prior
right? And that evidence is a probability of x. So that's some
is a rule for naive Bayes. Whoa, okay, let's digest that a little
let me use a different color. What is this side of the equation
what is the probability that we are in some class K, CK, given
input, this is my second input, this is, you know, my third,
say that our classification is, do we play soccer today or not?
okay, is it how much wind is there? How much rain is there? And
So let's say that it's raining, it's not windy, but it's
So let's use Bayes rule on this. So this here
is equal to the probability of x one, x two, all these joint
times the probability of that class, all over the probability of
Okay. So what is this fancy symbol over here, this means
so how our equal sign means it's equal to this like little
proportional to okay, and this denominator over here, you might
the class like this, that number doesn't depend on the class,
for all of our different classes. So what I'm going to do is make
going to say that this probability x one, x two, all the way to x
to the numerator, I don't care about the denominator, because it's
single class. So this is proportional to x one, x two, x n given
that class. Okay. All right. So in naive Bayes, the point of it
this joint probability, we're just assuming that all of these
are all independent. So in my soccer example, you know, the
or the probability that, you know, it's windy, and it's rainy,
things are independent, we're assuming that they're independent.
actually write this part of the equation here as this. So each
all of them together. So the probability of the first feature,
times the probability of the second feature and given this
all the way up until, you know, the nth feature of given that it's
all of this. All right, which means that this here is now
expanded times this. So I'm going to write that out. So the
And I'm actually going to use this symbol. So what this means is
it means multiply everything to the right of this. So this
but do it for all the i's. So I, what is I, okay, we're going to
the first x i all the way to the nth. So that means for every
these probabilities together. And that's where this up here comes
oops, this should be a line to wrap this up in plain English.
is a probability that you know, we're in some category, given that
features is proportional to the probability of that class in
each of those features, given that we're in this one class that
of it, you know, of us playing soccer today, given that it's
Wednesday, is proportional to Okay, well, what is what is the
anyways, and then times the probability that it's rainy, given
times the probability that it's not windy, given that we're
we playing soccer when it's windy, how you know, and then how many
that's Wednesday, given that we're playing soccer. Okay. So how do
classification. So that's where this comes in our y hat, our
something called the arg max. And then this expression over here,
the arg max. Well, we want. So okay, if I write out this, again,
being in some class CK given all of our evidence. Well, we're
this expression on the right. That's what arc max means. So if K
one through K, so this is how many categories are, we're going to
to solve this expression over here and find the K that makes that
that instead of writing this, we have now a formula, thanks to
approximate that right in something that maybe we can we maybe we
we have the answers for that based on our training set. So this
these and finding whatever class whatever category maximizes this
this is something known as MAP for short, or maximum a
Pick the hypothesis. So pick the K that is the most probable so
of misclassification. Right. So that is MAP. That is naive Bayes.
just like how I imported k nearest neighbor, k neighbors
I can go to SK learn naive Bayes. And I can import Gaussian naive
Right. And here I'm going to say my naive Bayes model is equal.
had above. And I'm just going to say with this model, we are going
All right, just like above. So this, I might actually, so I'm
exactly, just like above, I'm going to make my prediction. So
naive Bayes model. And of course, I'm going to run the
just going to put these in the same cell. But here we have the y
is still our original test data set. So if I run this, you'll see
we get worse scores, right? Our precision, for all of them, they
you know, for our precision, our recall, our f1 score, they look
categories. And our total accuracy, I mean, it's still 72%, which
72%. Okay. Which, you know, is not not that great. Okay, so let's
Here, I've drawn a plot, I have y. So this is my label on one
my features. So let's just say I only have one feature in this
we see that, you know, I have a few of one class type down here.
because it's zero. And then we have our other class type one up
y. Okay. So many of you guys are familiar with regression. So
draw a regression line through this, it might look something like
doesn't seem to be a very good model. Like, why would we use this
Right? It's, it's iffy. Okay. For example, we might say, okay,
everything from here downwards would be one class type in here,
But when you look at this, you're just you, you visually can tell,
make sense. Things are not those dots are not along that line. And
are doing classification, not regression. Okay. Well, first of
this model, if we just use this line, it equals m x. So whatever
which is the y intercept, right? And m is the slope. But when we
is it actually y hat? No, it's not right. So when we're working
what we're actually estimating in our model is a probability,
and one, that is class zero or class one. So here, let's rewrite
Okay, well, m x plus b, that can range, you know, from negative
right? For any for any value of x, it goes from negative infinity
But probability, we know probably one of the rules of probability
between zero and one. So how do we fix this? Well, maybe instead
equal to that, we can set the odds equal to this. So by that, I
divided by one minus the probability. Okay, so now becomes this
take on infinite values. But there's still one issue here. Let me
The one issue here is that m x plus b, that can still be negative,
I have a negative slope, if I have a negative b, if I have some
but that can be that's allowed to be negative. So how do we fix
the log of the odds. Okay. So now I have the log of you know, some
the probability. And now that is on a range of negative infinity
because the range of log should be negative infinity to infinity.
the probability? Well, the first thing I can do is take, you know,
the not the e to the whatever is on both sides. So that gives me
over the one minus the probability is now equal to e to the m x
that out. So the probability is equal to one minus probability e
e to the m x plus b minus P times e to the m x plus b. And now we
one side. So if I do P, so basically, I'm moving this over, so I'm
to the m x plus b is equal to e to the m x plus b and let me
little bigger. So now my probability can be e to the m x plus b
Okay, well, let me just rewrite this really quickly, I want a
Okay, so what I'm going to do is I'm going to multiply this by
and then also the bottom by negative m x plus b, and I'm allowed
this over this is one. So now my probability is equal to one over
one plus e to the negative m x plus b. And now why did I rewrite
It's because this is actually a form of a special function, which
function. And for the sigmoid function, it looks something like
that some x is equal to one over one plus e to the negative x. So
is rewrite this in some sigmoid function, where the x value is
So maybe I'll change this to y just to make that a bit more clear,
the variable name is. But this is our sigmoid function. And
looks like is it goes from zero. So this here is zero to one. And
curved s, which I didn't draw too well. Let me try that again.
something if I can draw this right. Like that. Okay, so it goes in
And you might notice that this form fits our shape up here.
Oops, let's draw it sharper. But if it's our shape up there a lot
Alright, so that is what we call logistic regression, we're
to the sigmoid function. Okay. And when we only have, you know,
one feature x, and that's what we call simple logistic regression.
so that's only x zero, but then if we have x zero, x one, all the
multiple logistic regression, because there are multiple features
when we're building our model, logistic regression. So I'm going
And again, from SK learn this linear model, we can import logistic
And just like how we did above, we can repeat all of this. So
this log model, or LG logistic regression. I'm going to change
So I'm just going to use the default logistic regression. But
you see that you can use different penalties. So right now we're
an L2 penalty. But L2 is our quadratic formula. Okay, so that
you know, outliers, it would really penalize that. For all these
you can toggle these different parameters, and you might get
If I were building a production level logistic regression model,
would want to figure out how to do that. So I'm going to go ahead
I would want to figure out, you know, what are the best parameters
based on my validation data. But for now, we'll just we'll just
So again, I'm going to fit the X train and the Y train. And I'm
so I can just call this again. And instead of LG, NB, I'm going to
precision 65% recall 71, f 168, or 82 total accuracy of 77. Okay,
better than I base, but it's still not as good as K and N.
classification that I wanted to talk about is something called
or SVMs for short. So what exactly is an SVM model, I have two
x one on the axes. And then I've told you if it's you know, class
blue and red labels, my goal is to find some sort of line between
the data. Alright, so this line is our SVM model. So I call it a
line, but in 3d, it would be a plane and then you can also have
proper term is actually I want to find the hyperplane that best
classes. Let's see a few examples. Okay, so first, between these
and C, which one is the best divider of the data, which one has
or the other, or at least if it doesn't, which one divides it the
is has the most defined boundary between the two different groups.
pretty straightforward. It should be a right because a has a clear
know, everything on this side of a is one label, it's negative and
is the other label, it's positive. So what if I have a but then
like this, and my C, maybe like this, sorry, they're kind of the
But now which one is the best? So I would argue that it's still a,
Right? And why is it still a? Because in these other two, look at
to these points. Right? So if I had some new point that I wanted
say I didn't have A or B. So let's say we're just working with C.
that's right here. Or maybe a new point that's right there. Well,
looking at this. I mean, without the boundary, that would probably
right? I mean, it's pretty close to that other positive. So one
is something known as the margin. Okay, so not only do we want to
well, we also care about the boundary in between where the points
are, and the line that we're drawing. So in a line like this, the
might be like here. And I'm trying to draw these perpendicular.
if I switch over to these dotted lines, if I can draw this right.
are what's known as the margins. Okay, so these both here, these
And our goal is to maximize those margins. So not only do we want
two different classes, we want the line that has the largest
on the margin lines, the data. So basically, these are the data
divider. These are what we call support vectors. Hence the name
so the issue with SVM sometimes is that they're not so robust to
if I had one outlier, like this up here, that would totally change
vector to be, even though that might be my only outlier. Okay. So
in mind. As you know, when you're working with SVM is, it might
are outliers in your data set. Okay, so another example of SVMs
data like this, I'm just going to use a one dimensional data set
say we have a data set that looks like this. Well, our, you know,
perpendicular to this line. But it should be somewhere along this
anywhere like this. You might argue, okay, well, there's one here.
draw another one over here, right? And then maybe you can have two
SVMs work. But one thing that we can do is we can create some sort
that one thing I forgot to do was to label where zero was. So
Now, what I'm going to do is I'm going to say, okay, I'm going to
have x, sorry, x zero and x one. So x zero is just going to be my
x one equal to let's say, x squared. So whatever is this squared,
you know, maybe somewhere here, here, just pretend that it's
Right. And now my pluses might be something like
that. And I'm going to run out of space over here. So I'm just
use your imagination. But once I draw it like this, well, it's a
right? Now our SVM could be maybe something like this, this. And
our data set. Now it's separable where one class is this way. And
Okay, so that's known as SVMs. I do highly suggest that, you know,
mentioned, if you're interested in them, do go more in depth
do we how do we find this hyperplane? Right? I'm not going to go
because you're just learning what an SVM is. But it's a good idea
technique behind finding, you know, what exactly are the are the
that we're going to use. So anyways, this transformation that we
as the kernel trick. So when we go from x to some coordinate x,
what we're doing is we are applying a kernel. So that's why it's
So SVMs are actually really powerful. And you'll see that here. So
to import SVC. And SVC is our support vector classifier. So with
we are going to, you know, create SVC model. And we are going to,
could have just copied and pasted this, I should be able to do
again, fit this to X train, I could have just copied and pasted
done that. Okay, taking a bit longer. All right. Let's predict
let's see if I can hover over this. Right. So again, you see a lot
parameters here that you can go back and change if you were
but in this specific case, we'll just use it out of the box again.
you'll note that Wow, the accuracy actually jumps to 87% with the
there's nothing less than, you know, point eight, which is great.
I mean, everything's at 0.9, which is higher than anything that we
So so far, we've gone over four different classification models,
logistic regression, naive Bayes and cannon. And these are just
them. Each of these they have different, you know, they have
go and you can toggle. And you can try to see if that helps later
they perform, they give us around 70 to 80% accuracy. Okay, with
let's see if we can actually beat that using a neural net. Now the
I wanted to talk about is known as a neural net or neural network.
like this. So you have an input layer, this is where all your
all these arrows pointing to some sort of hidden layer. And then
sort of output layer. So what is what is all this mean? Each of
something known as a neuron. Okay, so that's a neuron. In a neural
features that we're inputting into the neural net. So that might
x n. Right. And these are the features that we talked about there,
the pregnancy, the BMI, the age, etc. Now all of these get
are multiplied by some w number that applies to that one specific
feature. So these two get multiplied. And the sum of all of these
so basically, I'm taking w zero times x zero. And then I'm adding
I'm adding you know, x two times w two, etc, all the way to x n
input into the neuron. Now I'm also adding this bias term, which
to shift this by a little bit. So I might add five or I might add
I don't know. But we're going to add this bias term. And the
the sum of this, this, this and this, go into something known as
okay. And then after applying this activation function, we get an
neuron would look like. Now a whole network of them would look
So I kind of gloss over this activation function. What exactly is
looks like if we have all our inputs here. And let's say all of
of addition, right? Then what's going on is we're just adding a
the some sort of weight times these input layer a bunch of times.
and factor that all out, then this entire neural net is just a
layers, which I don't know about you, but that just seems kind of
literally just write that out in a formula, why would we need to
we wouldn't. So the activation function is introduced, right? So
function, this just becomes a linear model. An activation function
this. And as you can tell, these are not linear. And the reason
our entire model doesn't collapse on itself and become a linear
something known as a sigmoid function, it runs between zero and
one all the way to one. And this is ReLU, which anything less than
greater than zero is linear. So with these activation functions,
is no longer just the linear combination of these, it's some sort
that the input into the next neuron is, you know, it doesn't it
become linear, because we've introduced all these nonlinearities.
model, the loss, right? And then we do this thing called training,
back into the model, and make certain adjustments to the model to
Let's talk a little bit about the training, what exactly goes on
Let's go back and take a look at our L2 loss function. This is
looks like it's a quadratic formula, right? Well, up here, the
large. And our goal is to get somewhere down here, where the loss
means that our predicted value is closer to our true value. So
this way. Okay. And thanks to a lot of properties of math,
gradient descent, in order to follow this slope down this way.
different slopes with respect to some value. Okay, so the loss
w zero, versus w one versus w n, they might all be different.
think about it is, to what extent is this value contributing to
figure that out through some calculus, which we're not going to
But if you want to learn more about neural nets, you should
and figure out what exactly back propagation is doing, in order to
how much do we have to backstep by. So the thing is here, you
this curve at all of these different points. And the closer we get
this step becomes. Now stick with me here. So my new value, this
I'm going to take w zero, and I'm going to set some new value for
set for that is the old value of w zero, plus some factor, which
times whatever this arrow is. So that's basically saying, okay,
and just decrease it this way. So I guess increase it in this
this direction. But this alpha here is telling us, okay, don't
just in case we're wrong, take a small step, take a small step in
closer. And for those of you who, you know, do want to look more
the reason why I use a plus here is because this here is the
just the if you were to use the actual gradient, this should be a
Now this alpha is something that we call the learning rate. Okay,
we're taking steps. And that might, you know, tell our that that
how long it takes for our neural net to converge. Or sometimes if
diverge. But with all of these weights, so here I have w zero, w
update to all of them after we calculate the loss, the gradient of
weight. So that's how back propagation works. And that is
calculate the loss, we're calculating gradients, making
all the all the weights to something adjusted slightly. And then
gradient. And then we're saying, Okay, let's take the training set
again, and go through this loop all over again. So for machine
libraries that we use, right, we've already seen SK learn. But
networks, this is kind of what we're trying to program. And it's
do this from scratch, because not only will we probably have a lot
not going to be fast enough, right? Wouldn't it be great if there
full time professionals that are dedicated to solving this
just give us their code that's already running really fast? Well,
And that's why we use TensorFlow. So TensorFlow makes it really
we also have enough control over what exactly we're feeding into
this line here is basically saying, Okay, let's create a
just, you know, what we've seen here, it just goes one layer to
a dense layer means that all of them are interconnected. So here,
nodes, and this one's all these, and then this one gets connected
So we're going to create 16 dense nodes with relu activation
to create another layer of 16 dense nodes with relu activation.
to be just one node. Okay. And that's how easy it is to define
is an open source library that helps you develop and train your ML
for a neural net. So we're using a neural net for classification.
we are going to use TensorFlow, and I don't think I imported that
that down here. So I'm going to import TensorFlow as TF. And
is going to be, I'm going to use this. So essentially, this is
things that I'm about to pass in. So yeah, layer them linear stack
And what that means, nope, not that. So what that means is I can
some sort of layer, and I'm just going to use a dense layer.
Oops, dot dense. And let's say we have 32 units. Okay, I will
set the activation as really. And at first we have to specify the
and comma. Alright. Alright, so that's our first layer. Now our
another dense layer of 32 units all using relu. And that's it. So
just going to be my output layer, it's going to just be one node.
be sigmoid. So if you recall from our logistic regression, what
a sigmoid, it looks something like this, right? So by creating a
we're essentially projecting our predictions to be zero or one,
And that's going to help us, you know, we can just round to zero
Okay. So this is my neural net model. And I'm going to compile
we have to compile it. It's really cool, because I can just
I want, and it'll do it. So here, if I go to optimizers, I'm
And you'll see that, you know, the learning rate is 0.001. So I'm
So 0.001. And my loss is going to be binary cross entropy. And the
include on here, so it already will consider loss, but I'm, I'm
So we can actually see that in a plot later on. Alright, so I'm
And one thing that I'm going to also do is I'm going to define
actually copying and pasting this, I got these from TensorFlow. So
tutorial, they actually have these, this like, defined. And that's
So I'm actually going to move this cell up, run that. So we're
over all the different epochs. epochs means like training cycles.
means like training cycles. And we're going to plot the accuracy
Alright, so we have our model. And now all that's left is, let's
So I'm going to say history. So TensorFlow is great, because it
of the training, which is why we can go and plot it later on. Now
this neural net model. And fit that with x train, y train, I'm
equal to let's say just let's just use 100 for now. And the batch
let's say 32. Alright. And the validation split. So what the
here somewhere. Okay, so yeah, this validation split is just the
to be used as validation data. So essentially, every single epoch,
saying, leave certain if this is point two, then leave 20% out.
model performs on that 20% that we've left out. Okay, so it's
set. But TensorFlow does it on our training data set during the
outside of just our validation data set to see, you know, what's
I'm going to make that 0.2. And we can run this. So if I run that,
to set verbose equal to zero, which means, okay, don't print
for 100 epochs might get kind of annoying. So I'm just going to
and then we'll see what happens. Cool, so it finished training.
because you know, I've already defined these two functions, I can
oops, loss of that history. And I can also plot the accuracy
So this is a little bit ish what we're looking for. We definitely
decreasing loss and an increasing accuracy. So here we do see
accuracy improves from around point seven, seven or something all
point, maybe eight one. And our loss is decreasing. So this is
loss and accuracy is performing worse than the training loss or
our model is training on that data. So it's adapting to that data.
you know, stuff that it hasn't seen yet. So, so that's why. So in
we could change a bunch of the parameters, right? Like I could
a row of 64 nodes, and then 32, and then one. So I can change some
And a lot of machine learning is trying to find, hey, what do we
So what I'm actually going to do is I'm going to rewrite this so
known as a grid search. So we can search through an entire space
we have 64 nodes and 64 nodes, or 16 nodes and 16 nodes, and so
we can, you know, we can change this learning rate, we can change
you know, the batch size, all these things might affect our
I'm also going to add what's known as a dropout layer in here. And
saying, hey, randomly choose with at this rate, certain nodes, and
in a certain iteration. So this helps prevent overfitting. Okay,
define this as a function called train model, we're going to pass
the number of nodes, the dropout, you know, the probability that
learning rate. So I'm actually going to say lr batch size. And we
right? I mentioned that as a parameter. So indent this, so it goes
I'm going to set this equal to number of nodes. And now with the
to set dropout prob. So now you know, the probability of turning
is equal to dropout prob. And I'm going to keep the output layer
but this here is now going to be my learning rate. And I still
accuracy. We are actually going to train our model inside of this
epochs equal epochs, and this is equal to whatever, you know,
y train belong right here. Okay, so those are getting passed in as
end, I'm going to return this model and the history of that model.
is let's just go through all of these. So let's say let's keep
do is I can say, hey, for a number of nodes in, let's say, let's
happens for the different dropout probabilities. And I mean, zero
Also, to see what happens. You know, for the learning rate in
maybe we want to throw on 0.1 in there as well. And then for the
64 as well. Actually, and let's also throw in 128. Actually, let's
so 128 in there. That should be 01. I'm going to record the model
train model here. So we're going to do x train y train, the number
you know, the number of nodes that we've defined here, dropout,
Okay. And then now we have both the model and the history. And
I want to plot the loss for the history. I'm also going to plot
Probably should have done them side by side, that probably would
Okay, so what I'm going to do is split up, split this up. And that
the subplots. So now this is just saying, okay, I want one row and
plots. Okay, so I'm going to plot on my axis one, the loss. I
work. Okay, we don't care about the grid. Yeah, let's let's keep
So now on here, I'm going to plot all the accuracies on the second
I might have to debug this a bit.
We should be able to get rid of that. If we run this, we already
in here. So if I just run it on this, okay, it has no attribute x
it's like set x label or something. Okay, yeah, so it's, it's set
So let's see if that works. All right, cool. Um, and let's
Okay, so we can actually change the figure size that I'm gonna
set that to. Oh, that's not the way I wanted it. Okay, so that
And that's just going to be my plot history function. So now I can
Here, I'm going to plot the history. And what I'm actually going
I'm going to print out all these parameters. So I'm going to print
the F string to print out all of this stuff. So here, I'm going to
Uh, all of this stuff. So here, I'm printing out how many nodes,
And we already know how many you found, so I'm not even going to
So once we plot this, uh, let's actually also figure out what the,
losses on our validation set that we have that we created all the
Alright, so remember, we created three data sets. Let's call our
validation data with the validation data sets loss would be. And I
let's say I want to record whatever model has the least validation
first, I'm going to initialize that to infinity so that you know,
So if I do float infinity, that will set that to infinity. And
track of the parameters. Actually, it doesn't really matter. I'm
the model. And I'm gonna set that to none. So now down here, if
less than the least validation loss, then I am going to simply
Hey, this validation for this least validation loss is now equal
And the least loss model is whatever this model is that just
So we are actually just going to let this run for a while. And
last model after that. So let's just run. All right, and now we
All right, so we've finally finished training. And you'll notice
actually gets to like 0.29. The accuracy is around 88%, which is
okay, why is this accuracy in this? Like, these are both the
is on the validation data set that we've defined at the beginning,
this is actually taking 20% of our tests, our training set every
and saying, Okay, how much of it do I get right now? You know,
train with any of that. So they're slightly different. And
that I probably you know, probably what I should have done is over
the model fit, instead of the validation split, you can define the
And you can pass in the validation data, I don't know if this is
that's probably what I should have done. But instead, you know,
we have here. So you'll see at the end, you know, with the 64
performance 64 nodes with a dropout of 0.2, a learning rate of
And it does seem like yes, the validation, you know, the fake
loss is decreasing, and then the accuracy is increasing, which is
so finally, what I'm going to do is I'm actually just going to
this model, which we've called our least loss model, I'm going to
and I'm going to predict x test on that. And you'll see that it
are really close to zero and some that are really close to one.
output. So if I do this, and what I can do is I can cast them. So
greater than 0.5, set that to one. So if I actually, I think what
Oh, okay, so I have to cast that as type. And so now you'll see
actually going to transform this into a column as well. So here
I didn't mean to do that. Okay, no, I wanted to just reshape it to
Okay. And using that we can actually just rerun the classification
neural net output. And you'll see that okay, the the F ones are
seems like what happened here is the precision on class zero. So
but the recall decreased. But the F one score is still at a good
class, it looked like the precision decreased a bit the recall
That's also been increased. I think I interpreted that properly. I
work and we got a model that performs actually very, very
had earlier. And the whole point of this exercise was to
define your models. But it's also to say, hey, maybe, you know,
powerful, as you can tell. But sometimes, you know, an SVM or some
appropriate. But in this case, I guess it didn't really matter
accuracy score is still pretty good. So yeah, let's now move on to
We just saw a bunch of different classification models. Now let's
the other type of supervised learning. If we look at this plot
data points. And here we have our x value for those data points.
value, which is now our label. And when we look at this plot,
the line of best fit that best models this data. Essentially,
some new value of x that we don't have in our sample, we're trying
prediction for y be for that given x value. So that, you know,
I don't know. But remember, in regression that, you know, given
we're trying to predict some continuous numerical value for y.
In linear regression, we want to take our data and fit a linear
our linear model might look something along the lines of here.
considered as maybe our line of best fit. And this line is modeled
it down here, y equals b zero, plus b one x. Now b zero just means
extend this y down here, this value here is b zero, and then b one
line, defines the slope of this line. Okay. All right. So that's
for linear regression. And how exactly do we come up with that
with this linear regression? You know, we could just eyeball where
not very good at eyeballing certain things like that. I mean, we
better at giving us a precise value for b zero and b one. Well,
something known as a residual. Okay, so residual, you might also
And what that means is, let's take some data point in our data
far off is our prediction from a data point that we already have.
this is 12345678. So this is y eight, let's call it, you'll see
I in order to represent, hey, just one of these points. Okay. So
would be the prediction. Oops, this here would be the prediction
with this hat. Okay, if it has a hat on it, that means hey, this
my prediction for you know, this specific value of x. Okay. Now
here between y eight and y hat eight. So y eight minus y hat
give us this here. And I'm just going to take the absolute value
the line, right, then you would get a negative value, but distance
just going to put a little hat, or we're going to put a little
And that gives us the residual or the error. So let me rewrite
to all the points, I'm going to say the residual can be calculated
So this just means the distance between some given point, and its
prediction on the line. So now, with this residual, this line of
decrease these residuals as much as possible. So now that we have
our line of best fit is trying to decrease the error as much as
data points. And that might mean, you know, minimizing the sum of
here, this is the sum symbol. And if I just stick the residual
it looks something like that, right. And I'm just going to say,
data set, so for all the different points, we're going to sum up
to try to decrease that with my line of best fit. So I'm going to
me the lowest value of this. Okay. Now in other, you know,
we might attach a squared to that. So we're trying to decrease the
And what that does is it just, you know, it adds a higher penalty
you know, points that are further off. So that is linear
this equation, some line of best fit that will help us decrease
with respect to all the data points that we have in our data set,
the best prediction for all of them. This is known as simple
And basically, that means, you know, our equation looks something
multiple linear regression, which just means that hey, if we have
think of our feature vectors, we have multiple values in our x
look something more like this. Actually, I'm just going to say
up with some coefficient for all of the different x values that I
might have noticed that I have some assumptions over here. And you
what in the world do these assumptions mean? So let's go over
So let's go over them. The first one is linearity.
And what that means is, let's say I have a data set. Okay.
Linearity just means, okay, my does my data follow a linear
increases? Or does y decrease at as x increases? Does so if y
rate as x increases, then you're probably looking at something
nonlinear data set? Let's say I had data that might look something
visually judging this, you might say, okay, seems like the line of
curve like this. Right. And in this case, we don't satisfy that
So with linearity, we basically just want our data set to follow
And independence, our second assumption
just means this point over here, it should have no influence on
or this point over here, or this point over here. So in other
all the samples in our data set should be independent. Okay, they
one another, they should not affect one another.
Okay, now, normality and homoscedasticity, those are concepts
I have a plot that looks something like this, and I have a plot
something like this. And my line of best fit is somewhere here,
In order to look at these normality and homoscedasticity
the residual plot. Okay. And what that means is I'm going to keep
of plotting now where they are relative to this y, I'm going to
going to plot y minus y hat like this. Okay. And now you know,
so it might be here, this one down here is negative, it might be
it's literally just a plot of how you know, the values are
fit. So it looks like it might, you know, look something like
residual plot. And what normality means, so our assumptions are
I might have butchered that spelling, I don't really know. But
saying, okay, these residuals should be normally distributed.
it should follow a normal distribution. And now what
of these points should remain constant throughout. So this spread
same as this spread over here. Now, what's an example of where you
not held? Well, let's say that our original plot actually looks
Okay, so now if we looked at the residuals for that, it might look
like that. And now if we look at this spread of the points, it
is not constant, which means that homoscedasticity, this
might not be appropriate to use linear regression. So that's just
we have a bunch of data points, we want to predict some y value
up with this line of best fit that best describes, hey, given some
guess of what y is. So let's move on to how do we evaluate a
measure that I'm going to talk about is known as mean absolute
for short, okay. And mean absolute error is basically saying, all
all the errors. So all these residuals that we talked about, let's
for all of them, and then take the average. And then that can
we. So the mathematical formula for that would be, okay, let's
Alright, so this is the distance. Actually, let me redraw a plot
suppose I have a data set, look like this. And here are all my
say my line looks something like that. So my mean absolute error
values. This was a mistake. So summing up all of these, and then
I have. So what would be all the residuals, it would be y i,
minus y hat i, so the prediction for that on here. And then we're
all of the different i's in our data set. Right, so i, and then we
we have. So actually, I'm going to rewrite this to make it a
whatever the first data point is all the way through the nth data
it by n, which is how many points there are. Okay, so this is our
telling us, okay, in on average, this is the distance between our
actual value in our training set. Okay. And mae is good because it
get this value here, we can literally directly compare it to
So let's say y is we're talking, you know, the prediction of the
dollars. Once we have once we calculate the mae, we can literally
price, the average, how much we're off by is literally this many
mean absolute error. An evaluation technique that's also closely
squared error. And this is MSE for short. Okay. Now, if I take
and move it down here, well, the gist of mean squared error is
of the absolute value, we're going to square. So now the MSE is
okay, let's sum up something, right, so we're going to sum up all
So now I'm going to do y i minus y hat i. But instead of absolute
I'm going to square them all. And then I'm going to divide by n in
basically, now I'm taking all of these different values, and I'm
them to one another. And then I divide by n. And the reason why we
is that it helps us punish large errors in the prediction. And
because of differentiability, right? So a quadratic equation is
if you're familiar with calculus, a quadratic equation is
value function is not totally differentiable everywhere. But if
don't worry about it, you won't really need it right now. And now
error is that once I calculate the mean squared error over here,
want to compare the values. Well, it gets a little bit trickier to
error is in terms of y squared, right? It's this is now squared.
you know, how many dollars off am I I'm talking how many dollars
you know, to humans, it doesn't really make that much sense. Which
something known as the root mean squared error. And I'm just going
because it's very, very similar to mean squared error. Except now
Okay, so this is our messy, and we take the square root of that
term in which you know, we're defining our error is now in terms
So that's a pro of root mean squared error is that now we can say,
to this metric is this many dollar signs off from our predictor.
which is one of the pros of root mean squared error. And now
of determination, or r squared. And this is a formula for r
to one minus RSS over TSS. Okay, so what does that mean?
of the squared residuals. So maybe it should be SSR instead, but
RSS sum of the squared residuals, and this is equal to if I take
and I take y i minus y hat, i, and square that, that is my RSS,
residuals. Now TSS, let me actually use a different color for
So TSS is the total sum of squares.
And what that means is that instead of being with respect to this
take each y value and just subtract the mean of all the y values,
actually, let's use a different color. Let's use green. If this
so RSS is giving me this measure here, right? It's giving me some
our regressor that we predicted. Actually, I'm gonna take this
and actually, I'm going to use red for that. Well, TSS, on the
how far off are these values from the mean. So if we literally
line of best fit, if we just took all the y values and average all
this is the average value for every single x value, I'm just going
instead, then it's asking, okay, how far off are all these points
Okay, and remember that this square means that we're punishing
they look somewhat close in terms of distance, the further a few
the larger our total sum of squares is going to be. Sorry, that
squares is taking all of these values and saying, okay, what is
any regressor, and I literally just calculated the average of all
and for every single x value, I'm just going to predict that
like, that means that maybe y and x aren't associated with each
best thing that I can do for any new x value, just predict, hey,
And this total sum of squares is saying, okay, well, with respect
what is our error? Right? So up here, the sum of the squared
our what what is our error with respect to this line of best fit?
saying what is the error with respect to, you know, just the
of best fit is a better fit, then this total sum of squares, that
that means that this numerator is going to be smaller than this
And if our errors in our line of best fit are much smaller, then
of the RSS over TSS is going to be very small, which means that R
one. And now when R squared is towards one, that means that that's
good predictor. It's one of the signs, not the only one. So over
that there's this adjusted R squared. And what that does, it just
So x1, x2, x3, etc. It adjusts for how many extra terms we add,
you know, add an extra term, the R squared value will increase
y some more. But the value for the adjusted R squared increase if
improves this model fit more than expected, you know, by chance.
R squared is. I'm not, you know, it's out of the scope of this one
And now that's linear regression. Basically, I've covered the
And, you know, how do we use that in order to find the line of
our computer can do all the calculations for us, which is nice.
it's trying to minimize that error, right? And then we've gone
ways of actually evaluating a linear regression model and the pros
So now let's look at an example. So we're still on supervised
talk about regression. So what happens when you don't just want to
What happens if you actually want to predict a certain value? So
learning repository. And here I found this data set about bike
So this data set is predicting rental bike count. And here it's
hour. So what we're going to do, again, you're going to go into
to download this CSV file. And we're going to move over to collab
this FCC bikes and regression. I don't remember what I called the
regression. Now I'm going to import a bunch of the same things
I'm going to also continue to import the oversampler and the
also just going to let you guys know that I have a few more things
library that lets us copy things. Seaborn is a wrapper over a
to plot certain things. And then just letting you know that we're
TensorFlow. Okay, so one more thing that we're also going to be
sklearn linear model library. Actually, let me make my screen a
awesome. Run this and that'll import all the things that we need.
you know, give some credit to where we got this data set. So let
And I will also give credit to this here.
Okay, cool. All right, cool. So this is our data set. And again,
attributes that we have right here. So I'm actually going to go
Feel free to copy and paste this if you want me to read it out
It's byte count, hour, temp, humidity, wind, visibility, dew
snow, and functional, whatever that means. Okay, so I'm going to
by dragging and dropping. All right. Now, one thing that you guys
you might actually have to open up the CSV because there were, at
characters in mine, at least. So you might have to get rid of
but my computer wasn't recognizing it. So I got rid of that. So
and get rid of some of those labels that are incorrect. I'm going
after we've done that, we've imported in here, I'm going to create
all right, so now what I can do is I can read that CSV file and I
So so like data dot CSV. Okay, so now if I call data dot head,
various labels, right? And then I have the data in there. So I'm
going to get rid of some of these columns that, you know, I don't
I'm going to, when I when I type this in, I'm going to drop maybe
holiday, and the various seasons. So I'm just not going to care
one means drop it from the columns. So now you'll see that okay,
I guess you don't really notice it. But if I set the data frames
and I look at, you know, the first five things, then you'll see
It's a lot easier to read. So another thing is, I'm actually going
df functional. And we're going to create this. So remember that
at language, we want it to be in zeros and ones. So here, I will
Well, if this is equal to yes, then that that gets mapped as one.
Great. Cool. So the thing is, right now, these by counts are for
to make this example simpler, I'm just going to index on an hour,
we're only going to use that specific hour. So I'm just going to
going to use an hour. So here, let's say. So this data frame is
the hour, let's say it equals 12. Okay, so it's noon. All right.
equal to 12. And I'm actually going to now drop that column. Our
so we run this cell. Okay, so now we got rid of the hour in here.
the temperature, humidity, wind, visibility, and yada, yada, yada.
is I'm going to actually plot all of these. So for i in all the
whatever its data frame is, and all the columns, because I don't
actually, it's my first thing. So what I'm going to do is say for
columns, everything after the first thing, so that would give me
onwards. So these are all my features, right? I'm going to just
label how that specific data, how that affects the by count. So
the y axis. And I'm going to plot, you know, whatever the specific
And I'm going to title this, whatever the label is. And, you know,
at noon. And the x label as just the label. Okay, now, I guess we
We don't even need the legend. So just show that plot. All right.
not really doesn't really give us any utility. So then snow rain
you know, is fairly linear dew point temperature, visibility, wind
much humidity, kind of maybe like an inverse relationship. But the
looks like there's a relationship between that and the number of
going to do is I'm going to drop some of the ones that don't don't
maybe wind, you know, visibility. Yeah, so I'm going to get rid of
So now data frame, and I'm going to drop wind, visibility, and
axis again is the column. So that's one. So if I look at my data
temperature, the humidity, the dew point temperature, radiation,
what I want to do is I want to split this into my training, my
just as we talked before. Here, we can use the exact same thing
numpy dot split, and sample, you know that the whole sample, and
of the data frame. And we're going to do that. But now set this to
So I don't really care about, you know, the the full grid, the
use an underscore for that variable. But I will get my training x
have a function for getting the x and y's. So here, I'm going to
get x y. And I'm going to pass in the data frame. And I'm actually
of the y label is, and what the x what specific x labels I want to
then I'm just like, like, I'm only going to I'm going to get
not the wildlife. So here, I'm actually going to make first a deep
And that basically means I'm just copying everything over. If, if
so if not x labels, then all I'm going to do is say, all right, x
data frame is. And I'm just going to take all the columns. So C
if C does not equal the y label, right, and I'm going to get the
is the x labels, well, okay, so in order to index only one thing,
one thing in here, then my data frame is, so let me make a case
labels is equal to one, then what I'm going to do is just say that
and add that just that label values, and I actually need to
So I'm going to pass in negative one comma one there. Now,
specific x labels that I want to use, then I'm actually just going
frame of those x labels, dot values. And that should suffice.
extracting x. And in order to get my y, I'm going to do y equals
label. And at the very end, I'm going to say data equals NP dot h
one next to each other. And I'll take x and y, and return that.
And I'm actually going to reshape this to make it 2d as well so
And I will return data x, y. So now I should be able to say, okay,
frame. And the y label, so my y label is byte count. And actually,
going to let's just do like one dimension right now. And earlier,
had seen that maybe, you know, the temperature dimension does
to use that to predict why. So I'm going to label this also that,
temperature. And I am also going to do this again for, oh, this
validation. And this should be a test. Because oh, that's Val.
And this should be test. Alright, so we run this and now we have
data sets for just the temperature. So if I look at x train temp,
Okay, and I'm doing this first to show you simple linear
create a regressor. So I can say the temp regressor here. And then
linear regression model. And just like before, I can simply fix
in order to train train this linear regression model. Alright, and
this regressor is coefficients and the intercept. So if I do that,
for whatever the temperature is, and then the the x intercept,
right. And I can, you know, score, so I can get the the r squared
and y test. All right, so it's an r squared of around point three
zero, which would mean, hey, there's absolutely no association.
good, it depends on the context. But, you know, the higher that
the two variables would be correlated, right? Which here, it's all
maybe some association between the two. But the reason why I want
you, you know, if we plotted this, this is what it would look
and let's take the training. So this is our data. And then let's
also plotted, so something that I can do is say, you know, the x
is when space, and this goes from negative 20 to 40, this piece of
let's take 100 things from there. So I'm going to plot x, and I'm
this, like, regressor, and predict x with that. Okay, and this
the fit. And this color, let's make this red. And let's actually
I can change how thick that value is. Okay. Now at the very end,
all right, let's also create, you know, title, all these things
here, let's just say, this would be the bikes, versus the
would be number of bikes. And the x label would be the
might cause an error. Yeah. So it's expecting a 2d array. So we
Okay, there we go. So I just had to make this an array and then
we see that, all right, this increases. But again, remember those
linear regression, like this, I don't really know if this fits
wanted to show you guys though, that like, all right, this is what
data would look like. Okay. Now, we can do multiple linear
and do that as well. Now, if I take my data set, and instead of
my current data set right now. Alright, so let's just use all of
right. So I'm going to just say for the x labels, let's just take
remove the byte count. So does that work? So if this part should
this should work now. Oops, sorry. Okay, so I have Oh, but this
temperature anymore, we should actually do this, let's say all,
rerun this piece here so that we have our temperature only data
all data set. Okay. And this regressor, I can do the same thing.
And I'm going to make this the linear regression. And I'm going to
train all. Okay. Alright, so let's go ahead and also score this
R squared performs now. So if I test this on the test data set,
square seems to improve it went from point four to point five,
And I can't necessarily plot, you know, every single dimension.
to say, okay, this is this is improved, right? Alright, so one
tensorflow is you can actually do regression, but with the neural
to we already have our our training data for just the temperature
different columns. So I'm not going to bother with splitting up
ahead and start building the model. So in this linear regression
it does help if we normalize it. So that's very easy to do with
normalizer layer. So I'm going to do tensorflow Keras layers, and
And the input shape for that will just be one because let's just
temperature and the access I will make none. Now for this temp
an equal sign there. I'm going to adapt this to X train temp, and
So that should work great. Now with this model, so temp neural net
you know, dot keras, sequential. And I'm going to pass in this
going to say, hey, just give me one single dense layer with one
is saying, all right, well, one single node just means that it's
sort of activation function to it, the output is also linear. So
Keras layers dot dense. And I'm just going to have one unit. And
So with this model, let's compile. And for our optimizer, let's
let's use the atom again, dot atom, and we have to pass in the
and our learning rate, let's do 0.01. And now, the loss, we
loss, I'm going to do mean squared error. Okay, so we run that
And just like before, we can call history. And I'm going to fit
if I call fit, I can just fit it, and I'm going to take the x
but reshape it. Y train for the temperature. And I'm going to set
that it doesn't, you know, display stuff. I'm actually going to
1000. And the validation data should be let's pass in the
as a tuple. And I know I spelled that wrong. So let's just run
And up here, I've copied and pasted the plot loss from our
to MSC. Because now we're talking we're dealing with mean squared
the loss of this history after it's done. So let's just wait for
plot. Okay, so this actually looks pretty good. We see that the
this actually looks pretty good. We see that the values are
I'm going to go back up and take this plot. And we are going to
here, instead of this temperature regressor, I'm going to use the
And if I run that, I can see that, you know, this also gives me a
you'll notice that this this fit is not entirely the same as the
up here. And that's due to the training process of, you know, of
different ways to try and try to find the best linear regressor.
propagation to train a neural net node, whereas in the other one,
Okay, they're probably just trying to actually compute the line of
well, we can repeat the exact same exercise with our with our
but I'm actually going to skip that part. I will leave that as an
so now what would happen if we use a neural net, a real neural net
one single node in order to predict this. So let's start on that
normalizer. So I'm actually going to take the same setup here. But
one dense layer, I'm going to set this equal to 32 units. And for
Relu. And now let's duplicate that. And for the final output, I
want one cell. And this activation is also going to be Relu,
zero bytes. So I'm just going to set that as Relu. I'm just going
Okay. And at the bottom, I'm going to have this neural net model.
net model, I'm going to compile. And I will actually use the same
instead of a learning rate of 0.01, I'll use 0.001. Okay. And I'm
So the history is this neural net model. And I'm going to fit that
temp, and valid validation data, I'm going to set this again equal
Now, for the verbose, I'm going to say equal to zero epochs, let's
size, actually, let's just not do a batch size right now. Let's
here. And again, we can plot the loss of this history after it's
run this. And that's not what we're supposed to get. So what is
we have our temperature normalizer, which I'm wondering now if we
Do that. Okay, so we do see this decline, it's an interesting
So this is our loss, which all right, if decreasing, that's a good
And actually, what's interesting is let's just let's plot this
And you'll see that we actually have this like, curve that looks
what if I got rid of this activation? Let's train this again. And
Alright, so even even when I got rid of that really at the end, it
it's not the best model, if we had maybe one more layer in here,
to play around with. When you're, you know, working with machine
know what the best model is going to be. For example, this also is
it's okay. So my point is, though, that with a neural net, I mean,
there's like no data down here, right? So it's kind of hard for
we probably should have started the prediction somewhere around
with this neural net model, you can see that this is no longer a
get an estimate of the value, right? And we can repeat this exact
do that. Right. And we can repeat this exact same exercise with
if I now pass in all of the data, so this is my all normalizer
and I should just be able to pass in that. So let's move this to
I'm going to pass in my all normalizer. And let's compile it.
Great. So here with the history, when we're trying to fit this
we're going to use our larger data set with all the features. And
And of course, we want to plot the loss.
Okay, so that's what our loss looks like. So an interesting curve,
So before we saw that our R squared score was around point five,
that with a neural net anymore. But one thing that we can measure
error, right? So if I come down here, and I compare the two mean
so I can predict x test all right. So these are my predictions
will linear multiple multiple linear regressor. So these are my
Okay. I'm actually going to do that at the bottom. So let me just
it down here. So now I'm going to calculate the mean squared error
and the neural net. Okay, so this is my linear and this is my
model, and I predict x test all, I get my two, you know, different
the mean squared error, right? So if I want to get the mean
and y real, I can do numpy dot square, and then I would need the y
real. So this this is basically squaring everything. And this
this entire thing and take the mean of that, that should give me
And the y real is y test all, right? So that's my mean squared
And this is my mean squared error for the neural net. So that's
I guess. So my guess is that it's probably coming from this
shape is probably just six. And okay, so that works now. And the
my inputs are only for every vector, it's only a one dimensional
have I should have just had six, comma, which is a tuple of size
a tuple containing one element, which is a six. Okay, so it's
net results seem like they they have a larger mean squared error
One thing that we can look at is, we can actually plot the real
results versus what the predictions are. So if I say, some access,
axes and make these equal, then I can scatter the the y, you know,
values are on the x axis, and then what the prediction are on the
label this as the linear regression predictions. Okay, so then let
x axis, I'm going to say is the true values. The y axis is going
Or actually, let's plot. Let's just make this predictions.
And then at the end, I'm going to plot. Oh, let's set some
Because I think that's like approximately the max number of
So I'm going to set my x limit to this and my y limit to this.
So here, I'm going to pass that in here too. And all right, this
linear regressor. You see that actually, they align quite well, I
probably too much 2500. I mean, looks like maybe like 1800 would
And I'm actually going to label something else, the neural net
Let's add a legend. So you can see that our neural net for the
it's a little bit more spread out. And it seems like we tend to
here in this area. Okay. And for some reason, these are way off as
But yeah, so we've basically used a linear regressor and a neural
sometimes where a neural net is more appropriate and a linear
I think that it just comes with time and trying to figure out, you
like, hey, what works better, like here, a linear, a multiple
better than a neural net. But for example, with the one
never be able to see this curve. Okay. I mean, I'm not saying this
just saying like, hey, you know, sometimes it might be more
linear. So yeah, I will leave regression at that. Okay, so we just
And in supervised learning, we have data, we have some a bunch of
different samples. But each of those samples has some sort of
a category, a class, etc. Right, we were able to use that label in
right, we were able to use that label in order to try to predict
we haven't seen yet. Well, now let's move on to unsupervised
learning, we have a bunch of unlabeled data. And what can we do
anything from this data? So the first algorithm that we're going
clustering. What k means clustering is trying to do is it's trying
So in this example below, I have a bunch of scattered points. And
is x zero and x one on the two axes, which means I'm actually
right of each point, but we don't know what the y label is for
at these scattered points, we can kind of see how there are
right. So depending on what we pick for k, we might have different
right, then we might pick, okay, this seems like it could be one
another cluster. So those might be our two different clusters. If
for example, then okay, this seems like it could be a cluster.
cluster. And maybe this could be a cluster, right. So we could
data set. Now, this k here is predefined, if I can spell that
the model. So that would be you. All right. And let's discuss how
goes through and computes the k clusters. So I'm going to write
Now, the first step that happens is we actually choose well, the
points on this plot to be the centroids. And by centuries, I just
Okay. So three random points, let's say we're doing k equals
random points to be the centroids of the three clusters. If it
random points. Okay. So maybe the three random points I'm choosing
Here, here, and here. All right. So we have three different
the distance for each point to those centroids. So between all the
So basically, I'm saying, all right, this is this distance, this
all of these distances, I'm computing between oops, not those two,
centroids themselves. So I'm computing the distances for all of
Okay. And that comes with also assigning those points to the
What do I mean by that? So let's take this point here, for
this distance, this distance, and this distance. And I'm saying,
is the closest. So I'm actually going to put this into the red
all of these points, it seems slightly closer to red, and this one
right? Now for the blue, I actually wouldn't put any blue ones in
actually, that first one is closer to red. And now it seems like
closer to green. So let's just put all of these into green here,
have, you know, our two, three, technically centroid. So there's
this group here. And then blue is kind of just this group here, it
of the points yet. So the next step, three that we do is we
centroid. So we compute new centroids based on the points that we
And by that, I just mean, okay, well, let's take the average of
new centroid? That's probably going to be somewhere around here,
any points in there. So we won't touch and then the screen one, we
over here, oops, somewhere over here. Right. So now if I erase all
I can go and I can actually redo step two over here, this
Alright, so I'm going to go back and I'm going to iterate through
and I'm going to recompute my three centroids. So let's see, we're
these are definitely all red, right? This one still looks a bit
this part, we actually start getting closer to the blues.
So this one still seems closer to a blue than a green, this one as
would belong to green. Okay, so now our three centroids are three,
would be this, this, and then this, right? Those are our three
and we compute the new sorry, those would be the three clusters.
the three centroids. So I'm going to get rid of this, this and
red be centered, probably closer, you know, to this point here,
up here. And then this green would probably be somewhere. It's
before. But it seems like it'd be pulled down a bit. So probably
All right. And now, again, we go back and we compute the distance
and the centroids. And then we assign them to the closest
it's very clear. Actually, let me just circle that. And this it
it actually seemed like this point is closer to this blue now. So
be maybe this point looks like it'd be blue. So all these look
And the greens would probably be this cluster right here. So we go
bam. This one probably like almost here, bam. And then the green
here ish. Okay. And now we go back and we compute the we compute
So red, still this blue, I would argue is now this cluster here.
Okay, so we go and we recompute the centroids, bam, bam. And, you
to go and assign all the points to clusters again, I would get the
that's when we know that we can stop iterating between steps two
converged on some solution when we've reached some stable point.
these points are really changing out of their clusters anymore, we
and say, Hey, these are our three clusters. Okay. And this
expectation maximization. This part where we're assigning the
this is something this is our expectation step. And this part
centroids, this is our maximization step. Okay, so that's
And we use this in order to compute the centroids, assign all the
according to those centroids. And then we're recomputing all that
some stable point where nothing is changing anymore. Alright, so
of unsupervised learning. And basically, what this is doing is
some pattern in the data. So if I came up with another point, you
I can say, Oh, it looks like that's closer to if this is a, b, c,
cluster B. And so I would probably put it in cluster B. Okay, so
in the data based on just how, how the points are scattered
the second unsupervised learning technique that I'm going to
principal component analysis. And the point of principal component
used as a dimensionality reduction technique. So let me write that
reduction. And what do I mean by dimensionality reduction is if I
x1 x2 x3 x4, etc. Can I just reduce that down to one dimension
about how all these points are spread relative to one another. And
principal component analysis. Let's say I have some points in the
Okay, so these points might be spread, you know, something like
Okay. So for example, if this were something to do with housing
this here might be x zero might be hey, years since built, right,
and x one might be square footage of the house. Alright, so like
right now it's been, you know, 22 years since a house in 2000 was
analysis is just saying, alright, let's say we want to build a
you know, display something about our data, but we don't we don't
How do we display, you know, how do we how do we demonstrate that
this point than this point. And we can do that using principal
take what you know about linear regression and just forget about
you might get confused. PCA is a way of trying to find direction
variance. So this principal component, what that means is
So some direction in this space with the largest variance, okay,
data set without the two different dimensions. Like, let's say we
mentions, and somebody's telling us, hey, you only get one
What dimension do you want to show us? Okay, so let's say we want
what dimension like what do we do, we want to project our data
Alright, so that in this case might be a dimension that looks
this. And you might say, okay, we're not going to talk about
We don't have a y value. So linear regression, this would be why
have a label for that. Instead, what we're doing is we're taking
all of these take that's not very visible. But take this right
And what PCA is doing is saying, okay, map all of these points
So the transformed data set would be here.
This one's on the data sets are on the line. So we just put that
new one dimensional data set. Okay, it's not our prediction or
If somebody came to us said you only get one dimension, you only
each of these 2d points. What number would you give us? What
So this would be our new one dimensional data set. Okay, it's not
What number would you give me? This would be the number that we
this is where our points are the most spread out. Right? If I took
and let me actually duplicate this so I don't have to rewrite
Or so I don't have to erase and then redraw anything. Let me get
And I just got rid of a point there too. So let me draw that
Alright, so if this were my original data point, what if I had
the PCA dimension? Okay, well, I then would have points that let
color. So if I were to draw a right angle to this for every point,
like this. And so just intuitively looking at these two different
we can see that the points are squished a little bit closer
variance that's not the space with the largest variance. The thing
is that this will give us the most discrimination between all of
variance, the further spread out these points will likely be. Now,
dimension that we should project it on a different way to actually
dimension with the largest variance. It's actually it also happens
to be the dimension that decreases that minimizes the residuals.
we take the residual from that the XY residual, so in linear
we were looking only at this residual, the differences between the
y and y hat, it's not that here in principal component analysis,
from our current point in two dimensional space, and then it's
taking that dimension. And we're saying, alright, how much, you
between that projection residual, and we're trying to minimize
actually equates to this largest variance dimension, this
you can either look at it as minimizing, minimize, let me get rid
the projection residuals. So that's the stuff in orange.
Or to maximizing the variance between the points.
Okay. And we're not really going to talk about, you know, the
calculate out the principal components, or like what that
need to understand linear algebra for that, especially
I'm not going to cover in this class. But that's how you would
now, with this two dimensional data set here, sorry, this one
from a 2d data set, and we now boil it down to one dimension.
dimension, and we can do other things with it. Right, we can, like
then we can now show x versus y, rather than x zero and x one in
Now we can just say, oh, this is a principal component. And we're
the y. Or for example, if there were 100 different dimensions, and
them, well, you could go and you could find the top five PCA
more useful to you than 100 different feature vector values.
analysis. Again, we're taking, you know, certain data that's
some sort of estimation, like some guess about its structure from
wanted to take, you know, a 3d thing, so like a sphere, but we
on. Well, what's the best approximation that we can make? Oh, it's
the same thing. It's saying if we have something with all these
show all of them, how do we boil it down to just one dimension?
information from that multiple dimensions? And that is exactly
residuals, or you maximize the variance. And that is PCA. So we'll
Now, finally, let's move on to implementing the unsupervised
Here, again, I'm on the UCI machine learning repository. And I
you know, I have a bunch of kernels that belong to three different
comma, Rosa and Canadian. And the different features that we have
geometric parameters of those wheat kernels. So the area
width, asymmetry, and the length of the kernel groove. Okay, so
which is easy to work with. And what we're going to do is we're
or I guess we're going to try to cluster the different varieties
So let's get started. I have a colab notebook open again. Oh,
go to the data folder, download this. And so I'm going to go to
and let's get started. So the first thing to do is to import our
notebook. So I've done that here. Okay, and then we're going to
so pandas. And then I'm also going to import seedborn because I'm
specific class. Okay. Great. So now our columns that we have in
the perimeter, the compactness, the length, with asymmetry,
to call it groove. And then the class, right, the wheat kernels
I'm going to do that using pandas read CSV. And it's called seeds
that into a data frame. And the names are equal to the columns
do that? Oops, what did I call this seeds data set text? Alright,
data frame right now, you'll notice something funky. Okay. And
stuff under area. And these are all our numbers with some dash t.
haven't actually told pandas what the separator is, which we can
just a tab. So in order to ensure that like all whitespace gets
we can actually this is for like a space. So any spaces are going
separators. So if I run that, now our this, you know, this is a
So now let's actually go and like visualize this data. So what I'm
each of these against one another. So in this case, pretend that
class, right? Pretend that so this class here, I'm just going to
that like, hey, we can predict our classes using unsupervised
in unsupervised learning, we don't actually have access to the
plot these against one another and see what happens. So for some I
the columns minus one because the classes in the columns. And I'm
so take everything from I onwards, you know, so I like the next
So this will give us basically a grid of all the different like
going to be columns I our y label is going to be the columns j. So
And I'm going to use seaborne this time. And I'm going to say
to be our x label. Or y is going to be our y label. And our data
we're passing in. So what's interesting here is that we can say
like if I give this class, it's going to separate the three
hues. So now what we're doing is we're basically comparing the
and the compactness. But we're going to visualize, you know, what
ahead and I might have to show. So great. So basically, we can see
we get these three groups. The area compactness, we get these
kind of look honestly like somewhat similar. Right, so Wow, look
we have the compactness and the asymmetry. And it looks like
it just looks like they're blobs, right? Sure, maybe class three
one and two kind of look like they're on top of each other. Okay.
might look slightly better in terms of clustering. But let's go
clustering examples that we talked about, and try to implement
going to do is just straight up clustering. So what we learned
So from SK learn, I'm going to import k means. Okay. And just for
you know, any x and any y, I'm just going to say, hey, let's use
I mean, perimeter asymmetry could be a good one. So x could be
Okay. And for this, the x values, I'm going to just extract those
Alright, well, let's make a k means algorithm, or let's, you know,
and in this specific case, we know that the number of clusters is
I'm going to fit this against this x that I've just defined right
create this clusters, so one thing, one cool thing is I can
say k mean dot labels. And it'll give give me if I can type
predictions for all the clusters are. And our actual, oops, not
and we get the class, and the values from those, we can actually
like, you know, everything in general, most of the zeros that it's
And in general, the twos are the twos here. And then this third
to three. Now remember, these are separate classes. So the labels,
really matter. We can say a map zero to one map two to two and map
you know, our mapping would do fairly well. But we can actually
that, I'm going to create this cluster cluster data frame. So I'm
And I'm going to pass in a horizontally stacked array with x, so
the clusters that I have here, but I'm going to reshape them. So
Okay. And the columns, the labels for that are going to be x, y,
to go ahead and do that same seaborne scatter plot. Again, where x
the hue is again the class. And the data is now this cluster data
this here is my k means like, I guess classes.
So k means kind of looks like this. If I come down here and I
this is my original classes with respect to this specific x and y.
like it doesn't do too poorly. Yeah, there's I mean, the colors
For the most part, it gets information of the clusters, right. And
higher dimensions. So with the higher dimensions, if we make x
except for the last one, which is our class, we can do the exact
We can do the exact same thing. So here, and we can
predict this. But now, our columns are equal to our data frame
And then with this class, actually, so we can literally just say
And we can fit all of this. And now, if I want to plot the k means
Alright, so this was my that's my clustered and my original. So
get these on the same page. So yeah, I mean, pretty similar to
actually really cool is even something like, you know, if we
where they were like on top of each other? Okay, so compactness
Right. So if I come down here, and I say compactness and
this is what my scatterplot. So this is what you know, my k means
dimensions for compactness and asymmetry, if we just look at those
right? And we know that the original looks something like this.
alike? No. Okay, so now if I come back down here, and I rerun this
but actually, this clusters, I need to get the labels of the k
Okay, so if I rerun this with higher dimensions
well, if we zoom out, and we take a look at these two, sure, the
there are the three groups are there, right? This does a much
what group is what. So, for example, we could relabel the one in
And then we could make sorry, okay, this is kind of confusing. But
were projected onto this darker pink here, and then this dark one
and this light one was this dark one, then you kind of see like
right? Like even these two up here are the same class as all the
the same in the same color. So you don't want to compare the two
you want to compare which points are in what colors in each of the
application. So this is how k means functions, it's basically
All right, where are my clusters given these pieces of data? And
talked about is PCA. So PCA, we're reducing the dimension, but
you know, seven dimensions. I don't know if there are seven, I
mapping multiple dimensions into a lower dimension number. Right.
So from SK learn decomposition, I can import PCA and that will be
So if I do PCA component, so this is how many dimensions you want
And you know, for this exercise, let's do two. Okay, so now I'm
And my transformed x is going to be PCA dot fit transform, and the
And the same x that I had up here. Okay, so all the other all the
perimeter, compactness, length, width, asymmetry, groove. Okay. So
transformed it. So let's look at what the shape of x used to be.
I had 210 samples, each seven, seven features long, basically. And
is 210 samples, but only of length two, which means that I only
I'm plotting. And we can actually even take a look at, you know,
Okay, so now we see each each one is a two dimensional point
each sample is now a two dimensional point in our new in our new
So what's cool is I can actually scatter these
zero and transformed x. So I actually have to
take the columns here. And if I show that
basically, we've just taken this like seven dimensional thing, and
single or I guess to a two dimensional representation. So that's a
And actually, let's go ahead and do the same clustering exercise
the k means this PCA data frame, I can let's construct data frame
frame is going to be H stack. I'm going to take this transformed x
So actually, instead of clusters, I'm going to use k means dot
So it's 2d. So we can do the H stack. And for the columns, I'm
and the class. All right. So now if I take this, I can also do the
But instead of the k means labels, I want from the data frame the
And I'm just going to take the values from that. And so now I have
with PCA and then a data frame for the truth with also the PCA.
to how I plotted these up here. So let me actually take these
Instead of the cluster data frame, I want the this is the k means
to be class, but now x and y are going to be the two PCA
dimensions. And you can see that the data frame is going to be the
So these are my two PCA dimensions. And you can see that, you
out. And then here, I'm going to go to my truth classes. Again,
of k means this should be truth PCA data frame. So you can see
along these two dimensions, we actually are doing fairly well in
seem like this is slightly more separable than the other like
up here. So that's a good sign. And up here, you can see that hey,
another. I mean, for the most part, our algorithm or unsupervised
to give us is able to spit out, you know, what the proper labels
specific labels to the different types of kernels. But for
kernel kernels and same here. And then these might all be the
be the Canadian kernels. So it does struggle a little bit with,
But for the most part, our algorithm is able to find the three
fairly good job at predicting them without without any information
algorithm any labels. So that's a gist of unsupervised learning. I
this course. I hope you know, a lot of these examples made sense.
that I have done, and you know, you're somebody with more
in the comments and we can all as a community learn from this
Can't find what you're looking for?
Get subtitles in any language from opensubtitles.com, and translate them here.