All language subtitles for 10. Cost Functions and Gradient Descent

Welcome back everyone.

In this lecture we're going to be talking about cost functions which is going to allow us to measure

how far off we are in the output predictions of our neural network.

And then we'll talk about gradient descent which is going to help us minimize that cost or minimize

that error.

So often cost functions you also hear them label as lost functions or error functions.

So we don't talk about all that really interesting topic here.

And it really fundamentally is gonna help us understand better how the neural network will actually

learn so we already understand that neural networks they taken inputs in that first layer then they

multiply them by weights and then add biases to them and then maybe that gets passed on through an activation

function like the sigmoid function or the rectified linear unit and then that ends up going into another

layer and then you add another set of weights and biases and then so on and so on all the way to that

last output layer so that last output layer we can maybe call that or refer to it as y hat.

That's essentially the model's estimation of what it predicts the label to be.

And so we have two main questions after the network creates that prediction.

How do we actually evaluate it against the true label.

And then after the evaluation how can we update the networks weights and biases.

So really we're going to focus on this lecture is how do we evaluate how far off our prediction is for

updating the networks weights and biases.

We'll have to learn about back propagation which is coming up in a future lecture.

So right now let's focus on that first question so what we need to do is we need to take the estimated

outputs of the network and then compare them to the real values of the label and keep in mind what I'm

referring to right now is happening during the training or fitting portion of the supervised learning

process.

So right now what we're doing is we're using just the training data set that way we can go back and

update our weights and biases during the test set.

We're not really updating weights and biases.

Instead we're just again evaluating overall on the entire dataset.

How does our neural network perform.

Right now we're doing these small evaluations on these training batches in order to actually then go

back and update weights and biases in our network.

So keep that in mind OK.

So in order to actually compare our neural networks output to the true value we're gonna be using what's

known as a cost function.

And this is also often referred to as a lost function or error function.

Essentially it's just something that measures how far off you are from the true value based off your

prediction.

And one important caveat is it should be an average that we can output a single value then you can keep

track of that loss or cost during training to monitor network performance.

So hopefully during each epoch of training your loss or cost goes down down down until you kind of like

converge to some minimum cost value so we're going to here is I want to introduce a couple of variables.

We're gonna be using Y to represent the true value and then we'll be using a to represent the neurons

prediction.

So in terms of weights and biases what we have here is recall we set Z to be a representation of weights

times x plus B and then we pass that Z into an activation function such as the sigmoid function so that

Z passed into the sigmoid.

Is that equal to a.

So all I'm trying to say here is that a represents kind of that final output of a neuron which takes

into account what the activation function was.

And that also then takes into account Z which in turn takes to account w which are the weights and the

biases.

So a just keep that in mind holds a lot of information.

It holds information about the activation function the weights and the biases so probably the most common

cost function you'll see is known as the quadratic cost function.

And if you've done any machine learning this probably looks really familiar to you because it looks

like root mean square error which is kind of essentially what it is.

Here is just not notated for multidimensional data so all we're doing here is that the other day we're

calculating the difference between the real values here we label it y of x.

So if you add some X input we specify Y is the true function so that would be the true value.

And then we're subtracting our predicted values.

So here we have a level of X.

And keep in mind that a of L that notation just signifies that that is the activation function.

Output of the L layer where L is your last layer which means the layer before that in the network is

L minus 1.

Then the layer before that is L minus 2 and so on.

Later on we'll see why it's more convenient to kind of Mark L as your last layer and then work backwards

from there instead of starting from the beginning.

So again AFL of X that's essentially your predicted output so keep in mind the notation again shown

here kind of corresponds to vector inputs and outputs.

Since we're really dealing with a batch of training points and predictions as we go along but the main

idea is you have C as the cost function you're doing some sort of averaging so that's we have one over

two times and so n is the number of points there.

Then you're taking the sum of all those differences and squaring them so really come question is why

are we actually squaring this into this two useful things for us.

1 It keeps everything positive because well you have a positive air or negative error.

If you square it it becomes positive which is good because we want some sort of absolute measurement

of error.

If we weren't squaring this and we had stuff that was negative and positive when you average it out

that could hover around zero which is actually not a true indication of the absolute value or absolute

units of how far off you are.

So squaring it make sure that everything's positive the other and much more important thing it does

is it's gonna punish really large errors.

So sometimes you have some data points that you're gonna be really off on and if you actually square

that error then it exponentially grows as terms of your cost.

So maybe you're off by ten dollars in whatever unit you're trying to measure but your cost is going

to report that and unit squared.

So it's going to say you're off by one hundred instead of just ten.

You're going to really punish your network for being really off on certain points which is good because

you don't want your network to suddenly be able to not predict well on even just a few points where

it gives you a huge error.

You'd rather suffer a little bit on all the other points and not be totally off on those few kind of

edge cases.

So that really helps punish large errors now in general we can think of the cost function as a function

of four main things so the cost function is going to be a function of W which is our neural networks

weights B which is all the biases in our neural network SFR which is the input of a single training

sample and each of our which is the desired output of that training sample.

And that makes a lot of sense because the cost is dependent on what the current weights and biases are.

It's also dependent on what you passed in as the actual training example.

And then it's also dependent on what you're comparing it to which is e of our so notice how that information

was actually all encoded in that simplified notation that we had.

So I showed you this as the cost function and you may be wondering didn't you just say that the cost

function is a function of W and B was W and B in the quadratic function here.

Well it's actually encoded within a of X because remember a of X holds information about weights and

biases because of X is Z passed into the activation function where z then contains information about

W and B Ok so this means that if we have a huge network we can expect the actual cost function to be

really quite complex with a huge vector or tensor of weights and another huge tensor of biases.

So for example if we were just to take a small network and start labeling every way and every bias and

every output as kind of the output of the activation function which is a you can see all the parameters

labeled here you get really complicated really fast and this is a really small network.

There is only kind of four layers here two hidden layers and those hidden layers aren't even that big.

We can see here if we start noticing everything it can become quite complex.

You already have quite a large matrix of weights and biases so how do we actually calculate this.

How do we calculate that cost function and then figure out how to minimize it so in a real case this

means that we have some cost function C dependent on lots of weights that cost function is going to

dependent on the weight of that first input than weight.

Second one weight on third one all the way to W M and we do is when you figure out which particular

weights lead us to the lowest cost because we want to go back here and figure out for all these weights

how do we change them to minimize my cost function.

At the very end so for simplicity and just thinking about this let's imagine that we're dealing with

a really simple network that only has a single weight essentially just one year on.

So what we want to do is we want to minimize our loss or cost essentially our overall error which again

that means we need to figure out what value of W do we use that's going to result in the minimum of

C of value so here is our plot it out really simple cost function where it's a really simple network

it only contains one weight well what we want to do is we want to figure out what value of W minimizes

this cost function and while this is a really simple example you can probably just tell that in order

to minimize the cost function here you can see that the minimum probably fall somewhere where that arrow

is so that's the weight that is going to minimize that cost function.

Which means that's probably the weight we want in the actual neuron or that input to the neuron.

Because that reduces the cost to its minimum.

Now students of calculus know what we could do is we could just take the derivative of this cost function

and then solve for zero.

But recall our real cost function is gonna be super complex and it's not going to be one dimensional

two dimensional and three dimensional.

If you take a look back at that network it's gonna be as many dimensions as there are W..

And that's not even something I can actually plot so again it's going to be n dimensional which means

taking that derivative and setting it equal to zero is actually not going to be you're not gonna be

able to calculate that without spending kind of a thousand years of computational time so our networks

especially when we build out really large networks are gonna have thousands of weights to them hundreds

of weights.

We're not gonna build take that derivative.

So instead what we do is a stochastic process to what we can do is we can use gradient descent to solve

this sort of problem so let's again go back to this kind of simplified version of our network we just

have one wait and see how gradient descent would work on this simple example.

And then we can easily expand it to more complex examples.

So what we do is we just start off at one point on this cost function and then again what we're searching

for here is that w value that minimizes this cost function so what we do is we calculate the slope at

one point and then we move in the downward direction of the slope and you keep repeating this process

until eventually you're going to converge to zero indicating a minimum.

So what we could have done is keep in mind we could've changed our step size to find the next point

so here we kind of took equal step sizes and if you take smaller step sizes it takes longer to find

the minimum if you take larger steps sizes you'll go faster.

But what happens is you risk overshooting the minimum.

So if you go too large of a step size you may actually miss that minimum weight or a minimum weight

tensor and then kind of overshoot and you don't end up converging.

So that's step size is known as the learning rate.

So if you ever see in your own networks that they're editing the learning rate what they're really doing

here is they're editing how fast they're going to try to find that minimum weight value and it works

the same for biases you're finding those minimum weights really the values of the weights and biases

that minimize that cost function.

Now in those previous examples something I should know is that the learning rate was constant.

That is to say each step size was equal.

So regardless of which one we're actually looking at such as the smaller step sizes or the larger step

sizes the actual step is equal for all of these.

Now we can be actually be a little clever and adapt our step size as we go along.

You can imagine that since you're starting off kind of an around the position in this and dimensional

space of possible weights and biases if you start with larger steps well you can do is you can then

go smaller and smaller in your step size as that gradient or that slope gets closer to zero.

And this is known as Adaptive gradient descent depending on that gradient you get back you're going

to adapt your step size so in 2015 Kingman and Bob published a paper called Adam a method first stochastic

optimization.

And Adam is a much more efficient way of searching for these minimums.

So you're going to see us actually kind of sometimes state Adam as our optimizer during the code.

So keep that in mind if you ever see Adam all we're really referring to is this optimized way of performing

this gradient descent where we have kind of this adaptive step side.

So we kind of start off large and then depending on where we are maybe go smaller and smaller.

So you kind of get the best of both worlds.

You get to use the larger step sizes and kind of speed up finding that minimum.

But then as you get closer and closer to it and you don't want to overshoot you can go a smaller step

sizes so you can actually then compare Adam versus other gradient descent algorithms.

And here it's showing you the training cost versus iterations over the entire dataset and you can see

Adam here is outperforming these other adaptive gradient descent algorithms.

So all the ones listed here that are not Adam they're also actually adaptive gradient descents however

Adam performs better than all these.

So we're going to be doing is we'll be using Adam since it's really quite common to use it for neural

networks.

So if you see that all we're doing here is we're kind of saying OK optimize the way we find this minimum

now realistically we're showing you that illustration of gradient descent on just a single W..

So that was kind of a one dimensional W. or really doing is we're calculating gradient descent in an

end dimensional space for all our weights.

Here you can see calculating gradient descent in a two dimensional plane including the weights and the

bias.

Realistically I'm not even gonna be able to illustrate the end dimensional space because it's gonna

be end dimensions of tens or hundreds of weights and biases.

So there's really no way we can illustrate that for you.

Which is why we simplify it down to kind of a single way you can understand gradient descent is doing.

And then what we're actually doing or what the computer will be doing for us is doing the same sort

of calculations that on an n dimensional space that we can't really realistically illustrate for you.

So when dealing with these n dimensional vectors otherwise known as sensors the notation changes from

derivative to gradient.

So that's why you've heard me say a gradient a couple of times instead of the term derivative.

So when we're dealing with n dimensions instead of saying the derivative that actually the correct term

becomes gradient.

And so that means we calculate the gradient of the cost function with respect to all these weights.

So if you see that upside that upside down triangle that's essentially kind of the way you notate gradient

instead of just saying derivative.

Now before we finish up and wrap up this lecture on loss or cost functions and gradient descent on a

quickly I mentioned that for classification problems instead of using the quadratic cost function we

end up often using is the cross entropy loss function.

And what's nice about this cross entropy loss function is that basically what it does it assumes that

your model predicts a probability distribution for each class.

So maybe it has a distribution for class one class two plus three and so on.

And the way this formula actually works out is for a binary classification that is only two classes.

You have the top formula resulting and then those logs actually represent natural logs and then for

any number of classes greater than two you are using the formula below.

And the worry too much about this formula because essentially more coding it out we're just going to

specify use cross entropy.

So when we're performing classification especially multi class classification that's greater than just

binary classification.

We'll be calling upon cross entropy to be our cost function.

Okay.

So just keep that in mind if you ever see us coding and we specify cross entropy.

These are the actual formulas we're specifying the computer to use in order to figure out a probability

distribution for each of those classes so as a quick review we talked about cost functions.

We talked about gradient descent.

We talked about the fact that there's different optimizer is like the atom optimizer and then we also

talked about quadratic cost and cross entropy so far we understand the networks take an input affect

that input of weights and biases and activation functions to produce an estimated output.

Then we learned how to evaluate that output against the true labels.

The last thing you need to do in our theory discussions is the following question.

Once we actually get that cost or loss value we understand how far off we are.

We still haven't really talked about how we go back and adjust all those weights and biases.

Still kind of this magical thing and that magical thing and just the second will no longer be so magical.

So it'll be mathematical and it's called Back propagation.

You essentially propagate backwards through your network and then update all those weights and biases.

So that's exactly we're going to cover in the next lecture.

I'll see you there.