All language subtitles for 03_gradient-descent-intuition.en

Now let's dive more deeply
in gradient descent to gain

better intuition about what

it's doing and why
it might make sense.

Here's the gradient
descent algorithm

that you saw in the
previous video.

As a reminder, this variable,

this Greek symbol Alpha,
is the learning rate.

The learning rate controls
how big of a step you take

when updating the model's
parameters, w and b.

This term here, this d over dw,

this is a derivative term.

By convention in math,

this d is written with
this funny font here.

In case anyone watching
this has PhD in

math or is an expert in
multivariate calculus,

they may be wondering,
that's not the derivative,

that's the partial derivative.
Yes, they be right.

But for the purposes of

implementing a machine
learning algorithm,

I'm just going to
call it derivative.

Don't worry about these
little distinctions.

What we're going to focus on now

is get more intuition about

what this learning rate
and what this derivative

are doing and why when
multiplied together like this,

it results in updates
to parameters w and b.

That makes sense. In order
to do this let's use

a slightly simpler
example where we

work on minimizing
just one parameter.

Let's say that you have
a cost function J of

just one parameter w
with w is a number.

This means the gradient
descent now looks like this.

W is updated to w minus
the learning rate Alpha

times d over dw
of J of w. You're

trying to minimize the
cost by adjusting the

parameter w. This is like

our previous example where
we had temporarily set b

equal to 0 with one
parameter w instead of two,

you can look at
two-dimensional graphs

of the cost function j,

instead of three
dimensional graphs.

Let's look at what

gradient descent does
on just function J of

w. Here on the horizontal
axis is parameter w,

and on the vertical axis is
the cost j of w. Now less

initialized gradient descent
with some starting value for

w. Let's initialize
it at this location.

Imagine that you start off at

this point right here
on the function J,

what gradient descent
will do is it will update

w to be w minus learning rate

Alpha times d over dw of J of

w. Let's look at what this
derivative term here means.

A way to think about the
derivative at this point on

the line is to draw
a tangent line,

which is a straight line that

touches this curve
at that point.

Enough, the slope
of this line is

the derivative of the
function j at this point.

To get the slope, you can

draw a little
triangle like this.

If you compute the
height divided by

the width of this triangle,
that is the slope.

For example, this slope
might be 2 over 1,

for instance and when

the tangent line is pointing
up and to the right,

the slope is positive,

which means that this derivative
is a positive number,

so is greater than 0.

The updated w is going to be

w minus the learning rate
times some positive number.

The learning rate is
always a positive number.

If you take w minus
a positive number,

you end up with a new value
for w, that's smaller.

On the graph, you're
moving to the left,

you're decreasing the
value of w. You may notice

that this is the right
thing to do if your goal

is to decrease the cost J,

because when we move towards
the left on this curve,

the cost j decreases,

and you're getting
closer to the minimum

for J, which is over here.

So far, gradient descent,

seems to be doing
the right thing.

Now, let's look at
another example.

Let's take the same
function j of w as above,

and now let's say
that you initialized

gradient descent at a
different location.

Say by choosing a
starting value for

w that's over here on the left.

That's this point
of the function j.

Now, the derivative term,

remember is d over dw of J of w,

and when we look at
the tangent line

at this point over here,

the slope of this line is

a derivative of J at this point.

But this tangent line is
sloping down into the right.

This lines sloping down into

the right has a negative slope.

In other words, the
derivative of J at

this point is a negative number.

For instance, if you
draw a triangle,

then the height like this is

negative 2 and the width is 1,

the slope is negative
2 divided by 1,

which is negative 2,

which is a negative number.

When you update w,

you get w minus the
learning rate times

a negative number.

This means you subtract
from w, a negative number.

But subtracting a
negative number

means adding a positive number,

and so you end up increasing

w. Because subtracting a
negative number is the

same as adding a
positive number to

w. This step of gradient
descent causes w to increase,

which means you're moving to
the right of the graph and

your cost J has
decrease down to here.

Again, it looks like

gradient descent is doing
something reasonable,

is getting you closer
to the minimum.

Hopefully, these last
two examples show

some of the intuition behind
what a derivative term

is doing and why this host
gradient descent change

w to get you closer
to the minimum.

I hope this video gave
you some sense for why

the derivative term in
gradient descent makes sense.

One other key quantity in

the gradient descent algorithm

is the learning rate Alpha.

How do you choose Alpha?

What happens if it's too

small or what happens
when it's too big?

In the next video,

let's take a deeper look at

the parameter Alpha to help

build intuitions
about what it does,

as well as how to make a
good choice for a good value

of Alpha for your implementation
of gradient descent.