Original subtitles

Let's take a look at how you can actually

implement the gradient descent algorithm.

Let me write down the gradient descent algorithm.

Here it is. On each step,

w, the parameter,

is updated to the old value of w minus

Alpha times this term d/dw

of the cos function J of wb.

What this expression is saying is,

after your parameter w by taking the current value of

w and adjusting it a small amount,

which is this expression on the right,

minus Alpha times this term over here.

If you feel like there's a lot

going on in this equation,

it's okay, don't worry about it.

We'll unpack it together.

First, this equal notation here.

Now, since I said we're assigning

w a value using this equal sign,

so in this context,

this equal sign is the assignment operator.

Specifically, in this context,

if you write code that says a equals c,

it means take the value c and store it in your computer,

in the variable a.

Or if you write a equals a plus 1,

it means set the value of a to be equal to a plus 1,

or increments the value of a by one.

The assignment operator encoding is

different than truth assertions in mathematics.

Where if I write a equals c,

I'm asserting, that is,

I'm claiming that the values

of a and c are equal to each other.

Hopefully, I will never write a truth assertion a equals

a plus 1 because that just can't possibly be true.

In Python and in other programming languages,

truth assertions are sometimes written as equals equals,

so you may see oh,

that says a equals equals c if you're

testing whether a is equal to c. But in math notation,

as we conventionally use it,

like in these videos,

the equal sign can be used for

either assignments or for truth assertion.

I try to make sure I was clear

when I write an equal sign,

whether we're assigning a value to a variable,

or whether we're asserting the truth of

the equality of two values.

Now, this dive more deeply

into what the symbols in this equation means.

The symbol here is the Greek alphabet Alpha.

In this equation, Alpha is also called the learning rate.

The learning rate is usually a small positive number

between 0 and 1 and it might be say, 0.01.

What Alpha does is,

it basically controls how

big of a step you take downhill.

If Alpha is very large,

then that corresponds to

a very aggressive gradient descent procedure

where you're trying to take huge steps downhill.

If Alpha is very small,

then you'd be taking small baby steps downhill.

We'll come back later to dive more deeply into

how to choose a good learning rate Alpha.

Finally, this term here,

that's the derivative term of the cost function J.

Let's not worry about the details

of this derivative right now.

But later on, you'll get to

see more about the derivative term.

But for now, you can think of

this derivative term that I drew

a magenta box around as telling you

in which direction you want to take your baby step.

In combination with the learning rate Alpha,

it also determines the size

of the steps you want to take downhill.

Now, I do want to mention that

derivatives come from calculus.

Even if you aren't familiar with

calculus, don't worry about it.

Even without knowing any calculus,

you'd be able to figure out all you need

to know about this derivative term

in this video and the next. One more thing.

Remember your model has two parameters,

not just w, but also b.

You also have an assignment operations

update the parameter b that looks very similar.

b is assigned the old value of b minus

the learning rate Alpha times

this slightly different derivative term,

d/db of J of wb.

Remember in the graph of the surface plot

where you're taking baby steps

until you get to the bottom of the value,

well, for the gradient descent algorithm,

you're going to repeat

these two update steps until the algorithm converges.

By converges, I mean that you

reach the point at a local minimum where

the parameters w and b no longer

change much with each additional step that you take.

Now, there's one more subtle detail

about how to correctly in semantic gradient descent,

you're going to update two parameters, w and b.

This update takes place for both parameters, w and b.

One important detail is that for gradient descent,

you want to simultaneously update w and b,

meaning you want to update

both parameters at the same time.

What I mean by that, is that in this expression,

you're going to update w from the old w to a new w,

and you're also updating b from

its oldest value to a new value of b.

The way to implement this is to compute the right side,

computing this thing for w and b,

and simultaneously at the same time,

update w and b to the new values.

Let's take a look at what this means.

Here's the correct way to implement

gradient descent which does a simultaneous update.

This sets a variable temp_w equal to that expression,

which is w minus that term here.

There's also a set in another variable temp_b to that,

which is b minus that term.

You compute both for hand sides, both updates,

and store them into variables temp_w and temp_b.

Then you copy the value of temp_w into w,

and you also copy the value of temp_b into b.

Now, one thing you may notice is that this value of

w is from the for w gets updated.

Here, I noticed that the pre-update w

is where it goes into the derivative term over here.

In contrast, here is

an incorrect implementation of

gradient descent that does not do a simultaneous update.

In this incorrect implementation,

we compute temp_w,

same as before, so far that's okay.

Now here's where things start to differ.

We then update w with the value in

temp_w before calculating the new value

for the other parameter to be.

Next, we calculate temp_b as b minus that term here,

and finally, we update b with the value in temp_b.

The difference between the right-hand side and

the left-hand side implementations

is that if you look over here,

this w has already been updated to this new value,

and this is updated w that actually

goes into the cost function j of w, b.

It means that this term here on the right is not the

same as this term over here that you see on the left.

That also means this temp_b term on

the right is not quite the

same as the temp b term on the left,

and thus this updated value for

b on the right is not the same

as this updated value for variable b on the left.

The way that gradient descent is implemented in code,

it actually turns out to be more natural to implement

it the correct way with simultaneous updates.

When you hear someone talk about gradient descent,

they always mean the gradient descents where you

perform a simultaneous update of the parameters.

If however, you were

to implement non-simultaneous update,

it turns out it will probably work more or less anyway.

But doing it this way isn't

really the correct way to implement it,

is actually some other algorithm

with different properties.

I would advise you to just stick to

the correct simultaneous update and

not use this incorrect version on the right.

That's gradient descent.

In the next video,

we'll go into details of

the derivative term which you saw in this video,

but that we didn't really talk about in detail.

Derivatives are part of calculus,

and again, if you're not familiar with

calculus, don't worry about it.

You don't need to know calculus at all in order to

complete this course or this specialization,

and you have all the information you need

in order to implement gradient descent.

Coming up in the next video,

we'll go over derivatives together,

and you come away with

the intuition and knowledge you need to

be able to implement and apply gradient descent yourself.

I think that'll be an exciting thing

for you to know how to implement.

Let's go on to the next video to see how to do that.