Original subtitles

1

Hello and welcome back so of course on deep learning today we talk about the Kostic gradient descent.

2

Previously we learned about gradient descent and we found out that it is a very efficient method to

3

solve our optimization problem where we're trying to minimize the cost function.

4

It basically takes us from 10 to the power of 57 years to solving a problem within minutes or hours

5

or within a day or so.

6

And it really helps speed things up because we can see which way is downhill and we can just go in that

7

direction and take steps and get to the minimum faster.

8

But the thing with the stick with gradient descent is that this method requires for the cost function

9

to be convex.

10

And as you can see here we've specifically chosen a convex cost function basically convex means that

11

the function looks similar to what we are seeing now that it's just kind of vext into one direction

12

and that in essence has one global minimum.

13

And that's the one that we're going to find.

14

But what if our function is not convex.

15

What if our cost function is not correct.

16

What if it looks something like this.

17

Well first of all how could that happen.

18

Well that could happen because if we first of all choose a cost function which is not the square difference

19

between why how and why or if we do choose the cost function which is like that.

20

But then in a multi dimensional space it can actually turn into something that is not convex.

21

And so what would happen in this case if we just tried to apply our normal gradient decent method something

22

like this could happen.

23

We could find a local minimum of the cost function rather than the global one.

24

So this one was the best one and we found the wrong one and therefore we don't have the correct weight.

25

We don't have an optimized neural network.

26

We have a subpar neural network.

27

And so what do we do in this case.

28

Well the answer here is stochastic.

29

Gradient descent.

30

And it turns out the sarcastic gradient descent doesn't require for the cause function to be convex.

31

So let's have a look at the two differences between the normal gradient descent that we talked about

32

and the stochastic range.

33

So normal green descent is when we take all of our rows we plug them into our neural network and once

34

again here we've got the neural network copied over several times but the rows are being plugged into

35

that same neural network every time.

36

So there's only one year old trick this is just for Kissel's action purposes.

37

And then once we plug them in we've calculated our cost function based on the formula right and looking

38

at the chart on the at the bottom and then we adjust the weights then this is called the gradient descent

39

method or it's also the proper term is that batch gradient descent method.

40

So we take the whole batch of from our sample we apply it and then we run that the stochastic gradient

41

descent method is a bit different.

42

Here we take the rows one by one so we take this row we run our neural network and then we adjust the

43

weights.

44

Then we move onto the second row we take the second row we run our neural network.

45

We look at the cost function and then we adjust the weights again and then we take another Rohtak rose

46

three we run our neural network will look at the cost function we adjust the weight.

47

So basically we're looking at we're adjusting the weights after every single row rather than doing everything

48

together and then testing weights two different approaches.

49

And now we're going to just compare the two side by side.

50

So here they are this is how to visually remember them.

51

So you've got the best gradient descent where you are adjusting the weights after you've run them after

52

you've run all of the rows in your neural network and then basically just the weights and you run the

53

whole thing again iteration iteration iteration in the sixth grade in December and you run one row at

54

a time and you adjust the weights just the way it's just the weights and then you do everything again

55

and again and that is called discussing.

56

And you said that the main two differences are that the sarcastic gradient descent method helps you

57

avoid the problem where you find those local extremities or local minimums rather than the overall overall

58

global minimum.

59

And the reason for that in simple terms is that there is video of the stochastic gradient descent method

60

has much higher fluctuations because it can afford them.

61

It's doing one iteration or one row at a time and therefore the fluctuations are much higher and it

62

is much more likely to find the global minimum rather than just the local minimum.

63

And the other thing about the sarcastic gradient descent I think is a bad gradient is the it's foster

64

like the first impression that you might have is because it's doing grow one at a time it is slower

65

but actually in fact it is faster because it is it doesn't have to load up all the data into memory

66

and run and wait until all of those rules are on altogether.

67

You can just roll around them one by one so it's a much lighter algorithm is much faster in that sense

68

so though it has way more in that sense as it has more advantages over the bad.

69

Gradient descent method.

70

The main advantage of or domain kind of like profer the bad gradient descent method is that it is a

71

deterministic algorithm or other than to cast a gradient descent being a sarcastic algorithm meaning

72

it's random and with the best gradient and method as long as you have the same starting weights for

73

your neural network.

74

Every time you run the batch gradient descent method you will get the same iterations the same results

75

for you all the way your weights are being updated for us to have for the sarcastic gradient decent

76

method.

77

You won't get that because it is a stochastic method you're picking your roles possibly at random and

78

you are updating your neural network in a sarcastic manner and therefore you're just going to every

79

single time you run the category a decent method even if you have the same weights at the start you're

80

going to have a different process and different iterations to get there.

81

So that's in a nutshell what's to castigate and dissent is also there's a method in-between the two

82

called the Mini batch gradient descent method where you combine the two and you basically run rather

83

than running a whole batch of running one at a time.

84

You run batches of rows maybe 5 10 100 however many rows you decide to set you run those that number

85

of rows at a time then you update your way single digits and so on.

86

And that's called the Mini Bache gradient descent method if you'd like to learn more about gradient

87

descent there's a great article which you can have a look at.

88

It's called a neural network in 13 lines of Python part to great and descend by Andrew Trask and the

89

links below it's an good 12 15 article very well-written very very simple terms.

90

It's got some interesting philosophical or just interesting thoughts on how to apply green decent water

91

you know advantages and disadvantages and how to be how to do things in certain situations so you got

92

some very cool tips tricks and hacks.

93

Very easy read so definitely check that out.

94

And another one a bit more heavier read.

95

For those of you who are into mathematics who want to get to the bottom of the mathematics why.

96

Gradient descent is that specific.

97

What are the formulas that are driving gradings And how is it calculate and so on.

98

Check out the article or actually the book.

99

It's a free online book called neural networks and deep learning by Michael Nielsen 2015 book.

100

It's just basically it's all on line you can go ahead and check it out there.

101

And there again very soft introduction to the mathematics.

102

But then for a mother the math but the mathematics are pretty heavy as you go along as you read through

103

the article.

104

But at the same time it gets you into into that mood I think you mean has like a warm up chapter where

105

you first warm up the math and then you jump into I'm so interested in math then this is the article

106

to go to.

107

And there we go so that's in a nutshell the difference between Graney sense to cast the gradient descent

108

and how to work.

109

And on that note we're going to wrap up today said Tauriel.

110

I look forward to seeing you on the next one.

111

And until then enjoy deep learning.