Original subtitles

1

Hello and welcome back to the course on artificial intelligence.

2

And finally we're on to the fun stuff we're onto deep learning.

3

All right so let's have a look.

4

Bruce we spoke about killer earning and what it's all about.

5

And we learned about Agent environment and how the agent will look at the state.

6

Or she is in take an action get a reward.

7

Enter into a new state and based on that feedback loop they will continue taking actions and they will

8

learn from that.

9

Understand what are the best actions to take.

10

And so we looked at this basic example of a maze.

11

We understood that as Asian explores environment understands what the values of the states are.

12

Then we moved on from dealing with the values of the states to dealing with the values of the actions

13

with the values and then A-Basin that we understood how plans in a non sarcastic environments work and

14

how policies work in stochastic environments and this is an example of a policy.

15

So that's a quick recap of everything we discussed in the basic learning.

16

And now let's have a look at how this can be taken to the next level through deep learning through adding

17

deep learning.

18

OK.

19

So this is our environment and what we're going to do now is we're going to add instead of just doing

20

basic calculations in this matrix that we have which is pretty simple.

21

What we're going to do is we're going to add two axes which adds an x and y axis or we'll call them

22

x1 and x2.

23

Just to make things even more general.

24

And here we've got the real number the row the columns 1 2 three 4 he'll rule number the rows 1 to 3.

25

And so now every single state can be described by a pair of two values x1 and x2 so any one of these

26

squares in which the agent can possibly be in can be described by x1 x2.

27

So for instance right now he's in the square with X1 equal to 1 and x 2 equal to 2.

28

And therefore that's not some way we can escape in your square meaning we can describe in your state.

29

Then of course this is a very simplified version of an environment of describing States.

30

But nevertheless it works in this case.

31

And that means that now we can feed this these states into a neural network.

32

And by the way here I would just like to mention that at the end of the course of good annexes we've

33

got an x number one and antics and two in order to proceed successfully with this section.

34

Highly advisable that you check out unaccessible one which is on artificial neural network so you understand

35

how they work so that we can we don't have to delve into that here and we can just use the benefits

36

of the knowledge of how artificial neural networks work and so we feed this information on the state

37

into a neural network and then it will process this information the X1 and x2 depending on the structure

38

of the neural network it might have multiple hidden layers and so on.

39

So that's something that you'll figure out in the practical tutorials but at the end we will structure

40

in such a way that it spits out for values and these four values are actually going to be our Q value.

41

So the values which dictate which action we need to take and the don't in this tutorial will see exactly

42

how these key values are used to decide which action is taken.

43

But the main point here is that we no longer look at just this maze from a learning perspective.

44

We're now taking the states of the maze and we're feeding them into a deep neural network in order to

45

get these cubicles and.

46

And at the end of the day we're still going to come up with an action we're still going to understand

47

what action we need to take and we'll discuss all this in more detail but the question right now is

48

why why are we doing all of this why we called it.

49

Why are making things so much more complicated when that initial approach of learning was working already

50

well.

51

The reason for that is the to learning was working in this very simplistic environment and we're continuing

52

to deal for now with this very simplistic environment in order to better understand the concepts.

53

But at the same time that simple Kial learning will no longer work in more complex environments and

54

we're talking about for instance the self-driving cars which will be creating or playing Doom when the

55

artificial intelligence is playing Doom or other Atari games like breakout or even self-driving cars

56

and more advanced reinforcement learning things such as like robots walking around and performing actions

57

in all those cases basically learning is insufficient is not strong is not powerful enough to be able

58

to master those challenges.

59

And just like we've seen in the deep learning course if you've been in our discipline or if you've done

60

the annex sections on x number one and X-2 you will where you know that deep learning is by far superior

61

to any type of machine learning let alone a simple cool learning.

62

And that's why we're leveraging the power of deep learning here so we're feeding in the information

63

about the environment as a vector of values.

64

In this case just to use into a deep neural network and then we're using that to perform the actions

65

that we want to to decide which actions are agents going to take.

66

So that's kind of like a high level overview of why we're doing this.

67

And now let's have a look at in a bit more detail what happens to the concept of cool learning when

68

we transfer when we make the transformation from or transition from simple learning into deep Killary.

69

So as you saw in the previous intuition tutorials we had a slide like this which is the foundation of

70

temporal difference learning.

71

This is the formula for temporal difference and basically So let's go through.

72

So basically we had an agent who was in this state over here which is indicated the blue arrow.

73

And we were understanding how temporal difference works for this value of for instance going up.

74

And so what we saw here was before this is in the simple Killary not the deep learning is in the simple

75

killer.

76

What we saw was before the agent had a subsequent hue value that he had learned about this action of

77

going up.

78

And so then he decided to take ception to go up.

79

And right after he takes his action he gets a reward for taking this action in this state.

80

And that is the reward plus now he can evaluate the value of the current state he's in which is the

81

maximum of all of the new q values of all of the cube of the new actions he can take a prime in the

82

new state as print and read multiplied by the DK factor of gamma.

83

So that is essentially the cue the new cube value or kind of like the the empirical cube value that

84

he has just received for taking that action.

85

And ideally these two two should be the same.

86

So the actually the Q value that he had in his memory about this action in this state should equate

87

to the actual reward Plus the gamma times the value of the state that he ended up in.

88

And therefore that's how we calculate the temporal difference we take what you are after minus what

89

he got what he had in mind what he was expecting.

90

You'd subtract one from the other.

91

That's a temporal difference.

92

And then you use your learning rate Alpha to adjust your q value your your new q value by the temporal

93

difference but with a coefficient of Alpha.

94

So that is the essence of the simple learning.

95

Now let's have a look at how it changes in deep Killary and so we still going to work with the slide

96

but we're going to just see exactly what's happening.

97

So in a deep learning the neural network will predict for Valis as we saw in the previous and as we'll

98

see.

99

Donna Citronelle the neural network will predict for values or it might predict more values of more

100

possible actions in a given state.

101

But in this case we know that there's only four actions upright left to done.

102

And so the neural network will predict four of these values so there will be no end in a deep learning

103

situation is important is that there is no before or after.

104

And this is how we'll we'll get to know this a bit better.

105

So the neural network will predict four of these values and it will compare not to what will happen

106

after but the neural network will compare to this exact value but it was the value which was calculated

107

in the previous step.

108

So in the previous time when the agent was in this exact square.

109

So let's say I don't know some time ago the agent was again was in this exact square as well and it's

110

calculated this value previously.

111

So in the previous time a long time ago the agent calculated this value then the agents stored this

112

value for the future and now the future has come.

113

So now he's in the square again and now he's got these cube values which is predicted and one of them

114

is for the four going up.

115

So now what he's going to do is going to compare the predicted value of Q to this value which he had

116

recorded from the previous step and will understand exactly why this is important right now so just

117

important understand here.

118

There's no before an officer in this specific square this specific time.

119

We're taking the Q value that he's predicted using the neural network this time and we comparing it

120

to this value which he had from the previous time from the previous time he was in this square assessing

121

all of the situation and you know like the previous time he actually performed this action.

122

So there we go.

123

Now let's have a look at how this all works out in the neural network and why.

124

Why is it like I know it sounds a bit complicated right now but we'll break it down into simple terms

125

just in a second.

126

So this on your own network we're feeding in the states of the environment into the neural network is

127

going through the hidden layers that it's coming out with these outputs Q1 Q2 Q3 Q4.

128

In that specific state these are the cube values that the neural network is predicting for possible

129

actions.

130

Those are the cumulous.

131

So then we're appearing to target and these targets exist exactly so if we go back here this is the

132

target so this is the value that was predicted.

133

And then but also we know that we have a target from the last time we were in the square.

134

We have a target for this same action which is up for instance.

135

So here we've got a target and we're going to compare we're comparing Q1 versus that target we're comparing

136

Q2 versus that target the target that we had from previously Q3 versus a target Q4 versus the target.

137

And so this is the part where the neural network or the agent is now learning through deep learning

138

how to better go through.

139

And the key point here is that we're still applying cool learning but the concepts answer is simple

140

you learn you learn through temporal differences which are pretty straightforward which we've already

141

discussed and we know quite well why not.

142

But at the same time in deep learning how do neural networks learn neural networks learn through our

143

adjusting the weights.

144

So we have to adapt the concepts of reinforcement the concepts of simple kill learning to the way neural

145

networks actually work.

146

And that is through updating their weights.

147

And so this is what we're trying to figure out here how do we adapt that concept of temporal difference

148

to your own network so that we can leverage the full power of neural networks.

149

So far we've gotten this so we enter our environment state here as a vector goes through a neural network

150

we get predictions of key values and then from the previous time the agent was in that state.

151

We have these new target to target one two three and four for each of these respective actions.

152

And so now we're up to.

153

OK let's compare each one with each one.

154

And from here it's it becomes pretty straightforward if you're up to speed with neural networks.

155

Once again that's on a Anax.

156

Number one we're going to calculate a loss which is here and we're going to be q target this one minus

157

Q minus this one.

158

We're going to square that so the square difference of the each one of these and we're going to sum

159

them.

160

So we take the sum of the squared differences of these values and their targets and we're going to send

161

them up and that's going to be a loss.

162

And so ideally just as we had into in the temporal difference learning so if we go back for a second

163

remember we said Ideally we want this to be equal to this so we want the temporal difference to be zero

164

so that's that means basically the agent is predicting exactly correctly what you know the Q value is

165

that the agent is predicting are exactly or that he has and memory are exactly descriptive of the environment

166

and therefore the agent can never get the environed pretty well right.

167

There's no surprises there's no there's no s.a as long as a temporal difference is a pilot highly positive

168

or highly negative.

169

Then we've got some surprises.

170

But if the general differences zero then he knows environment so well that he can predict what's going

171

on and he can and therefore his policy is going to be very good and he's going to be able to navigate.

172

So here.

173

Same thing so we want this law to be as close to zero I suppose as smallest possible.

174

And that's why now we're going to this is the part where we are going to leverage the real true power

175

of neural network so we're going to take this loss and we're going to use back propagation or stick

176

as the gradient descent to take this loss and pass it through the network posit back or back propagated

177

through a network and through to cast a great and decent a date the weights.

178

All of these synopses in the network so that next time we go through this network the way it's already

179

a bit better descriptive of the environment and that's exactly what we're.

180

So here you have if you go back this is calculated losses Kalka and guess prove propagator for the network

181

the weights are updated.

182

Then the next time we get here this happens again and again here this happens again and so on and so

183

and it keeps happening and that's how this agent learns or basically now the neural network which is

184

the brain of the agent is learning is becoming more and more descriptive of the environment and therefore

185

the agent is able to navigate the environment.

186

When we say descriptive environment basically means that when we put in the states of the environment

187

that this agent is in we are more likely to get closer and closer to the actual cue values and that

188

happens because the cube values that we want to find the right action and that happens because these

189

new targets are actually empirically derived so he every day how does he find these cute targets.

190

That's actually there this so he actually observes.

191

OK so once I do take this step what's the reward I get.

192

And then what's the values of this state.

193

So same thing as we saw previously in Q learning and the simple learning intuition.

194

So he learns this through trial and error and then he constructs his network or that's the way it is

195

in such a way that the predicted values are close and close.

196

Consummating that target.

197

Q values so very similar to the concept we discussed here in the simple temporal difference learning

198

of the simple skill learning algorithm.

199

So there you go.

200

That's that's how the agent learns.

201

So we're up to here.

202

And that's the learning part.