Original subtitles

Welcome back everyone to this lecture on neural networks.

So we just learned about a single percent Trump however a single perception model won't be enough to

learn complicated systems.

Fortunately we can expand on the idea of a single perception to create a multilayer perception model

commonly known as a basic artificial neural network.

We'll also introduce the idea of activation functions which will cover in a future lecture.

So how do we actually build out a neural network from the idea of a single perception.

Well it's actually quite simple.

What we do is we simply build a network of perceptions reconnect layers of perceptions using what's

known as the multilayer perception model.

Essentially we have a vertical layer of these neurons.

The single perceptions and we take their outputs and then feed them to the next layer of perceptions

so that the output of the previous layer becomes the input of the next layer.

And you'll notice here that every neuron at least in this illustration is connected to every neuron

in the next layer.

And this is known as a fully connected layer.

We'll learn about different types of layers and different network configurations like recurrent neural

networks and convolution neural networks in future section of the course.

But right now we're just focused on a feed forward that is to say that all the information is going

from the input layer all the way to the end the output layer and it's fully connected which means every

neuron in one layer is connected to every neuron in the next layer so the outputs of one perception

are then directly fed as inputs to another perception.

And you'll notice I've begun to kind of switch the words perception to neurons and it's up to you which

one you prefer.

But one word in general discussing neural networks we'll start to lay off the single perceptual on terminology

and then use the more flexible neuron terminology.

So from now on you'll probably hear me say neuron a lot more than single perceptual.

So basically this allows the network as a whole to learn about interactions and relationships between

features so the first layer coming in is the input layer and the layer that directly receives the data.

And this could be things like tabular data that has features on information that you're trying to predict

a label off of the last layer is known as the output layer.

And keep in mind even though this illustration just shows one neuron in the output layer this last layer

can be more than one neuron especially when we're dealing with things like multi class classification

then any layers in between the input layer and the output layer are known as hidden layers hidden layers

are difficult to interpret due to their high interconnectivity and distance away from known input and

output values so the input layer that's really easy to interpret because that's directly accepting the

inputs of the raw data that you already know and understand the output layer is also a little easier

to interpret since it's closely associated with the label that you're actually trying to predict.

The hidden layers especially as you have a deeper and deeper network with more and more hidden layers.

Those become more and more like a black box since for really large networks.

It's difficult to understand why a single neuron is picking up as far as the interconnectivity of the

last layer and the next layer so questions students often ask is when there's a neural network become

a deep neural network and really the terminology is quite simple.

You have a deep neural network when you contain two or more hidden layers so here we can see on the

left hand side.

This network only has a single hidden layer.

You'll notice that the hidden layer is quite wide meaning it has quite a few neurons so the width of

a network is how many neurons are in the layer and the depth of a network is how many layers there are

in total.

So a deep neural network is two or more hidden layers.

So again to overview the terminology we have the input layer first layer that directly accepts real

data values hidden layer any layer in between those input output layers and if we have two or more we

begin to have what's known as a deep neural network and then the output layer is the final estimate

of the output that the network estimates.

Now something that I find really incredible about the neural network framework is that it can actually

be used to approximate any continuous function and researchers Lou and then also later on Boris Hannan

proved mathematically in the same way you would have a proof of geometry that neural networks can approximate

any convex continuous function.

So for any convex continuous function essentially a function that you can continually integrate over

there should be somewhere out there some network that can directly approximate that function.

And while you can't maybe find that network directly and it may take quite a bit of time before you

train the network find the right number of neurons and the width and depth there theoretically and it's

been proven mathematically.

There is some network out there that can actually approximate that function so for more details on this

since the scope of that proof in the mathematics is outside the scope of this course and it's not really

important that you understand that to actually build out these neural networks you can check out the

wikipedia page for universal approximation theorem.

It's a really cool page.

I definitely encourage you to check it out but that page basically discusses in more detail how those

mathematical proofs actually work and what's really cool about a lot of those mathematical proofs is

years even proofs on placing restraints on the width in order to match up to the inputs of the continuous

function and you can still prove mathematically that there is some network out there even with those

constraints of the number of neurons with and then at work that will still approximate that continuous

function.

So definitely something I encourage you to check out if you're interested in that now previously if

you recall in our simple model of the single perceptual one we saw that that single perceptual on that

neuron if you will it contains a very simple summation function.

We essentially took our inputs X multiplied them by their own weights W and then we added that neurons

bias and sum that all up.

However in most used cases we really don't want to just do a straight sum.

Instead we're going to want to be able to set constraints to our output values especially in classification

tasks so you can imagine in the classification tasks it would be really useful to have all the outputs

fall between 0 and 1 and then these values can present the probability assignments for each class.

If we just have a large summation we have no upper limit on what values can be.

So you should always start thinking well what kind of functions can we apply to those inputs times the

weights plus the bias in order to put constraints or upper and lower limits and what the neuron value

will actually spit out.

And that's where activation functions come into play.

So in the next lecture we're gonna explore how to use activation functions to set boundaries to output

values from the neuron instead of just summing everything up like within that simple perception model.

And there's a wide variety of activation functions which is why we'll have an entire lecture on it.

So up next we're going to talk about activation functions and a few options you'll have we'll see whether.