All language subtitles for 8. Modelling - Splitting Data

Look at us go.

We're moving to this framework at lightning pace.

We've done Problem Definition.

We've looked at data we've decided on an evaluation metric.

We've understood a few of the features we've got in our data.

Now we're up to step five which is modelling.

Now there's a few parts to modelling.

So we've broken this down into four different sections.

And this is where it's Section One.

And this is probably the most important concept in machine learning that three sets.

And now over the whole of modelling we want to answer the question based on our problem and data what

machine learning model should we use modelling can be broken down into three parts choosing and training

a model churning a model and model comparison before we get into these though.

Part one of modelling is and the most paramount topic to discuss in this whole entire course is the

most important concept in machine learning.

The train validation and test splits or commonly referred to as three sets.

Now since you want to be using machine learning models to gain insights on some data to predict the

future it's important to test how well they would go and do in the real world.

To do this you split your data into three different sets a training set to train your model on a validation

set to choosing your model on a test set to test and compare your different models why is this important.

Think of it like this when you're at university you might study the Course materials all through the

semester then before the final exam You might see how you could improve your knowledge on a practice

exam.

After doing well on the practice exam you're confident you'll do well on the final exam when you take

the final exam.

And although some of the problems you've never seen before you're able to adapt the knowledge you've

learned from the study materials to the slightly different but similar questions on the final exam.

Because of this you pass the final exam with great marks.

This adaptation that you had from the course materials and practice exams to the final exam is referred

to in machine learning as a generalisation or the ability for a machine learning model to perform well

on data it hasn't seen before because of what it's learned.

On another dataset Now where might this go wrong well if your professor accidentally sent out the final

exam for everyone to practice on when it came time to the actual exam.

Everyone would have already seen it now.

Since people know what they should be expecting they go through the exam.

They answer all the questions with ease and everyone ends up getting top marks.

Now top marks might appear good but did the students really learn anything or were they just expert

memorization machines for your machine learning models to be valuable at predicting something in the

future on unseen data you'll want to avoid them becoming memorization machines.

This is where training validation and test splits come in.

In our heart disease example let's say there were 100 patients you start off with 100.

One way to create these splits is to shuffle these patients.

Then select 70 percent for training which would mean that would be about 70.

Patient records.

And 15 percent for validation and 15 percent for testing which means to be 70 patients in the training

set.

15 patients in the validation split and 15 patients in the test split.

Now the percentages of each of these may vary but standard practice is usually around 70 to 80 percent

for training.

10 to 15 for validation and 10 15 for test.

You may see in some examples that some sets or some data sets only get split into training and test.

But that's case by case scenario.

Usually you'll have three different sets then once you've got these splits.

Using a model you've chosen you'd feed at the training data or the information of of these 70 patient

records.

And once your model had trained you can check its results and see if you can improve them on the validation

set.

This is where you do model tuning.

So just because you're machine learning the model's got one set of results and the patient records you

can actually improve them and we'll see this in a future lesson on the validation split.

Well the validation split is where you should be testing to see if you can improve.

Finally once you've improved your model you can check the models results as well as any other models

results that you might have done during experimentation on the test said what's important to remember

is that all three of these sets a separate during training the model never sees the validation split

or the test split.

And during testing you're doing it on the test split not the training set it's the same as when you

were studying for your exam if you saw the final exam whilst practicing that would be cheating and your

final result wouldn't reflect how well you'd learned.

For now think about it the last time you went for a test did you practice beforehand.

Was the practice you were doing helpful for the test and when you're thinking about this try and think

of how the lines to why it's important to not let a machine learning model see a test set or test data

simply whilst it's training.