All language subtitles for 006 How to Choose the Number of Clusters_en

Narrator: We've been juggling

with the number of clusters for too long.

Isn't there a criterion

for setting the proper number of clusters?

Luckily for us, there is,

probably the most widely adopted criterion

is the so-called, elbow method.

What's the rationale behind it?

Well, remember that clustering was about

minimizing the distance between points and a cluster

and maximizing the distance between clusters.

It turns out that for k-means

these two occur simultaneously,

if we minimize the distance

between points and a cluster,

we are automatically

maximizing the distance between clusters,

one less thing to worry about.

Now, the distance between

points and a cluster sounds clumsy, doesn't it?

That distance is measured in sum of squares

and the academic term is,

within-cluster sum of squares, or WCSS,

not much better, but at least the abbreviation is nice.

Okay, similar to SST, SSR and SSE from regressions

WCSS is a measure developed within the ANOVA framework,

if we minimize WCSS

we have reached the perfect clustering solution.

Here's the problem,

if we have the same six countries

and each one of them is a different cluster,

so a total of six clusters, then WCSS is zero,

that's because, there is just one point in each cluster

and we can't have a within-cluster sum of squares,

furthermore, the clusters

are as far as they can possibly be.

Imagine this with 1,000,000 observations,

a 1,000,000 cluster solution is definitely of no use,

similarly, if all observations are in the same cluster

the solution is useless and WCSS is at its maximum.

There must be some middle ground.

Applying some common sense,

we easily reach the conclusion

that we don't really want WCSS to be minimized,

instead, we want it to be as low as possible

while we can still have a small number of clusters,

so we can interpret them.

All right, if we plot WCSS against the number of clusters

we get this pretty graph.

It looks like an elbow, hence the name.

The point is that, the within-cluster sum of squares

is a monotonously decreasing function,

which is lower for a bigger number of clusters.

Here's the big revelation,

in the beginning, WCSS is declining extremely fast

at some point, it reaches the elbow

afterwards we are not reaching a much better solution

in terms of WCSS by increasing the number of clusters.

For our case,

we say that the optimal number of clusters

is three, as this is the elbow,

that's the biggest number of clusters for which

we are still getting a significant decrease In WCSS

thereafter, there is almost no improvement, cool.

How can we put that to use?

We need two pieces of information,

the number of clusters, k

and the WCSS for a specific number of clusters.

K is set by us at the beginning of the process,

while there is an SK learn method that gives us the WCSS,

for instance, to get the WCSS for our last example,

we just write k-means dot inertia underscore

to plot the elbow,

we actually need to solve the problem with 1, 2, 3 and so on

clusters and calculate WCSS for each of them.

Let's do that with a loop.

First, I'll declare an empty list called WCSS.

for i in range one to seven,

as we have a total of six observations, colons.

K-means equals k-means with capital K and M of i.

Next, I want to fit the input data x using k-means,

so k-means dot fit x

then we will calculate the WCSS for the iteration

using the inertia method.

let WCSS underscore iter be equal to k-means dot inertia.

Finally, we will add the WCSS for the iteration

to the WCSS list,

a handy method to do that is append,

if you are not familiar with it,

just pick the list dot append

and in brackets you can include the value

you'd like to append to the list,

so WCSS dot append brackets WCSS iter,

cool, let's run the code.

WCSS should be a list which contains

the within-cluster sum of squares

for one cluster, two clusters, and so on until six,

as you can see, the sequence is decreasing

with very big leaps in the first two steps

and much smaller ones later on,

finally, when each point is a separate cluster

we have a WCSS equal to zero, let's plot that,

we have WCSS, so let's declare a variable called,

number clusters, which is also a list from one to six.

Number clusters equals range one, seven, cool.

Then using some conventional plotting code,

we get the graph.

Finally, we will use the elbow method to decide

the optimal number of clusters.

There are two points, which can be the elbow,

this one and that one.

A three cluster solution is definitely the better one

as after it there's not much to gain.

A two cluster solution in this case would be suboptimal

as the leap from two to three is very big in terms of WCSS.

Okay, let's wrap it up here

and we will practice this new knowledge

on other data sets in our next lessons.

Thanks for watching.