All language subtitles for 8. Forecasting Metrics

So in this lecture, we are going to discuss error metrics commonly used in Time series analysis now

because Time series forecasting is essentially regression.

You'll find that if you've ever studied regression, these error metrics are the same as what you've

encountered before.

So one metric that shows up very often in statistics, machine learning, deep learning, engineering

and so forth is the sum of squared errors.

Suppose that we have end predictions so we have AI going from one up to N for the sum of squared errors.

We simply take the difference between each with Sabai and we had Hasbi Square that difference and add

all the square differences together.

Pretty simple.

The reason why we want to square these differences is because sometimes the prediction may be less than

the target, but other times the target may be less than the prediction.

Since we don't want them to cancel, scoring them ensures that the error is always non-negative.

One bonus to using the squared error is that it coincides with maximizing the Gaussian likelihood.

That is, it's the correct error metric to minimize when your errors are normally distributed.

Since this is a pretty common assumption, the squared error or the variance of it that we're about

to discuss make a lot of sense to use.

Now, one downside to the sum of squared errors is that it depends on the number of data points you

have supposed that you have an equals one hundred predictions.

You can imagine that if you have an equals one thousand predictions, this new era will be a lot bigger,

simply due to the fact that you had to make ten times more predictions.

So it's not easy to compare, say, two different data sets with a different number of samples using

the sum of squared errors.

However, there is an easy fix for this, which is to use the mean squared error.

It's very simple.

Just divide the sum of squared errors by the number of samples in.

By doing this, you make the error metric invariant to the number of samples.

One advantage of this is that it serves to represent the sample mean of the squared errors.

That is, it's an estimate of the expected value of the square error.

For many algorithms, their objective is to minimize this expected value.

So we can build on the squared error a little more.

One downside to both the sum of squared errors and the means square is that they don't have intuitive

units.

Imagine you're forecasting temperature in Calvin's, but using the squared error, we'll give you Kelvin

squared.

I don't know about you, but I have no intuition about the meaning of a squared Kelvin or a squared

temperature unit.

So one way to express this error metric in units that make sense is to take the square root of the mean

squared error.

We call this the root mean squared error or messy for obvious reasons.

The advantage of this error metric is that it's on the same scale as the original data.

So if you're predicting the price of a house, it makes more sense to say My arm AC is one hundred dollars

instead of my mzee is 10000 square dollars.

Note that this can still be a bit unintuitive when you're comparing numbers to see this.

Consider what happens when you take the square root of a number bigger than one.

In this case, the value will get smaller.

But if we take the square root of a number less than one, the value actually gets bigger.

So it's kind of a strange function in that sense.

Now, you might wonder, why should we work with Squarer is at all if we want positive values, why

not simply take the absolute value?

In fact, this is entirely possible if we take the average absolute difference between our targets and

our predictions, we get the mean absolute error.

So this should be pretty intuitive.

You can see right away some advantages of this error metric.

Clearly, it's immediately on the same scale as our data, so there's no need to take a square root.

It also happens to have a probabilistic interpretation specifically, whereas the squarer coincides

with optimizing a Gaussian likelihood, the absolute error coincides with optimizing a distributed likelihood.

Now the details are outside the scope of this course.

But essentially, if you optimize this lost function, your model will be less influenced by outliers,

which could be a good thing in practice.

Something I find quite interesting is that people will train their model by using the squared error,

but then report the absolute error as a metric.

Some of the libraries we will use in this course won't give you a choice, but it's my opinion that

if you're going to pick some error to minimize, then it makes more sense to also report that error

metric.

So let's see how we can take things a little bit further.

One downside to both the mean square error and the mean absolute error is that they depend on the scale

of the data.

For example, if you're trying to predict house prices, house prices are on the scale of hundreds of

thousands to millions of dollars, your error will be proportionally large.

On the other hand, if you're trying to predict daily stock returns.

These are very minuscule on the order of fractions of a percent.

So it's not straightforward to compare which of these tasks is easier.

Your error for stock returns might be a percent of a percent, but your error for house prices might

be in the thousands of dollars.

But this doesn't imply that predicting house prices is harder than predicting stock returns.

Note that this is unlike tasks like classification, where you're either correct or incorrect.

If you're correct 80 percent of the time, then your accuracy is 80 percent.

And this is the case no matter the data set.

So it seems like it would be pretty useful to have a scale invariant metric.

One common metric that is scale and invariant is the R squared.

Note that the R squared is not like an error in that we want this to be bigger, not smaller.

One simple way to express the R squared is by taking the ratio between the sum of squared errors called

the SC and the total sum of squares called the S.T. and then we subtract that from one.

So to explain this further, the S is essentially what we would get if our prediction was the mean and

we took the sum of squared errors of that prediction.

Another way to think of this is that if we divide both the top and bottom by N, we get the mean squared

error divided by the sample variance of the targets.

So this is one way to think of how good your model is.

If your model has perfect predictions, then your MSE will be zero and your R-squared will be one.

If your model is terrible and you can only predict the average of the targets, then your mzee will

be equal to the sample variance and you'll have one minus one, which is zero.

So just to drill that in and I'm scared of one is a model with perfect predictions and I sort of zero

is a model that does no better than simply predicting the mean.

Clearly, this is invariant to the scale of the data, which was our original motivation.

One thing you should note is that it's possible for the R-squared to be negative.

Imagine, for example, your predictions are worse than simply predicting the mean of the targets.

In this case, the difference between why and why hat will be bigger than that of why and why bother.

And so the numerator will be bigger than the denominator and the whole thing will be bigger than one.

One minus a number bigger than one will be negative, giving you a negative R squared.

In fact, the R squared is unbounded in the negative direction.

So this is unlike classification accuracy, which must be between zero and one.

It's worth noting that with Saikat Learn, which is probably the most popular machine learning library,

the score function computes the R-squared by default for regression.

For classification, you get the classification accuracy.

So just something to keep in mind for later when we use IQ, learn.

OK, so we're still not quite done since the Field of Time series analysis for some reason likes to

have lots of metrics.

So we're on the topic of scale invariant metrics.

One obvious way to think of how accurate your model is, is with a percentage if a house is one million

dollars.

But my prediction is one million.

One thousand dollars.

I don't mind because that's only a zero point one percent difference.

On the other hand, if I'm predicting the price of something that costs one thousand dollars and I'm

off by one thousand dollars, that's a huge error because I'm off by 100 percent.

So the mean absolute percentage error or the map expresses this idea.

Now, one downside to the map is that it's not symmetric as an example, if your target is 10 and your

prediction is 11.

This leads to a different value when your prediction is 10 and your target is 11.

Of course, some smart person has thought of this already and come up with the symmetric map or SMAP.

As you can see, this just takes the average of why and why hat in the denominator so that the result

is symmetric.

The reason I mention this one is that it shows up in a paper we're going to look at.

So it's nice to cover now.

So despite the map and the map being somewhat popular, there is one problem.

What happens when the denominator is zero?

The result is that the error explodes to infinity.

Of course, this makes no sense, since the error should not explode to infinity simply because the

data takes on certain values.

It should ideally only explode to infinity if your target in your prediction are very far apart.

Nonetheless, these are popular metrics, so they are worth knowing.

So you must be wondering, what is the point of having so many metrics to choose from?

Well, the goal is to give you exposure to this field so that when you're reading papers or communicating

with other professionals, you share a common language.

And again, a common theme of this course is that there are many options for you to try.

This leads to a combinatorial explosion of options to choose from in this course.

We will probably never use all of these metrics in the same example, if we tried every technique every

time, this course would never end.

So again, the purpose of this is to make you aware of these tools so that you can apply them in your

work if you think that they would be useful.