9. Metrics#
There are several metrics used in time series analysis to evaluate the accuracy of forecasting models. Here are some of the commonly used ones:
9.1 Mean Absolute Error (MAE)#
The MAE is the average of the absolute difference between the forecasted values and the actual values. It measures the magnitude of the errors in the forecasts without considering their direction. The corresponding mathematical formula for Mean Absolute Error (MAE) in time series analysis is:
\begin{equation} MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i| \end{equation}
In this formula, \(y_i\) is the actual value at time i, and \(\hat{y}_i\) is the forecasted value at time i, and n is the number of observations.
The MAE, or Mean Absolute Error, is a commonly used metric for evaluating the accuracy of a forecast model in time series analysis. It represents the average absolute difference between the actual values and the forecasted values over a period of time.
When the MAE is small, it indicates that the forecast model is accurately predicting the future values of the time series, and is well-fitted to the historical data. This means that the predicted values are very close to the actual values, on average.
On the other hand, if the MAE is large, it suggests that the forecast model is not accurately predicting the future values of the time series, and may not be well-fitted to the historical data. This means that the predicted values are often far from the actual values, on average.
In general, a small MAE is desirable and indicates that the forecast model is well-fitted to the time series data, while a large MAE suggests that the forecast model needs to be improved. However, it is important to keep in mind that the choice of appropriate error metrics for a particular time series forecasting problem depends on various factors, such as the nature of the data, the forecasting horizon, and the specific goals of the forecasting project.
9.2 The mean error (ME)#
The mean error (ME) is a simple metric used in time series analysis to measure the average difference between the actual values and the forecasted values over a period of time. The ME can be positive or negative, depending on whether the forecasted values are higher or lower than the actual values, respectively.
The mathematical formula for mean error in time series analysis is:
\begin{equation} ME = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i) \end{equation}
In this formula, \(y_i\) is the actual value at time i, \(\hat{y}_i\) is the forecasted value at time i, and n is the number of observations. A positive ME indicates that the forecasts tend to be lower than the actual values, while a negative ME indicates that the forecasts tend to be higher than the actual values. The ME can be used in conjunction with other metrics, such as MAE and RMSE, to provide a more comprehensive evaluation of the forecasting performance. The ME, or mean error, is a simple measure of the forecast accuracy in time series analysis. It represents the average difference between the actual values and the forecasted values over a period of time. The ME can be positive or negative, depending on whether the forecasted values are higher or lower than the actual values, respectively.
When the ME is close to zero, it indicates that the forecasts are relatively accurate and unbiased, and there is no significant trend in the forecast errors. This means that the forecast model is fitted well to the time series data, and the predicted values are reasonably close to the actual values.
However, if the ME is consistently positive or negative, it indicates that the forecast model tends to overestimate or underestimate the actual values, respectively, and there is a systematic bias in the forecast errors. This may suggest that the forecast model is not well-suited to the time series data, and adjustments may need to be made to improve the forecast accuracy.
Overall, the ME can provide a simple and intuitive measure of the forecast accuracy in time series analysis, and can help to identify whether the forecast model is well-fitted to the time series data or not.
9.3 Root Mean Squared Error (RMSE)#
The RMSE is the square root of the mean of the squared differences between the forecasted values and the actual values. It penalizes large errors more than small ones and provides a measure of the spread of the errors.The mathematical formula for Root Mean Squared Error (RMSE) in time series analysis is:
In this formula, \(y_i\) is the actual value at time i, and \(\hat{y}_i\) is the forecasted value at time i, and n is the number of observations.
9.4 Mean Absolute Percentage Error (MAPE)#
The MAPE is the average of the absolute percentage difference between the forecasted values and the actual values. It measures the accuracy of the forecasts in percentage terms, making it easier to interpret. The mathematical formula for Mean Absolute Percentage Error (MAPE) in time series analysis is:
\begin{equation} MAPE = \frac{1}{n}\sum_{i=1}^{n}\left|\frac{y_i - \hat{y}_i}{y_i}\right| \times 100% \end{equation}
In this formula, \(y_i\) is the actual value at time i, and \(\hat{y}_i\) is the forecasted value at time i, and n is the number of observations. The MAPE expresses the forecast errors as a percentage of the actual values, making it a useful metric for comparing forecasting accuracy across different time series with different scales.
A MAPE score of 0 indicates a perfect model, whereas a score greater than 1 implies that the model’s predictions are consistently higher than the targets.
9.5 Symmetric Mean Absolute Percentage Error (SMAPE)#
The SMAPE is a modification of the MAPE that takes into account the scale of the data. It is calculated as the average of the absolute percentage difference between the forecasted values and the actual values, divided by the average of the sum of the absolute values of the forecasted and actual values.The mathematical formula for Symmetric Mean Absolute Percentage Error (SMAPE) in time series analysis is:
\begin{equation} SMAPE = \frac{1}{n}\sum_{i=1}^{n}\frac{\left|y_i - \hat{y}_i\right|}{\left(\left|y_i\right|+\left|\hat{y}_i\right|\right)/2} \times 100% \end{equation}
In this formula, \(y_i\) is the actual value at time i, and \(\hat{y}_i\) is the forecasted value at time i, and n is the number of observations. The SMAPE measures the percentage difference between the actual and forecasted values, taking into account the scale of the time series. Unlike MAPE, SMAPE is symmetric, meaning that it does not treat positive and negative errors differently.
9.6 Theil’s U-Statistic#
The Theil’s U-statistic measures the proportion of the total variability in the actual values that is explained by the forecasting model. It ranges from 0 to 1, with 1 indicating perfect forecasting accuracy.
The mathematical formula for Theil’s U-Statistic in time series analysis is:
\begin{equation} U = \frac{\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(\frac{y_i-\hat{y}i}{y_i}\right)^2} }{\sqrt{\frac{1}{n}\sum{i=1}^{n}\left(\frac{y_i-\bar{y}}{y_i}\right)^2}+\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\hat{y}_i-\bar{y}}{\hat{y}_i}\right)^2}} \end{equation}
In this formula, \(y_i\) is the actual value at time i, and \(\hat{y}_i\) is the forecasted value at time i, \(\bar{y}\) is the mean of the actual values, and n is the number of observations. The Theil’s U-Statistic is a relative measure of forecast accuracy that compares the root mean squared forecast error to the root mean squared deviation of the actual values from the mean. A value of U = 1 indicates perfect forecast accuracy, while U > 1 or U < 1 indicates worse or better forecast accuracy, respectively, than a naive forecast that simply uses the mean of the historical data. in the other words, The value of U ranges between 0 and 1, with 0 representing a flawless match. It is calculated using conditional entropy and can serve as a gauge of uncertainty or even as a categorical-categorical correlation measure.
9.7 MRAE#
Mean relative absolute error (MRAE) is a commonly used metric in time series analysis to evaluate the accuracy of a forecasting model. It is calculated as the mean of the absolute difference between the forecasted values and the actual values, divided by the mean of the absolute values of the actual values.
MRAE is expressed as:
\(MRAE = \frac{1}{n} \sum_{i=1}^{n} \frac{\left|y_{i} - \hat{y}{i}\right|}{\frac{1}{n} \sum{i=1}^{n}\left|y_{i}\right|}\)
where \(y_i\) is the actual value at time i, \(ŷ_i\) is the forecasted value at time i, and n is the number of observations.
MRAE provides a measure of the average magnitude of the forecasting errors relative to the size of the actual values. A lower MRAE value indicates better accuracy of the forecasting model. However, like any metric, MRAE should be used in conjunction with other metrics to fully evaluate the performance of a forecasting model.
It is important to note that each of these metrics has its own advantages and limitations, and it is recommended to use multiple metrics to evaluate the performance of a forecasting model.