Implementing Time Series Cross Validation to Evaluate the Forecasting Model Performance

Abstract

Theoretically, forecast error increases as the forecast horizon increases. This study aims to assess whether the statement is generally accepted or not. This study applies time series cross-validation to evaluate forecasting results up to seven steps ahead. As an illustration, we use Malaysia’s hourly electricity load data. Each hour is considered a series of each, so there are 24 daily series. Time series cross-validation with a 334 window was applied to 24 data series, and then each daily series was modeled with the Autoregressive Integrated Moving Average (ARIMA), Neural Network Autoregressive (NNAR), ExponenTial Smoothing (ETS), Singular Spectrum Analysis (SSA), and General Regression Neural Network (GRNN) models. In terms of mean absolute percentage error (MAPE) from one to seven steps ahead, we then evaluate the performance of all models. The experimental results show that the MAPEs obtained from the GRNN model tend to increase along with the theory. However, MAPEs obtained from ETS increase by up to three steps ahead and decrease after that. Among the five models, ARIMA, NNAR, and SSA produce a reasonably stable MAPE value for one to seven steps ahead. However, SSA has the most stable error value compared to ARIMA and NNAR. Keywords: time series, cross-validation, evaluate, forecasting model performance