Comparing two model selection frameworks for probabilistic load forecasting

Model selection is an important step for both point and probabilistic load forecasting. In the point load forecasting literature and practices, point error measures, such as mean absolute percentage error (MAPE), are often used for model selection. On the other hand, many probabilistic load forecasting methodologies rely on the model selection mechanism developed for point load forecasting. In other words, the models for probabilistic load forecasting are selected to minimize point error measures rather than probabilistic ones, such as quantile score. Intuitively, selecting models for probabilistic forecasting based on a point error measure is less computationally intensive and less accurate than its counterpart. The practical question is whether we can gain significant accuracy by taking the more computationally intensive route. This paper presents a comparative study on model selection for probabilistic load forecasting, using point and probabilistic error measures respectively. The data for the case study is from the load forecasting track of the Global Energy Forecasting Competition 2014. We find that the two model selection mechanisms indeed return different underlying models. While on average, the models from quantile score based model selection method can lead to more accurate probabilistic forecasts, the improvement over the MAPE based model selection method is marginal.