Information-based Probabilistic Verification Scores for Two-dimensional Ensemble Forecast Data: A Madden-Julian Oscillation Index Example

Abstract

Probabilistic forecasting is a common activity in many fields of the Earth sciences. Assessing the quality of probabilistic forecasts—probabilistic forecast verification—is therefore an essential task in these activities. Numerous methods and metrics have been proposed for this purpose; however, the probabilistic verification of vector variables of ensemble forecasts has received less attention than others. Here we introduce a new approach that is applicable for verifying ensemble forecasts of continuous, scalar and two-dimensional vector data. The proposed method uses a fixed radius near-neighbors search to compute two information-based scores, the ignorance score (the logarithmic score) and the information gain, which quantifies the skill gain from the reference forecast. Basic characteristics of the proposed scores were examined using idealized Monte Carlo simulations. The results indicated that both the Continuous Ranked Probability Score (CRPS) and the proposed score with a relatively small ensemble size (< 25) are not proper in terms of the forecast dispersion. The proposed verification method was successfully used to verify the Madden-Julian Oscillation index, which is a two-dimensional quantity. The proposed method is expected to advance probabilistic ensemble forecasts in various fields.