Experimentally assessing the uncertainty of forecasts of geological carbon storage

Abstract

Geological storage of carbon dioxide is a cornerstone in almost every realistic emissions reduction scenario outlined by the Intergovernmental Panel on Climate Change. Our ability to accurately forecast storage efficacy is, however, mostly unknown due to the long timescales involved (hundreds to thousands of years). To study perceived forecast accuracy, we designed a double-blind forecasting study. As ground truth, we constructed a laboratory-scale carbon storage operation, retaining the essential physical processes active on the field scale, within a time span of five days. Separately, academic groups with experience in carbon storage research were invited to forecast key carbon storage efficacy metrics. The participating groups submitted forecasts in two stages: First independently without any cross-group interaction, then finally after workshops designed to share and assimilate understanding between the forecast groups. Their confidence in reported forecasts was monitored throughout the forecasting study. Our results show that participating groups provided forecasts that appear bias-free with respect to carbon storage as a technology, yet the forecast intervals are too narrow to capture the ground truth (overconfidence bias). When asked to qualitatively self-assess their forecast uncertainty (and later when asked to provide an external assessment of other forecast groups), the assessment of the participants indicated an understanding that the forecast intervals (both their own and those of others) were too narrow. However, the participants did not display an understanding of how poorly the forecast intervals calibrated to the ground truth. The quantitative uncertainty assessments contrast the qualitative comments supplied by the participants, which indicate an acute awareness of the challenges associated with assessing the uncertainty of forecasts for complex systems such as the geological storage of carbon dioxide.