Uncertainties in the observational reference: implications in skill assessment and model ranking of seasonal predictions

The probabilistic skill of seasonal prediction systems is often inferred using reanalysis data, assuming these benchmark observations to be error-free. However, an increasing number of studies report non-negligible levels of uncertainty affecting reanalysis observations, especially when it comes to variables like precipitation or wind speed. We consider different possibilities to account for such error in forecast quality assessment, either exploiting the newly produced ensemble reanalyses (e.g. ERA5-EDA) or applying methodologies that use scores that take observational uncertainty into account. We illustrate the benefits of employing ensemble reanalyses over traditional reanalyses, and show how the true skill can be approximated, whatever the observational reference. We ultimately emphasise the perils and quantify the error committed when the observational reference, either reanalysis or point dataset, is selected arbitrarily for verifying a seasonal prediction system.