Simulation testing performance of ensemble models when catch data are underreported

Abstract

Ensemble model use in stock assessment is increasing, yet guidance on construction and an evaluation of performance relative to single models is lacking. Ensemble models can characterize structural uncertainty and avoid the conundrum of selecting a “best” assessment model when alternative models explain observed data equally well. Through simulation, we explore the importance of identifying candidate models for both assessment and short-term forecasts and the consequences of different ensemble weighting methods on estimated quantities. Ensemble performance exceeded a single best model only when the set of candidate models spanned the true model configuration. Accuracy and precision depended on the model weighting scheme, and varied between two case studies investigating the impact of catch accuracy. Information theoretic weighting methods performed well in the case study with accurate catch, while equal weighting performed best when catch was underreported. In both cases, equal weighting produced multimodality. Ensuring that an ensemble spans the true state of nature will be challenging, but we observed that a change in sign of Mohn’s rho across candidate models coincided with the true OM being bounded. Further development of protocols to select an objective and balanced set of candidate models, and diagnostics to assess adequacy of candidates are recommended.