Models vetted against prediction error and parameter sensitivity standards can credibly evaluate ecosystem management options

Abstract

A new standard for assessing model credibility is developed. This standard consists of parameter estimation, prediction error assessment, and a parameter sensitivity analysis that is driven by outside individuals who are skeptical of the model’s credibility (hereafter, skeptics). Ecological/environmental models that have a one-step-ahead prediction error rate that is better than naive forecasting — and are not excessively sensitive to small changes in their parameter values are said here to be vetted. A procedure is described that can perform this assessment on any model being evaluated for possible participation in an ecosystem management decision. Uncertainty surrounding the model’s ability to predict future values of its output variables and in the estimates of all of its parameters should be part of any effort to vett a model. The vetting procedure described herein, Prediction Error Rate-Deterministic Sensitivity Analysis (PER-DSA), incorporates these two aspects of model uncertainty. DSA in particular, requires participation by skeptics and is the reason why a successful DSA gives a model sufficient credibility to have a voice in ecosystem management decision making. But these models need to be stochastic and represent the mechanistic processes of the system being modeled. For such models, performing a PER-DSA can be computationally expensive. A cluster computing algorithm to speed-up these computations is described as one way to answer this challenge. This new standard is illustrated through a PER-DSA of a population dynamics model of South African rhinoceros (Ceratotherium simum simum).