Multiple comparisons with a standard using false discovery rates

Abstract

We introduce a new framework for performing multiple comparisons with a standard when simulation models are available to estimate the performance of many different systems. In this setting, a large proportion of the systems have mean performance from some known null distribution, and the goal is to select alternative systems whose means are different from that of the null distribution. We employ empirical Bayes ideas to achieve a bound on the false discovery rate (proportion of selected systems from the null distribution) and a desired probability an alternate type system is selected.