VAMP1RE: a single criterion for rating and ranking confidence-interval procedures

Abstract

We propose VAMP1RE, a single criterion for rating and ranking confidence-interval procedures (CIPs) that use a fixed sample size. The quality of a CIP is traditionally thought to be many dimensional, typically composed of the probability of covering the unknown performance measure and the mean (and sometimes the standard deviation) of interval width, each of these over some set of nominal coverage probabilities. These many criteria reflect symptoms, rather than causes, of CIP quality. The VAMP1RE criterion focuses on two causes: departure from validity—violation of assumptions—and inability to mimic—the dissimilarity, for every data set, of a CIP’s interval to that of an ideal CIP. The ideal CIP is both valid (that is, adheres to all assumptions) and is an agreed-upon standard; possibly the ideal CIP is allowed knowledge not available to the real-world CIPs of interest. A high inability to mimic the ideal CIP implies that a CIP uses data inefficiently. For a given CIP, the VAMP1RE criterion is the expected squared difference between Schruben’s coverage values (analogous to p values) arising from the given CIP and from the ideal CIP. The implication is that an interval arising from a particular data set is good not because it is large or small but, rather, it is good to the extent that it is similar to the interval provided by the ideal CIP. We discuss the relationship to Schruben’s coverage function, provide a graphical interpretation, decompose the VAMP1RE criterion into the two cause components, and provide examples to illustrate that the VAMP1RE criterion provides numerical values that are useful for rating and ranking CIPs.