Adaptive Multiple Comparisons With the Best

Subset selection methods aim to choose a nonempty subset of populations including a best population with some prespecified probability. An example application involves location parameters that quantify yields in agriculture to select the best wheat variety. This is quite different from variable selection problems, for instance, in regression.

Unfortunately, subset selection methods can become very conservative when the parameter configuration is not least favorable. This will lead to a selection of many non-best populations, making the set of selected populations less informative. To solve this issue, we propose less conservative adaptive approaches based on estimating the number of best populations. We also discuss variants of our adaptive approaches that are applicable when the sample sizes and/or variances differ between populations. Using simulations, we show that our methods yield a desirable performance. As an illustration of potential gains, we apply them to two real datasets, one on the yield of wheat varieties and the other obtained via genome sequencing of repeated samples.