Statistical consequences of staging exploration and confirmation

Abstract

A number of methodologists have recently argued that it is inadvisable or even improper to use the same data for exploration (discovering effects) and for confirmation (validating the existence of effects). This has led to suggestions of a two-phased strategy: running an exploratory study (Phase 1) and then performing a Phase 2 validation/confirmation study (ideally pre-registered) that tests just the strongest effect(s) to emerge from Phase 1. Using simulations we ask a simple question: how does this phased strategy compare with the simpler alternative of running “one big study” that combines exploration and confirmation? At any given alpha level, two figures of merit trade off against each other, with the 2-phased strategy offering lower power and greater positive predictive value (PPV). However, a closer comparison of the results show that the “big study” option is strictly dominant in the sense that for any given alpha level used in the two-phased strategy, there is some alpha level for which the “big study” approach yields better power and better PPV. Bonferroni correction for multiple comparisons does not affect this result. The implications and their important limitations are discussed.