Sequential stratified inference for the mean

Abstract

We develop conservative tests for the mean of a bounded population using data from a stratified sample. The sample may be drawn sequentially, with or without replacement. The tests are "anytime valid," allowing optional stopping and continuation in each stratum. We call this combination of properties sequential, finite-sample, nonparametric validity. The methods express a hypothesis about the population mean as a union of intersection hypotheses describing within-stratum means. They test each intersection hypothesis using independent test supermartingales (TSMs) combined across strata by multiplication. The $P$-value of the global null hypothesis is then the maximum $P$-value of any intersection hypothesis in the union. This approach has three primary moving parts: (i) the rule for deciding which stratum to draw from next to test each intersection null, given the sample so far; (ii) the form of the TSM for each null in each stratum; and (iii) the method of combining evidence across strata. These choices interact. We examine the performance of a variety of rules with differing computational complexity. Approximately optimal methods have a prohibitive computational cost, while naive rules may be inconsistent -- they will never reject for some alternative populations, no matter how large the sample. We present a method that is statistically comparable to optimal methods in examples where optimal methods are computable, but computationally tractable for arbitrarily many strata. In numerical examples its expected sample size is substantially smaller than that of previous methods.