Comparison of noncentral t and distribution-free methods when using sequential procedures to control the width of a confidence interval for a standardized mean difference.

Abstract

sequential stopping rule (SSR) can generate a confidence interval (CI) for a standardized mean difference d that has an exact standardized width, ω. Two methods were tested using a broad range of ω and standardized effect sizes δ. A noncentral t (NCt) CI used with normally distributed data had coverages that were nominal at narrow widths but were slightly inflated at wider widths. A distribution-free (Dist-Free) method used with normally distributed data exhibited superior coverage and stopped on average at the expected sample sizes. When used with moderate to severely skewed lognormal distributions, the coverage was too low at large effect sizes even with a very narrow width where Dist-Free was expected to perform well, and the mean stopping sample sizes were absurdly elevated (thousands per group). SSR procedures negatively biased both the raw difference and the "unbiased" Hedges' g in the stopping sample with all methods and distributions. The d was the less biased estimator of δ when the distribution was normal. The poor coverage with a lognormal distribution resulted from a large positive bias in d that increased as a function of both ω and δ. Coverage and point estimation were little improved by using g instead of d. Increased stopping time resulted from the way an estimate of the variance is calculated when it encounters occasional extreme scores generated from the skewed distribution. The Dist-Free SSR method was superior when the distribution was normal or only slightly skewed but is not recommended with moderately skewed distributions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).