On the Q statistic with constant weights for standardized mean difference

Abstract

Cochran's Q statistic is routinely used for testing heterogeneity in meta-analysis. Its expected value is also used in several popular estimators of the between-study variance, ${\tau }^2$. Those applications generally have not considered the implications of its use of estimated variances in the inverse-variance weights. Importantly, those weights make approximating the distribution of Q (more explicitly, $Q_{\text{IV}}$) rather complicated. As an alternative, we investigate a new Q statistic, $Q_F$, whose constant weights use only the studies' effective sample sizes. For the standardized mean difference as the measure of effect, we study, by simulation, approximations to distributions of $Q_{\text{IV}}$ and $Q_F$, as the basis for tests of heterogeneity and for new point and interval estimators of ${\tau }^2$. These include new DerSimonian–Kacker-type moment estimators based on the first moment of $Q_F$, and novel median-unbiased estimators. The results show that: an approximation based on an algorithm of Farebrother follows both the null and the alternative distributions of $Q_F$ reasonably well, whereas the usual chi-squared approximation for the null distribution of $Q_{\text{IV}}$ and the Biggerstaff–Jackson approximation to its alternative distribution are poor; in estimating ${\tau }^2$, our moment estimator based on $Q_F$ is almost unbiased, the Mandel – Paule estimator has some negative bias in some situations, and the DerSimonian–Laird and restricted maximum likelihood estimators have considerable negative bias; and all 95% interval estimators have coverage that is too high when ${\tau }^2=0$, but otherwise the Q-profile interval performs very well.