Response to Hemilä and Chalker's “Pitfalls in choosing data examples for methodological work: Bayesian approaches to a fixed effects meta-analysis of zinc lozenges for the common cold”

Abstract

We thank Drs Harri Hemilä and Elizabeth Chalker for their post-publication review of our work. Using a metaanalysis from a now-withdrawn systematic review on zinc for the common cold was indeed an unfortunate oversight. In addition, we should have better corroborated the data from each of the studies in the meta-analysis, to avoid further propagation of numerical errors. We completely agree that this meta-analysis should not have been cited nor used as an applied example, and we apologize to researchers who were confused by our mistake. We take issue, however, with other points raised by Hemilä and Chalker. First, we disagree that the “validity” of Figures 3–5 is in question. The objective of our manuscript is to propose and evaluate a Bayesian approach to fixed effects meta-analysis and to compare the performance of such approach to other more traditional ones. The point made in Figures 3–5 is that, compared with existing defaults, the proposed method can provide considerable stability and robustness to choices of prior. This same point could have been made using a different example, or even fictional or simulated data. And indeed, the manuscript includes a second example in the Appendix section, from which similar conclusions about the approach can be drawn. Our manuscript is about statistical methods for meta-analysis, not a systematic review on zinc lozenges, and we stand by our method's validity. Second, Hemilä and Chalker object to our use of mean differences. We chose this scale deliberately, to simplify the exposition and enable readers to focus on novel aspects of our method. We are not convinced by their “strong” arguments for blanket use of one scale over another: interpreted with care, either absolute or relative differences can be usefully analyzed. Finally, Hemilä and Chalker complain that we insufficiently increase the understanding of zinc lozenges' effect on the common cold. This complaint simply misses the point of the manuscript which, as the title and abstract make clear, is to discuss novel statistical methods. Were our focus the understanding of zinc lozenges, we would have published a different paper in a different journal.