Conflicting results and statistical malleability: embracing pluralism of empirical results

Abstract

Conflicting results undermine making inferences from the empirical literature. So far, the replication crisis is mainly seen as resulting from honest errors and questionable research practices such as p-hacking or the base-rate fallacy. We discuss the malleability (researcher degrees of freedom) of quantitative research and argue that conflicting results can emerge from two studies using different but plausible designs (e.g., eligibility criteria, operationalization of concepts, outcome measures) and statistical methods. We also explore how the choices regarding study design and statistical techniques bias results in a way that makes them more or less relevant for a given policy or clinical question.