Credibility of results in emotion science: a Z-curve analysis of results in the journals Cognition & Emotion and Emotion.

Abstract

Failed replication attempts have raised concerns over the prevalence of publication bias and false positive results in the psychological literature. Using a sample of 65,970 test statistics from Cognition & Emotion and Emotion, this article assesses the credibility of results in emotional research. All test statistics were converted to z-scores and analysed with Z-curve. A Z-curve analysis provides information about the amount of selection bias, the expected replication rate and the false positive risk. Lastly, Z-curve is used to determine an alpha level that lessens the false positive risk without unnecessary loss of power. The results show evidence of selection bias in emotional research, but trend analyses showed a decrease over time. Based on the z-curve estimates, we predict a 15% and 70% success rate in replication studies. Therefore, replication studies should increase sample sizes to avoid type-II errors. The risk of false positives with the traditional alpha level of 5% is between 5% and 33%. Lowering alpha to 1% is sufficient to reduce the false positive risk to less than 5%. In sum, our findings may alleviate concerns about high false positive rates among emotional researchers. However, selection bias and low power remain challenges to be addressed.