What limitations are reported in short articles in social and personality psychology?

Abstract

Every research project has limitations. The limitations that authors acknowledge in their articles offer a glimpse into some of the concerns that occupy a field's attention. We examine the types of limitations authors discuss in their published articles by categorizing them according to the four validities framework and investigate whether the field's attention to each of the four validities has shifted from 2010 to 2020. We selected one journal in social and personality psychology (Social Psychological and Personality Science; SPPS), the subfield most in the crosshairs of psychology's replication crisis. We sampled 440 articles (with half of those articles containing a subsection explicitly addressing limitations), and we identified and categorized 831 limitations across the 440 articles. Articles with limitations sections reported more limitations than those without (avg. 2.6 vs. 1.2 limitations per article). Threats to external validity were the most common type of reported limitation (est. 52% of articles), and threats to statistical conclusion validity were the least common (est. 17% of articles). Authors reported slightly more limitations over time. Despite the extensive attention paid to statistical conclusion validity in the scientific discourse throughout psychology's credibility revolution, our results suggest that concerns about statistics-related issues were not reflected in social and personality psychologists' reported limitations. The high prevalence of limitations concerning external validity might suggest it is time that we improve our practices in this area, rather than apologizing for these limitations after the fact. (PsycInfo Database Record (c) 2023 APA, all rights reserved).