Pub Date : 2025-08-01Epub Date: 2023-07-27DOI: 10.1037/met0000597
Peter M Steiner, Patrick Sheehan, Vivian C Wong
Given recent evidence challenging the replicability of results in the social and behavioral sciences, critical questions have been raised about appropriate measures for determining replication success in comparing effect estimates across studies. At issue is the fact that conclusions about replication success often depend on the measure used for evaluating correspondence in results. Despite the importance of choosing an appropriate measure, there is still no widespread agreement about which measures should be used. This article addresses these questions by describing formally the most commonly used measures for assessing replication success, and by comparing their performance in different contexts according to their replication probabilities-that is, the probability of obtaining replication success given study-specific settings. The measures may be characterized broadly as conclusion-based approaches, which assess the congruence of two independent studies' conclusions about the presence of an effect, and distance-based approaches, which test for a significant difference or equivalence of two effect estimates. We also introduce a new measure for assessing replication success called the correspondence test, which combines a difference and equivalence test in the same framework. To help researchers plan prospective replication efforts, we provide closed formulas for power calculations that can be used to determine the minimum detectable effect size (and thus, sample sizes) for each study so that a predetermined minimum replication probability can be achieved. Finally, we use a replication data set from the Open Science Collaboration (2015) to demonstrate the extent to which conclusions about replication success depend on the correspondence measure selected. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
鉴于最近的证据对社会和行为科学结果的可复制性提出了挑战，在比较不同研究的效果估计时，确定复制成功的适当措施提出了关键问题。争论的焦点在于，关于复制成功与否的结论往往取决于用于评估结果一致性的测量方法。尽管选择适当的措施很重要，但对于应该使用哪些措施仍然没有广泛的共识。本文通过正式描述用于评估复制成功的最常用度量，并根据它们的复制概率（即给定特定研究设置的获得复制成功的概率）比较它们在不同上下文中的性能，来解决这些问题。这些措施可以被广泛地描述为基于结论的方法，评估两个独立研究关于效应存在的结论的一致性，以及基于距离的方法，测试两个效应估计的显着差异或等效性。我们还引入了一种评估复制成功的新方法，称为对应测试，它在同一框架中结合了差异测试和等效测试。为了帮助研究人员计划前瞻性的复制工作，我们提供了功率计算的封闭公式，可用于确定每个研究的最小可检测效应大小（从而确定样本量），从而可以实现预定的最小复制概率。最后，我们使用开放科学协作（2015）的复制数据集来证明关于复制成功的结论在多大程度上取决于所选择的对应度量。（PsycInfo Database Record (c) 2025 APA，版权所有）。
{"title":"Correspondence measures for assessing replication success.","authors":"Peter M Steiner, Patrick Sheehan, Vivian C Wong","doi":"10.1037/met0000597","DOIUrl":"10.1037/met0000597","url":null,"abstract":"<p><p>Given recent evidence challenging the replicability of results in the social and behavioral sciences, critical questions have been raised about appropriate measures for determining replication success in comparing effect estimates across studies. At issue is the fact that conclusions about replication success often depend on the measure used for evaluating correspondence in results. Despite the importance of choosing an appropriate measure, there is still no widespread agreement about which measures should be used. This article addresses these questions by describing formally the most commonly used measures for assessing replication success, and by comparing their performance in different contexts according to their replication probabilities-that is, the probability of obtaining replication success given study-specific settings. The measures may be characterized broadly as conclusion-based approaches, which assess the congruence of two independent studies' conclusions about the presence of an effect, and distance-based approaches, which test for a significant difference or equivalence of two effect estimates. We also introduce a new measure for assessing replication success called the correspondence test, which combines a difference and equivalence test in the same framework. To help researchers plan prospective replication efforts, we provide closed formulas for power calculations that can be used to determine the minimum detectable effect size (and thus, sample sizes) for each study so that a predetermined minimum replication probability can be achieved. Finally, we use a replication data set from the Open Science Collaboration (2015) to demonstrate the extent to which conclusions about replication success depend on the correspondence measure selected. (PsycInfo Database Record (c) 2025 APA, all rights reserved).</p>","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":"793-814"},"PeriodicalIF":7.8,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10259359","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yutian T. Thompson, Yaqi Li, Hairong Song, David Bard
{"title":"Joint variable selection in generalized linear mixed models with random regularized penalized quasi-likelihood technique.","authors":"Yutian T. Thompson, Yaqi Li, Hairong Song, David Bard","doi":"10.1037/met0000783","DOIUrl":"https://doi.org/10.1037/met0000783","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"52 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144748224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for An Explainable Artificial Intelligence Handbook for Psychologists: Methods, Opportunities, and Challenges","authors":"","doi":"10.1037/met0000772.supp","DOIUrl":"https://doi.org/10.1037/met0000772.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"27 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144715713","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for Joint Variable Selection in Generalized Linear Mixed Models With Random Regularized Penalized Quasi-Likelihood Technique","authors":"","doi":"10.1037/met0000783.supp","DOIUrl":"https://doi.org/10.1037/met0000783.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"283 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144715438","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This article proposes an approach for detecting multivariate outliers that combines robust estimation methods with signed detection information. Our method uses the Mahalanobis distance to quantify each observation's extremeness from the expected value relative to the covariance matrix, and we leverage robust estimation tools, i.e., the minimum covariance determinant, to estimate the mean vector and covariance matrix used in the Mahalanobis distance calculation. Furthermore, we incorporate a signing element into the distance calculation to give researchers greater control over the specific regions of multivariate space that should be prioritized when searching for outliers, which allows for more targeted risk assessment and classification. Lastly, we unify the robust and signed elements into a framework that can be used within bilinear models such as principal components analysis and factor analysis. Using simulated and real data examples, we demonstrate that the proposed approach can result in improved risk assessment and outlier detection, particularly when the sample is contaminated with a moderate-to-large number of outliers that have noteworthy contamination strengths. Overall, our results show that making use of a robust method when assessing multivariate risk leads to more accurate estimates, particularly when combined with relevant signing information. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
本文提出了一种检测多变量异常值的方法，该方法结合了鲁棒估计方法和签名检测信息。我们的方法使用马氏距离从相对于协方差矩阵的期望值来量化每个观测值的极值，并且我们利用鲁棒估计工具，即最小协方差行列式，来估计马氏距离计算中使用的平均向量和协方差矩阵。此外，我们在距离计算中加入了一个签名元素，使研究人员能够更好地控制在搜索异常值时应该优先考虑的多元空间的特定区域，从而允许更有针对性的风险评估和分类。最后，我们将鲁棒元素和签名元素统一到一个框架中，该框架可用于双线性模型，如主成分分析和因子分析。通过模拟和真实数据示例，我们证明了所提出的方法可以改进风险评估和异常值检测，特别是当样本被具有显著污染强度的中等到大量异常值污染时。总体而言，我们的研究结果表明，在评估多变量风险时使用稳健的方法可以获得更准确的估计，特别是在与相关签名信息相结合时。（PsycInfo Database Record (c) 2025 APA，版权所有）。
{"title":"Robust detection of signed outliers in multivariate data with applications to early identification of risk for autism.","authors":"Jesus E Delgado,Jed T Elison,Nathaniel E Helwig","doi":"10.1037/met0000775","DOIUrl":"https://doi.org/10.1037/met0000775","url":null,"abstract":"This article proposes an approach for detecting multivariate outliers that combines robust estimation methods with signed detection information. Our method uses the Mahalanobis distance to quantify each observation's extremeness from the expected value relative to the covariance matrix, and we leverage robust estimation tools, i.e., the minimum covariance determinant, to estimate the mean vector and covariance matrix used in the Mahalanobis distance calculation. Furthermore, we incorporate a signing element into the distance calculation to give researchers greater control over the specific regions of multivariate space that should be prioritized when searching for outliers, which allows for more targeted risk assessment and classification. Lastly, we unify the robust and signed elements into a framework that can be used within bilinear models such as principal components analysis and factor analysis. Using simulated and real data examples, we demonstrate that the proposed approach can result in improved risk assessment and outlier detection, particularly when the sample is contaminated with a moderate-to-large number of outliers that have noteworthy contamination strengths. Overall, our results show that making use of a robust method when assessing multivariate risk leads to more accurate estimates, particularly when combined with relevant signing information. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"26 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144669552","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Adapting methods for correcting selective reporting bias in meta-analysis of dependent effect sizes.","authors":"Man Chen, James E. Pustejovsky","doi":"10.1037/met0000773","DOIUrl":"https://doi.org/10.1037/met0000773","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"107 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144602945","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Generative pretrained transformer models can function as highly reliable second screeners of titles and abstracts in systematic reviews: A proof of concept and common guidelines.","authors":"Mikkel Helding Vembye, Julian Christensen, Anja Bondebjerg Mølgaard, Frederikke Lykke Witthöft Schytt","doi":"10.1037/met0000769","DOIUrl":"https://doi.org/10.1037/met0000769","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"11 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144602946","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Reevaluating R²med as an effect size measure for indirect effects.","authors":"Sang-June Park, Youjae Yi","doi":"10.1037/met0000771","DOIUrl":"https://doi.org/10.1037/met0000771","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"35 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144602944","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for Adapting Methods for Correcting Selective Reporting Bias in Meta-Analysis of Dependent Effect Sizes","authors":"","doi":"10.1037/met0000773.supp","DOIUrl":"https://doi.org/10.1037/met0000773.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"6 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144565719","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号