Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.

IF 3.3 2区 心理学 Q1 PSYCHOLOGY, CLINICAL Psychological Assessment Pub Date : 2024-04-01 Epub Date: 2024-02-29 DOI:10.1037/pas0001297
Yana Suchy, Michelle Gereau Mora, Libby A DesRuisseaux, Madison A Niermeyer, Stacey Lipio Brothers
{"title":"Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.","authors":"Yana Suchy, Michelle Gereau Mora, Libby A DesRuisseaux, Madison A Niermeyer, Stacey Lipio Brothers","doi":"10.1037/pas0001297","DOIUrl":null,"url":null,"abstract":"<p><p>The term \"ecological validity\" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (<i>n</i> = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of \"naturalistic\" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).</p>","PeriodicalId":20770,"journal":{"name":"Psychological Assessment","volume":null,"pages":null},"PeriodicalIF":3.3000,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Psychological Assessment","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1037/pas0001297","RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/2/29 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"PSYCHOLOGY, CLINICAL","Score":null,"Total":0}
引用次数: 0

Abstract

The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
新型执行功能测试生态有效性研究中的陷阱:系统回顾与行动呼吁。
传统上,"生态效度"(EV)一词指的是测验分数预测真实世界功能的能力。然而,测验与真实世界任务的相似性有时会被误认为是其预测日常生活能力的证据,有时会绕过严格的验证研究。本系统性综述的目的是研究用于支持新型执行功能(EF)面效测试EV声称的证据的类型和质量。我们使用以下术语对 MEDLINE 和 PsychINFO 数据库进行了检索:生态学*、神经心理学*和(执行功能*或执行功能障碍或执行能力*)。结果发现,有 32 篇文章明确指出,其结果显示了新型面效 EF 测试的 EV。结果表明,只有 60% 的研究以测试分数预测功能结果的能力作为 EV 的依据,其余 40% 的研究则依赖于其他证据(如与其他测量方法的相关性、参与者反馈、群体差异)。在根据测试分数预测结果的能力得出结论的研究中(n = 19),绝大多数都依赖于行为评分量表,使用的样本量和参与者与变量的比例较小,并且未能控制共变量和多重比较。在 "自然 "测试研究中,科学严谨性不足的问题尤为突出。本系统综述揭示了绝大多数声称支持新型面效EF测试EV的研究在概念、方法和统计上的重大缺陷。我们呼吁作者、审稿人和编辑保障该领域研究的科学严谨性。(PsycInfo Database Record (c) 2024 APA,保留所有权利)。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Psychological Assessment
Psychological Assessment PSYCHOLOGY, CLINICAL-
CiteScore
5.70
自引率
5.60%
发文量
167
期刊介绍: Psychological Assessment is concerned mainly with empirical research on measurement and evaluation relevant to the broad field of clinical psychology. Submissions are welcome in the areas of assessment processes and methods. Included are - clinical judgment and the application of decision-making models - paradigms derived from basic psychological research in cognition, personality–social psychology, and biological psychology - development, validation, and application of assessment instruments, observational methods, and interviews
期刊最新文献
Development and validation of a method for deriving MMPI-3 scores from MMPI-2/MMPI-2-RF item responses. Evaluation of the Multidimensional Personality Questionnaire (MPQ) Unlikely Virtues Scale in the detection of underreporting. Prospectively predicting violent and aggressive incidents in prison practice with the Risk Screener Violence (RS-V): Results from a multisite prison study. Development of the Food Addiction Symptom Inventory: The first clinical interview to assess ultra-processed food addiction. Does the Bayley-4 measure the same constructs across girls and boys and infants, toddlers, and preschoolers?
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1