Trust the “Process”? When Fundamental Motor Skill Scores are Reliably Unreliable

IF 1.7 4区教育学 Q2 EDUCATION & EDUCATIONAL RESEARCH Measurement in Physical Education and Exercise Science Pub Date : 2023-04-09 DOI:10.1080/1091367X.2023.2199126

Ryan M. Hulteen, L. True, Edward Kroc

{"title":"Trust the “Process”? When Fundamental Motor Skill Scores are Reliably Unreliable","authors":"Ryan M. Hulteen, L. True, Edward Kroc","doi":"10.1080/1091367X.2023.2199126","DOIUrl":null,"url":null,"abstract":"ABSTRACT The typical process for assessing inter-rater reliability is facilitated by training raters within a research team. Lacking is an understanding if inter-rater reliability scores between research teams demonstrate adequate reliability. This study examined inter-rater reliability between 16 researchers who assessed fundamental motor skills using the Test of Gross Motor Development-3rd edition. Total score agreement (ICC = 0.363) and locomotor subscale agreement (ICC = 0.383) were “very poor,” while ball skills subscale agreement (ICC = 0.478) was “poor.” Consistencies of total (ICC = 0.757), locomotor (ICC = 0.730), and ball skills (ICC = 0.746) scores were “fair.” Component percentage agreement ranged from 40.5% to 96.2%. These data suggest that there are significant differences in how different research groups evaluate fundamental motor skills based on the subjective nature of scoring. Consistency and agreement among users need to be addressed in motor development research to allow for direct comparisons across studies that use process-oriented measures.","PeriodicalId":48577,"journal":{"name":"Measurement in Physical Education and Exercise Science","volume":"27 1","pages":"391 - 402"},"PeriodicalIF":1.7000,"publicationDate":"2023-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Measurement in Physical Education and Exercise Science","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1080/1091367X.2023.2199126","RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 2

Abstract

ABSTRACT The typical process for assessing inter-rater reliability is facilitated by training raters within a research team. Lacking is an understanding if inter-rater reliability scores between research teams demonstrate adequate reliability. This study examined inter-rater reliability between 16 researchers who assessed fundamental motor skills using the Test of Gross Motor Development-3rd edition. Total score agreement (ICC = 0.363) and locomotor subscale agreement (ICC = 0.383) were “very poor,” while ball skills subscale agreement (ICC = 0.478) was “poor.” Consistencies of total (ICC = 0.757), locomotor (ICC = 0.730), and ball skills (ICC = 0.746) scores were “fair.” Component percentage agreement ranged from 40.5% to 96.2%. These data suggest that there are significant differences in how different research groups evaluate fundamental motor skills based on the subjective nature of scoring. Consistency and agreement among users need to be addressed in motor development research to allow for direct comparisons across studies that use process-oriented measures.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

信任“流程”？当基本运动技能分数不可靠时

摘要评估评分者间可靠性的典型过程是通过在研究团队中培训评分者来促进的。缺乏对研究团队之间评分者间可靠性得分是否证明足够可靠性的理解。这项研究调查了16名研究人员之间的评分者间可靠性，这些研究人员使用第三版《总运动发育测试》评估了基本运动技能。总分协议（ICC = 0.363）和运动分量表协议（ICC = 0.383）“非常差”，而球技分量表一致性（ICC = 0.478）为“差” = 0.757），运动（ICC = 0.730）和球技（ICC = 0.746）的得分是“公平的”。成分百分比一致性在40.5%到96.2%之间。这些数据表明，不同研究小组根据得分的主观性质评估基本运动技能的方式存在显著差异。在运动发展研究中，需要解决用户之间的一致性和一致性问题，以便在使用过程导向措施的研究之间进行直接比较。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Measurement in Physical Education and Exercise Science Medicine-Orthopedics and Sports Medicine

CiteScore

4.20

自引率

33.30%

发文量

期刊介绍： The scope of Measurement in Physical Education and Exercise Science (MPEES) covers original measurement research, special issues, and tutorials within six substantive disciplines of physical education and exercise science. Six of the seven sections of MPEES define the substantive disciplines within the purview of the original research to be published in the journal: Exercise Science, Physical Activity, Physical Education Pedagogy, Psychology, Research Methodology and Statistics, and Sport Management and Administration. The seventh section of MPEES, Tutorial and Teacher’s Toolbox, serves to provide an outlet for review and/or didactic manuscripts to be published in the journal. Special issues provide an avenue for a coherent set of manuscripts (e.g., four to five) to collectively focus in-depth on an important and timely measurement-related issue within the scope of MPEES. The primary aim of MPEES is to publish high-impact manuscripts, most of which will focus on original research, that fit within the scope of the journal.