{"title":"An Application of Theril Indexes for the Interrater Reliability: A Comparison with Intraclass Correlations","authors":"Tianshu Pan, Yue Yin","doi":"10.59863/wddk7257","DOIUrl":null,"url":null,"abstract":"This study proposes to apply the Theil-index ratios for the interrater reliability. We discuss the theoretical foundations and examine its function empirically using real data. Our analyses show that Theil-index rations and intraclass correlation (ICC) estimates are highly correlated. However, ICC may underestimate the interrater reliability by some extreme disagreement among raters and be more likely to be influenced by the extreme disagreement. As Theil-index ratios overcome the limitations of ICC to some degree, it seems that Theil-index ratios provide an alternative to evaluating interrater reliability, at least under certain conditions, e.g., when outliers exist in the data, it is difficult to obtain the variance component estimates, or ICC underestimates interrater reliability.","PeriodicalId":72586,"journal":{"name":"Chinese/English journal of educational measurement and evaluation","volume":"130 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2023-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Chinese/English journal of educational measurement and evaluation","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.59863/wddk7257","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
This study proposes to apply the Theil-index ratios for the interrater reliability. We discuss the theoretical foundations and examine its function empirically using real data. Our analyses show that Theil-index rations and intraclass correlation (ICC) estimates are highly correlated. However, ICC may underestimate the interrater reliability by some extreme disagreement among raters and be more likely to be influenced by the extreme disagreement. As Theil-index ratios overcome the limitations of ICC to some degree, it seems that Theil-index ratios provide an alternative to evaluating interrater reliability, at least under certain conditions, e.g., when outliers exist in the data, it is difficult to obtain the variance component estimates, or ICC underestimates interrater reliability.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
