An inter-rater variability study between human and automatic scorers in 5-s mini-epochs of sleep

IF 3.4 2区医学 Q1 CLINICAL NEUROLOGY Sleep medicine Pub Date : 2025-04-01 Epub Date: 2025-02-06 DOI:10.1016/j.sleep.2025.02.005

Louise Frøstrup Follin , Alexander Neergaard Zahid , Rannveig Viste , Janita Vevelstad , Tobias Kaufmann , Anette Ramm-Pettersen , Hilde T. Juvodden , Berit Hjelde Hansen , Julie Anja Engelhard Christensen , Stine Knudsen-Heier

{"title":"An inter-rater variability study between human and automatic scorers in 5-s mini-epochs of sleep","authors":"Louise Frøstrup Follin , Alexander Neergaard Zahid , Rannveig Viste , Janita Vevelstad , Tobias Kaufmann , Anette Ramm-Pettersen , Hilde T. Juvodden , Berit Hjelde Hansen , Julie Anja Engelhard Christensen , Stine Knudsen-Heier","doi":"10.1016/j.sleep.2025.02.005","DOIUrl":null,"url":null,"abstract":"<div><h3>Study objective</h3><div>Sleep is traditionally scored using 30-s epochs of polysomnographies. As sleep is physiologically dynamic and 30-s epochs may conceal important characteristics, we aim to challenge this standard by scoring sleep in 5-s mini-epochs and analyzing inter-rater variability between human and automatic scorers.</div></div><div><h3>Methods</h3><div>In 40 polysomnography recordings, 120 mini-epochs per polysomnography were scored manually by three human experts (expert1_5s, expert2_5s and expert3_5s) and automatically by a validated sleep classifier (USleep_5s). Additionally, 5-s mini-epochs (clinical_5s) extracted from conventional human-scored 30-s epochs were considered. We assessed inter-rater variability and stage shifting in epochs and mini-epochs and further in narcolepsy type 1 (NT1) patients and siblings.</div></div><div><h3>Results</h3><div>Agreement for mini-epochs was κ = 0.50 ± 0.11 (expert1_5s vs clinical_5s) and κ = 0.51 ± 0.12, (expert1_5s vs USleep_5s). Between human experts, agreement was κ = 0.51 ± 0.16 (expert1_5s vs expert2_5s), and κ = 0.57 ± 0.11 (expert1_5s vs expert3_5s). Stage shift percentages were significantly higher in mini-epochs scored by expert1_5s (27.75 %) and USleep_5s (22.88 %) than corresponding conventional epochs (5.12 %), with no significant difference between NT1 patients and siblings.</div></div><div><h3>Conclusion</h3><div>While mini-epoch scoring agreement was generally high, it was still lower than within epochs, likely due to a lack of standard mini-epoch scoring procedure and the automatic classifier being trained on epochs. However, stage discrepancies between epochs and mini-epochs and increased stage shifting in mini-epochs support that epochs can contain several stages, and that mini-epochs could supplement more detailed sleep characterization potentially enabling more precise diagnosis and finding new polysomnographic biomarkers. Future studies should include larger datasets to refine mini-epoch scoring rules and exploit automatic classifiers e.g. via transfer learning.</div></div>","PeriodicalId":21874,"journal":{"name":"Sleep medicine","volume":"128 ","pages":"Pages 139-150"},"PeriodicalIF":3.4000,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sleep medicine","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1389945725000504","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/2/6 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Study objective

Sleep is traditionally scored using 30-s epochs of polysomnographies. As sleep is physiologically dynamic and 30-s epochs may conceal important characteristics, we aim to challenge this standard by scoring sleep in 5-s mini-epochs and analyzing inter-rater variability between human and automatic scorers.

Methods

In 40 polysomnography recordings, 120 mini-epochs per polysomnography were scored manually by three human experts (expert1_5s, expert2_5s and expert3_5s) and automatically by a validated sleep classifier (USleep_5s). Additionally, 5-s mini-epochs (clinical_5s) extracted from conventional human-scored 30-s epochs were considered. We assessed inter-rater variability and stage shifting in epochs and mini-epochs and further in narcolepsy type 1 (NT1) patients and siblings.

Results

Agreement for mini-epochs was κ = 0.50 ± 0.11 (expert1_5s vs clinical_5s) and κ = 0.51 ± 0.12, (expert1_5s vs USleep_5s). Between human experts, agreement was κ = 0.51 ± 0.16 (expert1_5s vs expert2_5s), and κ = 0.57 ± 0.11 (expert1_5s vs expert3_5s). Stage shift percentages were significantly higher in mini-epochs scored by expert1_5s (27.75 %) and USleep_5s (22.88 %) than corresponding conventional epochs (5.12 %), with no significant difference between NT1 patients and siblings.

Conclusion

While mini-epoch scoring agreement was generally high, it was still lower than within epochs, likely due to a lack of standard mini-epoch scoring procedure and the automatic classifier being trained on epochs. However, stage discrepancies between epochs and mini-epochs and increased stage shifting in mini-epochs support that epochs can contain several stages, and that mini-epochs could supplement more detailed sleep characterization potentially enabling more precise diagnosis and finding new polysomnographic biomarkers. Future studies should include larger datasets to refine mini-epoch scoring rules and exploit automatic classifiers e.g. via transfer learning.

Abstract Image

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

人类和自动评分者在5秒睡眠小时段的评分间变异性研究

研究目标传统上使用30秒的多导睡眠描记仪对睡眠进行评分。由于睡眠是生理动态的，30秒的睡眠时间可能隐藏了重要的特征，我们的目标是挑战这一标准，以5秒的小睡眠时间进行评分，并分析人类和自动评分者之间的差异。方法对40个多导睡眠图记录，每个多导睡眠图分别由3位专家（expert1_5、expert2_5和expert3_5）和经过验证的睡眠分类器（usleep_5）手动评分120个小epoch。此外，还考虑了从常规人类评分的30-s epoch中提取的5-s mini-epoch （clinical_5s）。我们评估了发作性睡病1型（NT1）患者及其兄弟姐妹的期间变异性和分期转移。结果κ = 0.50±0.11 (expert1_5 vs clinical_5), κ = 0.51±0.12 （expert1_5 vs usleep_5）。人类专家之间的一致性为κ = 0.51±0.16 (expert1_5 vs expert2_5), κ = 0.57±0.11 （expert1_5 vs expert3_5）。用expert1_5和usleep_5评分的小分期（27.75%）和usleep_5评分的分期转移率（22.88%）明显高于相应的常规分期（5.12%），NT1患者和兄弟姐妹之间无显著差异。结论小epoch评分一致性总体较高，但仍低于epoch内评分一致性，这可能是由于缺乏标准的小epoch评分程序以及自动分类器是在epoch上训练的。然而，epoch和mini-epoch之间的阶段差异以及mini-epoch中阶段变化的增加支持epoch可以包含多个阶段，并且mini-epoch可以补充更详细的睡眠特征，从而可能实现更精确的诊断和发现新的多导睡眠图生物标志物。未来的研究应该包括更大的数据集，以改进小epoch评分规则，并利用自动分类器，例如通过迁移学习。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Sleep medicine 医学-临床神经学

CiteScore

8.40

自引率

6.20%

发文量

1060

审稿时长

49 days

期刊介绍： Sleep Medicine aims to be a journal no one involved in clinical sleep medicine can do without. A journal primarily focussing on the human aspects of sleep, integrating the various disciplines that are involved in sleep medicine: neurology, clinical neurophysiology, internal medicine (particularly pulmonology and cardiology), psychology, psychiatry, sleep technology, pediatrics, neurosurgery, otorhinolaryngology, and dentistry. The journal publishes the following types of articles: Reviews (also intended as a way to bridge the gap between basic sleep research and clinical relevance); Original Research Articles; Full-length articles; Brief communications; Controversies; Case reports; Letters to the Editor; Journal search and commentaries; Book reviews; Meeting announcements; Listing of relevant organisations plus web sites.