[Diagnostic performance evaluation of artificial intelligence-assisted diagnostic systems in cervical cytopathological examination].

Abstract

Objective: To evaluate the diagnostic performance of artificial intelligence-assisted diagnostic systems in cervical cytopathological examination. Methods: Cervical cytology slide data were retrospectively collected from four hospitals for the external validation of the developed artificial intelligence-assisted diagnostic system. Subsequently, prospective data collection was conducted for human-machine assisted studies. Results: In the retrospective study, a total of 3 162 valid samples were collected as external validation data. The system showed an area under the curve (AUC) of 0.890 (95%CI: 0.878-0.902), accuracy of 0.885 (95%CI: 0.873-0.896), sensitivity of 0.928 (95%CI: 0.914-0.941), and specificity of 0.852 (95%CI: 0.834-0.867). In the prospective study, 212 valid samples were collected, and five junior cytologists participated in the human-machine assisted study. Without artificial intelligence assistance, the average AUC for the five cytologists was 0.686 (95%CI: 0.650-0.722), the accuracy was 0.699 (95%CI: 0.671-0.727), the sensitivity was 0.653 (95%CI: 0.599-0.703), the specificity was 0.719 (95%CI: 0.685-0.750), the Fleiss κ value was 0.510, and the reading time was 223 seconds. With artificial intelligence assistance, the AUC, accuracy, sensitivity, and specificity increased by 0.166, 0.143, 0.225, and 0.107, respectively. Additionally, Fleiss κ was 0.730 and the reading time decreased by 188 seconds. All differences were statistically significant (all P<0.001). Conclusions: Artificial intelligence-assisted diagnosis system shows excellent performance and good generalizability, significantly improving the diagnostic accuracy, consistency, and efficiency of junior cytologists. It can be an effective auxiliary tool for junior cytologists in clinical practice.