Is Automated Machine Learning useful for ocular toxoplasmosis identification and classification of the inflammatory activity?

Purpose

To evaluate the performance of Automated Machine Learning (AutoML) models in diagnosing ocular toxoplasmosis (OT) and classifying its inflammatory activity from fundus photographs.

Design

Cross-sectional study.

Methods

Fundus photographs from OT patients in two Colombian referral centers and an open-source OT database were classified into active OT, inactive OT, and no OT. Image quality assessment excluded images with artifacts but included blurry images due to vitritis. Photos were uploaded to Amazon Web Services S3 and Google Cloud Bucket. Two models were developed on each platform: a binary model (active/inactive OT vs. no OT) and a multiclass model (active OT, inactive OT, and no OT). Datasets were split into 70% for training, 20% for testing, and 10% for validation. Sensitivity, specificity, precision, accuracy, F1-score, the area under the precision-recall curve (AUPRC), and Cohen's Kappa were calculated for each platform and model. An external validation using an open-source image bank was performed.

Results

The binary model on AWS showed a sensitivity of 0.97, specificity of 0.98, and AUPRC of 1.00, while the Google Cloud binary model had a sensitivity of 0.82, specificity of 0.91, and AUPRC of 0.91. The multiclass model on AWS achieved an F1 score of 0.88, with Cohen's Kappa of 0.81, while the Google Cloud model reached an F1 score of 0.88, with Cohen's Kappa of 0.82. External validation for Google Cloud achieved an accuracy of 87.5% and 80.3% in the binary and multiclass models, respectively.

Conclusions

AutoML is a powerful tool for diagnosing OT and classifying inflammatory activity, potentially guiding diagnosis and treatment decisions.