Assessing the transportability of radiomic models for lung cancer diagnosis: commercial vs. open-source feature extractors.

Abstract

Background: Radiomics has shown promise in improving malignancy risk stratification of indeterminate pulmonary nodules (IPNs) with many platforms available, but with no head-to-head comparisons. This study aimed to evaluate transportability of radiomic models across platforms by comparing performances of a commercial radiomic feature extractor (HealthMyne) with an open-source extractor (PyRadiomics) on diagnosis of lung cancer in IPNs.

Methods: A commercial radiomic feature extractor was used to segment IPNs from computed tomography (CT) scans, and a previously validated radiomic model based on commercial features was used as baseline (ComRad). Using same segmentation masks, PyRadiomics, an open-source feature extractor was used to build three open-source radiomic models (OpenRad) using different methods: de novo open-source model derived using least absolute shrinkage and selection operator (LASSO) for feature selection, selecting open-source features matched to ComRad features based upon Imaging Biomarker Standardization Initiative (IBSI) nomenclature, and selecting open-source features most highly correlated to ComRad features. Radiomic models were trained on an internal cohort (n=161) and externally validated on 3 cohorts (n=278). We added Mayo clinical risk score to OpenRad and ComRad models, creating integrated clinical radiomic (ClinRad) models. All models were compared using area under the curve (AUC) and evaluated for clinical improvement using bias-corrected clinical net reclassification indices (cNRIs).

Results: ComRad AUC was 0.76 [95% confidence interval (CI): 0.71-0.82], and OpenRad AUC was 0.75 (95% CI: 0.69-0.81) for LASSO model, 0.74 (95% CI: 0.68-0.79) for Spearman's correlation, and 0.71 (95% CI: 0.65-0.77) for IBSI. Mayo scores were added to OpenRad LASSO model, which performed best, forming open-source ClinRad model with AUC of 0.80 (95% CI: 0.74-0.86), identical to commercial ClinRad's AUC. Both ClinRad models showed clinical improvement compared to Mayo alone, with commercial ClinRad achieving cNRI of 0.09 (95% CI: 0.02-0.15) for benign and 0.07 (95% CI: 0.00-0.13) for malignant, and open-source ClinRad achieving cNRI of 0.09 (95% CI: 0.02-0.15) for benign and 0.06 (95% CI: 0.00-0.12) for malignant.

Conclusions: Transportability of radiomic models across platforms directly does not conserve performance, but radiomic platforms can provide equivalent results when building de novo models allowing for flexibility in feature selection to maximize prediction accuracy.