Inter-reader reliability and diagnostic accuracy of PI-RADS scoring between academic and community care networks: How wide is the gap?

Importance: The Prostate Imaging Reporting & Data System (PI-RADS) scoring guidelines were developed to address the substantial variation in interpretation and reporting of prostate cancer (PCa) multiparametric MRI (mpMRI) results, and subsequent updates have sought to further improve inter-reader reliability. Nonetheless, the variability of PI-RADS scoring in real-world settings may represent a continuing challenge to the widespread standardization of prostate mpMRI and limit its overall clinical benefit.

Objective: To assess variability in mpMRI interpretation and reporting of PCa, we evaluated the discrepancies in PI-RADS scoring between community practices and a tertiary academic care center.

Design, setting, and participants: We identified 262 mpMRI studies from nonacademic facilities, reinterpreted by radiologists at our institution between January 2016 and July 2022. Results of targeted MRI fusion biopsy were identified for 193 of these patients, totaling 302 lesions. PI-RADS scoring from both community and academic interpreters were recorded in addition to presence of clinically significant PCa (csPCa) on pathological analysis of targeted cores.

Main outcome and measures: The primary outcome was inter-reader reliability via intraclass correlation (ICC) and the kappa statistic. We also assessed the diagnostic accuracy of PI-RADS scoring for detecting csPCa for both cohorts via receiver operator characteristics (ROC) analysis and compared these findings using paired-sample area difference under curve analysis.

Results: Inter-reader agreement and reliability of PI-RADS scoring per lesion was generally poor (absolute agreement ICC = 0.393, 95% CI: 0.288-0.488; consistency ICC = 0.407, 95% CI: 0.308-0.497; kappa = 0.336, 95% CI: 0.267-0.406). Reliability results from studies obtained after the publication of PI-RADSv2.1 were similar to those of the overall analysis. No agreement was observed in the subgroup of lesions scored as PIRADS 3 by community interpreters. No statistically significant difference in diagnostic accuracy was observed between cohorts (ROC area under curve [AUC]: 0.759 vs. 0.785, respectively; P = 0.337). PI-RADS 3 was determined to be the optimal cutoff for detecting clinically significant disease in both cohorts.

Conclusions and relevance: Our results suggest that mpMRI diagnostic accuracy for detecting csPCa is not significantly different between academic and community practices. However, significantly poor reliability of mpMRI was observed between cohorts, suggesting the risk of introducing practice variation for community prostate cancer management. Variability, particularly for PI-RADS 3 lesions, can lead to inconsistent biopsy recommendations, which may result in missed csPCa or unnecessary biopsies.