A Randomized Controlled Crossover Trial to Evaluate the Efficacy of AI-Assisted Heart Contouring

Abstract

Purpose/Objective(s)

Artificial intelligence (AI) normal tissue contouring tools are now widely available and used in many treatment planning systems. However, few studies have prospectively demonstrated the value of such tools when employed in clinical workflows. We conducted a randomized controlled trial to assess the benefit of using an AI heart contouring algorithm to assist breast radiotherapy (RT) planning.

Materials/Methods

A single institution, 2-arm randomized controlled crossover trial was undertaken between December 2021 and October 2023. A convolutional neural network for heart auto-contouring, which had median Dice 0.95 compared to gold standard contours and reduced contouring time by 50% in pre-clinical studies, was implemented as a scripted tool within a treatment planning system. Eligible patients had breast cancer planned for RT, with hearts contoured according to the randomization of the dosimetrist assigned to their plan. Dosimetrists were stratified by experience and randomized to (1) manual heart contouring first or (2) AI-assisted heart contouring first. AI-assisted contouring consisted of running the model within the treatment planning system and editing the contour as needed. After completing 5 cases on the initial contour strategy, dosimetrists crossed to the other strategy. Co-primary outcomes were feasibility and efficacy, defined as heart contouring time (measured by recording screen during contouring). Secondary endpoints included dosimetrist and treating physician contour assessments. The trial had 90% power to detect a 30% reduction in contour time with 2-sided type I error of 5%.

Results

One hundred eighteen patients enrolled; 60 patients’ hearts were contoured manually and 58 with AI assistance. Eleven dosimetrists enrolled; 5 were randomized to manual first arm and 6 to AI-assisted first arm. There was no difference in manual vs. AI-assisted contour time overall (mean 277.3 ± 151.2 vs. 267.2 ± 199.0 seconds, P = 0.76), on the manual first arm only (P = 0.15), or on the AI first arm only (P = 0.67). There was no difference in contour time among only dosimetrists with ⩽2 yr experience (mean = 374.4 ± 174.2 vs. 403.3 ± 275.9 secs, P = 0.72), nor among dosimetrists with > 3 yr experience (mean = 241.9 ± 126.4 vs. 210.8 ± 121.9 secs, P = 0.25). Dosimetrists considered AI contours acceptable with minor/no modification in 13 of 47 (27.6%) cases and unacceptable in 34 of 47 (72.4%) cases; but considered the AI helpful in 35 of 47 (74.5%) cases and to improve subjective efficiency in 29 of 47 (61.7%) cases. Physicians, blinded to randomization, thought contours presented for review were unacceptable in 6 of 56 (10.7%) AI-assisted and 3 of 56 (5.4%) manual cases.

Conclusion

Despite improving efficiency pre-clinically, AI assistance did not reduce heart contour time compared to standard manual contouring. Physicians were more likely to find contours unacceptable when AI-assistance was used. Pre-clinical findings may not translate into clinical benefits, emphasizing the critical need to evaluate new AI technologies under their intended use in real clinical workflows.