Patient- and fraction-specific magnetic resonance volume reconstruction from orthogonal images with generative adversarial networks

Abstract

Background

Although deep learning (DL) methods for reconstructing 3D magnetic resonance (MR) volumes from 2D MR images yield promising results, they require large amounts of training data to perform effectively. To overcome this challenge, fine-tuning—a transfer learning technique particularly effective for small datasets—presents a robust solution for developing personalized DL models.

Purpose

A 2D to 3D conditional generative adversarial network (GAN) model with a patient- and fraction-specific fine-tuning workflow was developed to reconstruct synthetic 3D MR volumes using orthogonal 2D MR images for online dose adaptation.

Methods

A total of 2473 3D MR volumes were collected from 43 patients. The training and test datasets were separated into 34 and 9 patients, respectively. All patients underwent MR-guided adaptive radiotherapy using the same imaging protocol. The population data contained 2047 3D MR volumes from the training dataset. Population data were used to train the population-based GAN model. For each fraction of the remaining patients, the population model was fine-tuned with the 3D MR volumes acquired before beam irradiation of the fraction, named the fine-tuned model. The performance of the fine-tuned model was tested using the 3D MR volume acquired immediately after the beam delivery of the fraction. The model's input was a pair of axial and sagittal MR images at the isocenter level, and the output was a 3D MR volume. Model performance was evaluated using the structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), root mean square error (RMSE), and mean absolute error (MAE). Moreover, the prostate, bladder, and rectum in the predicted MR images were manually segmented. To assess geometric accuracy, the 2D Dice Similarity Coefficient (DSC) and 2D Hausdorff Distance (HD) were calculated.

Results

A total of 84 3D MR volumes were included in the performance testing. The mean ± standard deviation (SD) of SSIM, PSNR, RMSE, and MAE were 0.64 ± 0.10, 93.9 ± 1.5 dB, 0.050 ± 0.009, and 0.036 ± 0.007 for the population model and 0.72 ± 0.09, 96.2 ± 1.8 dB, 0.041 ± 0.007, and 0.028 ± 0.006 for the fine-tuned model, respectively. The image quality of the fine-tuned model was significantly better than that of the population model (p < 0.05). The mean ± SD of DSC and HD of the population model were 0.79 ± 0.08 and 1.70 ± 2.35 mm for prostate, 0.81 ± 0.10 and 2.75 ± 1.53 mm for bladder, and 0.72 ± 0.08 and 1.93 ± 0.59 mm for rectum. Contrarily, the mean ± SD of DSC and HD of the fine-tuned model were 0.83 ± 0.06 and 1.29 ± 0.77 mm for prostate, 0.85 ± 0.07 and 2.16 ± 1.09 mm for bladder, and 0.77 ± 0.08 and 1.57 ± 0.52 mm for rectum. The geometric accuracy of the fine-tuned model was significantly improved than that of the population model (p < 0.05).

Conclusion

By employing a patient- and fraction-specific fine-tuning approach, the GAN model demonstrated promising accuracy despite limited data availability.