Detection and characterization of pancreatic lesion with artificial intelligence: The SFR 2023 artificial intelligence data challenge

Abstract

Purpose

The purpose of the 2023 SFR data challenge was to invite researchers to develop artificial intelligence (AI) models to identify the presence of a pancreatic mass and distinguish between benign and malignant pancreatic masses on abdominal computed tomography (CT) examinations.

Materials and methods

Anonymized abdominal CT examinations acquired during the portal venous phase were collected from 18 French centers. Abdominal CT examinations were divided into three groups including CT examinations with no lesion, CT examinations with benign pancreatic mass, or CT examinations with malignant pancreatic mass. Each team included at least one radiologist, one data scientist, and one engineer. Pancreatic lesions were annotated by expert radiologists. CT examinations were distributed in balanced batches via a Health Data Hosting certified platform. Data were distributed into four batches, two for training, one for internal evaluation, and one for the external evaluation. Training used 83 % of the data from 14 centers and external evaluation used data from the other four centers. The metric (i.e., final score) used to rank the participants was a weighted average of mean sensitivity, mean precision and mean area under the curve.

Results

A total of 1037 abdominal CT examinations were divided into two training sets (including 500 and 232 CT examinations), an internal evaluation set (including 139 CT examinations), and an external evaluation set (including 166 CT examinations). The training sets were distributed on September 7 and October 13, 2023, and evaluation sets on October 15, 2023. Ten teams with a total of 93 members participated to the data challenge, with the best final score being 0.72.

Conclusion

This SFR 2023 data challenge based on multicenter CT data suggests that the use of AI for pancreatic lesions detection is possible on real data, but the distinction between benign and malignant pancreatic lesions remains challenging.