Proof-of-Concept Prompted Large Language Model for Radiology Procedure Request Routing

Abstract

Purpose

To measure the accuracy and cost of a proof-of-concept prompted large language model (LLM) to route procedure requests to the appropriate phone number or pager at a single large academic hospital.

Materials and Methods

At a large academic hospital, existing teams, pager/phone numbers, and schedules were used to create text-based rules for procedure requests. A prompted LLM was created to route procedure requests at specific days and times to the appropriate teams. The prompted LLM was tested on 250 “in-scope” requests (explicitly defined by provided rules) and 25 “out-of-scope” requests using generative pretrained transformer (GPT)–3.5-turbo and GPT-4 models from OpenAI and 4 open-weight models.

Results

The prompted LLM correctly routed 96.4% of in-scope and 76.0% of out-of-scope requests using GPT-4, which outperformed all other models (P < .001). All models demonstrated worse performance for requests during evening and weekend hours (P < .001). OpenAI application programming interface costs were approximately $0.03 per request for GPT-4 and $0.0006 per request for GPT-3.5-turbo.

Conclusions

This study demonstrates the accuracy of low-cost prompted LLMs to appropriately route procedure requests in a large academic hospital system. A similar approach may be used to help clinicians navigate a radiology phone tree or as a tool to help reading room coordinators route requests effectively.