Artificial Intelligence-Supported Development of Health Guideline Questions.

Abstract

Background: Guideline questions are typically proposed by experts.

Objective: To assess how large language models (LLMs) can support the development of guideline questions, providing insights on approaches and lessons learned.

Design: Two approaches for guideline question generation were assessed: 1) identification of questions conveyed by online search queries and 2) direct generation of guideline questions by LLMs. For the former, the researchers retrieved popular queries on allergic rhinitis using Google Trends (GT) and identified those conveying questions using both manual and LLM-based methods. They then manually structured as guideline questions the queries that conveyed relevant questions. For the second approach, they tasked an LLM with proposing guideline questions, assuming the role of either a patient or a clinician.

Setting: Allergic Rhinitis and its Impact on Asthma (ARIA) 2024 guidelines.

Participants: None.

Measurements: Frequency of relevant questions generated.

Results: The authors retrieved 3975 unique queries using GT. From these, they identified 37 questions, of which 22 had not been previously posed by guideline panel members and 2 were eventually prioritized by the panel. Direct interactions with LLMs resulted in the generation of 22 unique relevant questions (11 not previously suggested by panel members), and 4 were eventually prioritized by the panel. In total, 6 of 39 final questions prioritized for the 2024 ARIA guidelines were not initially thought of by the panel. The researchers provide a set of practical insights on the implementation of their approaches based on the lessons learned.

Limitation: Single case study (ARIA guidelines).

Conclusion: Approaches using LLMs can support the development of guideline questions, complementing traditional methods and potentially augmenting questions prioritized by guideline panels.

Primary funding source: Fraunhofer Cluster of Excellence for Immune-Mediated Diseases.