63. An introduction to publicly available AI-assisted chatbot-style search engines for cancer variant curation

Abstract

Publicly available artificial intelligence (AI) chatbot-style search engines are gaining popularity for various applications, ranging from writing poetry to determining oncogenic cancer variants and genes. However, to our knowledge, a systematic evaluation of the effectiveness of these tools in cancer variant interpretation is lacking in current literature.

In this proof-of-concept study, four free online AI-assisted search engines (ChatGPT, Perplexity AI, Claude AI, and Llama2) were given simple standardized queries to investigate the clinical relevance of multiple gene variants observed in lung adenocarcinoma, glioma, and acute myeloid leukemia. The queries were structured as follows: 'What is the clinical significance of [GENE] [protein-level (p.) nomenclature] in [cancer type]?'

As anticipated, variants of uncertain significance (VUS) illustrated challenges for using AI-assisted search engines in cancer variant interpretation. Each tool incorrectly attributed oncogenicity to at least 1 of the 6 VUS investigated: Perplexity AI (1/6 VUS incorrectly represented as oncogenic), ChatGPT (2/6), Llama2 (4/6), Claude AI (5/6).

The overestimation of oncogenicity in these tools may be driven by conditioning of these AI-assisted search engines by past and current users for positive assignation attributes or from application of a response format with incorrect extrapolation of studies describing variants in the same gene without the ability to draw nuanced conclusions from studies focusing on different aspects of gene function. While there are challenges in using AI-assisted search engines in the clinical genomic space currently, this rapidly improving technology could provide a useful supplement for cancer variant analysts when combined with caution and expert human oversight.