Chat-ePRO: Development and pilot study of an electronic patient-reported outcomes system based on ChatGPT

IF 4 2区医学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS Journal of Biomedical Informatics Pub Date : 2024-05-03 DOI:10.1016/j.jbi.2024.104651

Zikang Chen , Qinchuan Wang , Yaoqian Sun , Hailing Cai , Xudong Lu

{"title":"Chat-ePRO: Development and pilot study of an electronic patient-reported outcomes system based on ChatGPT","authors":"Zikang Chen , Qinchuan Wang , Yaoqian Sun , Hailing Cai , Xudong Lu","doi":"10.1016/j.jbi.2024.104651","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><p>Chatbots have the potential to improve user compliance in electronic Patient-Reported Outcome (ePRO) system. Compared to rule-based chatbots, Large Language Model (LLM) offers advantages such as simplifying the development process and increasing conversational flexibility. However, there is currently a lack of practical applications of LLMs in ePRO systems. Therefore, this study utilized ChatGPT to develop the Chat-ePRO system and designed a pilot study to explore the feasibility of building an ePRO system based on LLM.</p></div><div><h3>Materials and Methods</h3><p>This study employed prompt engineering and offline knowledge distillation to design a dialogue algorithm and built the Chat-ePRO system on the WeChat Mini Program platform. In order to compare Chat-ePRO with the form-based ePRO and rule-based chatbot ePRO used in previous studies, we conducted a pilot study applying the three ePRO systems sequentially at the Sir Run Run Shaw Hospital to collect patients’ PRO data.</p></div><div><h3>Result</h3><p>Chat-ePRO is capable of correctly generating conversation based on PRO forms (success rate: 95.7 %) and accurately extracting the PRO data instantaneously from conversation (Macro-F1: 0.95). The majority of subjective evaluations from doctors (>70 %) suggest that Chat-ePRO is able to comprehend questions and consistently generate responses. Pilot study shows that Chat-ePRO demonstrates higher response rate (9/10, 90 %) and longer interaction time (10.86 s/turn) compared to the other two methods.</p></div><div><h3>Conclusion</h3><p>Our study demonstrated the feasibility of utilizing algorithms such as prompt engineering to drive LLM in completing ePRO data collection tasks, and validated that the Chat-ePRO system can effectively enhance patient compliance.</p></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"154 ","pages":"Article 104651"},"PeriodicalIF":4.0000,"publicationDate":"2024-05-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Biomedical Informatics","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1532046424000698","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

Chatbots have the potential to improve user compliance in electronic Patient-Reported Outcome (ePRO) system. Compared to rule-based chatbots, Large Language Model (LLM) offers advantages such as simplifying the development process and increasing conversational flexibility. However, there is currently a lack of practical applications of LLMs in ePRO systems. Therefore, this study utilized ChatGPT to develop the Chat-ePRO system and designed a pilot study to explore the feasibility of building an ePRO system based on LLM.

Materials and Methods

This study employed prompt engineering and offline knowledge distillation to design a dialogue algorithm and built the Chat-ePRO system on the WeChat Mini Program platform. In order to compare Chat-ePRO with the form-based ePRO and rule-based chatbot ePRO used in previous studies, we conducted a pilot study applying the three ePRO systems sequentially at the Sir Run Run Shaw Hospital to collect patients’ PRO data.

Result

Chat-ePRO is capable of correctly generating conversation based on PRO forms (success rate: 95.7 %) and accurately extracting the PRO data instantaneously from conversation (Macro-F1: 0.95). The majority of subjective evaluations from doctors (>70 %) suggest that Chat-ePRO is able to comprehend questions and consistently generate responses. Pilot study shows that Chat-ePRO demonstrates higher response rate (9/10, 90 %) and longer interaction time (10.86 s/turn) compared to the other two methods.

Conclusion

Our study demonstrated the feasibility of utilizing algorithms such as prompt engineering to drive LLM in completing ePRO data collection tasks, and validated that the Chat-ePRO system can effectively enhance patient compliance.

Abstract Image

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

Chat-ePRO：基于 ChatGPT 的电子患者报告结果系统的开发和试点研究。

目的聊天机器人有可能提高电子患者报告结果（ePRO）系统中用户的依从性。与基于规则的聊天机器人相比，大语言模型（LLM）具有简化开发流程和提高对话灵活性等优势。然而，目前在 ePRO 系统中缺乏对 LLM 的实际应用。因此，本研究利用 ChatGPT 开发了 Chat-ePRO 系统，并设计了一项试点研究来探索基于 LLM 构建 ePRO 系统的可行性：本研究采用提示工程和离线知识提炼设计了对话算法，并在微信小程序平台上构建了Chat-ePRO系统。为了将 Chat-ePRO 与之前研究中使用的基于表单的电子病历和基于规则的聊天机器人电子病历进行比较，我们在邵逸夫医院进行了一项试点研究，依次应用这三种电子病历系统收集患者的 PRO 数据：结果：Chat-ePRO 能够根据 PRO 表格正确生成对话（成功率：95.7%），并能准确地从对话中即时提取 PRO 数据（Macro-F1：0.95）。大多数医生的主观评价（大于 70%）表明，Chat-ePRO 能够理解问题并持续生成回复。试点研究表明，与其他两种方法相比，Chat-ePRO 的回复率更高（9/10，90%），互动时间更长（10.86 秒/转）：我们的研究证明了利用提示工程等算法驱动 LLM 完成 ePRO 数据收集任务的可行性，并验证了 Chat-ePRO 系统能有效提高患者的依从性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Journal of Biomedical Informatics 医学-计算机：跨学科应用

CiteScore

8.90

自引率

6.70%

发文量

243

审稿时长

32 days

期刊介绍： The Journal of Biomedical Informatics reflects a commitment to high-quality original research papers, reviews, and commentaries in the area of biomedical informatics methodology. Although we publish articles motivated by applications in the biomedical sciences (for example, clinical medicine, health care, population health, and translational bioinformatics), the journal emphasizes reports of new methodologies and techniques that have general applicability and that form the basis for the evolving science of biomedical informatics. Articles on medical devices; evaluations of implemented systems (including clinical trials of information technologies); or papers that provide insight into a biological process, a specific disease, or treatment options would generally be more suitable for publication in other venues. Papers on applications of signal processing and image analysis are often more suitable for biomedical engineering journals or other informatics journals, although we do publish papers that emphasize the information management and knowledge representation/modeling issues that arise in the storage and use of biological signals and images. System descriptions are welcome if they illustrate and substantiate the underlying methodology that is the principal focus of the report and an effort is made to address the generalizability and/or range of application of that methodology. Note also that, given the international nature of JBI, papers that deal with specific languages other than English, or with country-specific health systems or approaches, are acceptable for JBI only if they offer generalizable lessons that are relevant to the broad JBI readership, regardless of their country, language, culture, or health system.