JMIR Medical Informatics最新文献

Background: Large language models (LLMs) have been proposed as valuable tools in medical education and practice. The Chinese National Nursing Licensing Examination (CNNLE) presents unique challenges for LLMs due to its requirement for both deep domain-specific nursing knowledge and the ability to make complex clinical decisions, which differentiates it from more general medical examinations. However, their potential application in the CNNLE remains unexplored.

Objective: This study aims to evaluates the accuracy of 7 LLMs including GPT-3.5, GPT-4.0, GPT-4o, Copilot, ERNIE Bot-3.5, SPARK, and Qwen-2.5 on the CNNLE, focusing on their ability to handle domain-specific nursing knowledge and clinical decision-making. We also explore whether combining their outputs using machine learning techniques can improve their overall accuracy.

Methods: This retrospective cross-sectional study analyzed all 1200 multiple-choice questions from the CNNLE conducted between 2019 and 2023. Seven LLMs were evaluated on these multiple-choice questions, and 9 machine learning models, including Logistic Regression, Support Vector Machine, Multilayer Perceptron, k-nearest neighbors, Random Forest, LightGBM, AdaBoost, XGBoost, and CatBoost, were used to optimize overall performance through ensemble techniques.

Results: Qwen-2.5 achieved the highest overall accuracy of 88.9%, followed by GPT-4o (80.7%), ERNIE Bot-3.5 (78.1%), GPT-4.0 (70.3%), SPARK (65.0%), and GPT-3.5 (49.5%). Qwen-2.5 demonstrated superior accuracy in the Practical Skills section compared with the Professional Practice section across most years. It also performed well in brief clinical case summaries and questions involving shared clinical scenarios. When the outputs of the 7 LLMs were combined using 9 machine learning models, XGBoost yielded the best performance, increasing accuracy to 90.8%. XGBoost also achieved an area under the curve of 0.961, sensitivity of 0.905, specificity of 0.978, F₁-score of 0.901, positive predictive value of 0.901, and negative predictive value of 0.977.

Conclusions: This study is the first to evaluate the performance of 7 LLMs on the CNNLE and that the integration of models via machine learning significantly boosted accuracy, reaching 90.8%. These findings demonstrate the transformative potential of LLMs in revolutionizing health care education and call for further research to refine their capabilities and expand their impact on examination preparation and professional training.

Background: Artificial intelligence chatbots are being increasingly used for medical inquiries, particularly in the field of ultrasound medicine. However, their performance varies and is influenced by factors such as language, question type, and topic.

Objective: This study aimed to evaluate the performance of ChatGPT and ERNIE Bot in answering ultrasound-related medical examination questions, providing insights for users and developers.

Methods: We curated 554 questions from ultrasound medicine examinations, covering various question types and topics. The questions were posed in both English and Chinese. Objective questions were scored based on accuracy rates, whereas subjective questions were rated by 5 experienced doctors using a Likert scale. The data were analyzed in Excel.

Results: Of the 554 questions included in this study, single-choice questions comprised the largest share (354/554, 64%), followed by short answers (69/554, 12%) and noun explanations (63/554, 11%). The accuracy rates for objective questions ranged from 8.33% to 80%, with true or false questions scoring highest. Subjective questions received acceptability rates ranging from 47.62% to 75.36%. ERNIE Bot was superior to ChatGPT in many aspects (P<.05). Both models showed a performance decline in English, but ERNIE Bot's decline was less significant. The models performed better in terms of basic knowledge, ultrasound methods, and diseases than in terms of ultrasound signs and diagnosis.

Conclusions: Chatbots can provide valuable ultrasound-related answers, but performance differs by model and is influenced by language, question type, and topic. In general, ERNIE Bot outperforms ChatGPT. Users and developers should understand model performance characteristics and select appropriate models for different questions and languages to optimize chatbot use.

Background: The field of digital health solutions (DHS) has grown tremendously over the past years. DHS include tools for self-management, which support individuals to take charge of their own health. The usability of DHS, as experienced by patients, is pivotal to adoption. However, well-known questionnaires that evaluate usability and satisfaction use complex terminology derived from human-computer interaction and are therefore not well suited to assess experienced usability of patients using DHS in a home setting.

Objective: This study aimed to develop, validate, and assess an instrument that measures experienced usability and satisfaction of patients using DHS in a home setting.

Methods: The development of the "Experienced Usability and Satisfaction with Self-monitoring in the Home Setting" (GEMS) questionnaire followed several steps. Step I consisted of assessing the content validity, by conducting a literature review on current usability and satisfaction questionnaires, collecting statements and discussing these in an expert meeting, and translating each statement and adjusting it to the language level of the general population. This phase resulted in a draft version of the GEMS. Step II comprised assessing its face validity by pilot testing with Amsterdam University Medical Center's patient panel. In step III, psychometric analysis was conducted and the GEMS was assessed for reliability.

Results: A total of 14 items were included for psychometric analysis and resulted in 4 reliable scales: convenience of use, perceived value, efficiency of use, and satisfaction.

Conclusions: Overall, the GEMS questionnaire demonstrated its reliability and validity in assessing experienced usability and satisfaction of DHS in a home setting. Further refinement of the instrument is necessary to confirm its applicability in other patient populations in order to promote the development of a steering mechanism that can be applied longitudinally throughout implementation, and can be used as a benchmarking instrument.

Background: In this study, we evaluate the accuracy, efficiency, and cost-effectiveness of large language models in extracting and structuring information from free-text clinical reports, particularly in identifying and classifying patient comorbidities within oncology electronic health records. We specifically compare the performance of gpt-3.5-turbo-1106 and gpt-4-1106-preview models against that of specialized human evaluators.

Objective: We specifically compare the performance of gpt-3.5-turbo-1106 and gpt-4-1106-preview models against that of specialized human evaluators.

Methods: We implemented a script using the OpenAI application programming interface to extract structured information in JavaScript object notation format from comorbidities reported in 250 personal history reports. These reports were manually reviewed in batches of 50 by 5 specialists in radiation oncology. We compared the results using metrics such as sensitivity, specificity, precision, accuracy, F-value, κ index, and the McNemar test, in addition to examining the common causes of errors in both humans and generative pretrained transformer (GPT) models.

Results: The GPT-3.5 model exhibited slightly lower performance compared to physicians across all metrics, though the differences were not statistically significant (McNemar test, P=.79). GPT-4 demonstrated clear superiority in several key metrics (McNemar test, P<.001). Notably, it achieved a sensitivity of 96.8%, compared to 88.2% for GPT-3.5 and 88.8% for physicians. However, physicians marginally outperformed GPT-4 in precision (97.7% vs 96.8%). GPT-4 showed greater consistency, replicating the exact same results in 76% of the reports across 10 repeated analyses, compared to 59% for GPT-3.5, indicating more stable and reliable performance. Physicians were more likely to miss explicit comorbidities, while the GPT models more frequently inferred nonexplicit comorbidities, sometimes correctly, though this also resulted in more false positives.

Conclusions: This study demonstrates that, with well-designed prompts, the large language models examined can match or even surpass medical specialists in extracting information from complex clinical reports. Their superior efficiency in time and costs, along with easy integration with databases, makes them a valuable tool for large-scale data mining and real-world evidence generation.

Background: Patients with heart failure frequently face the possibility of rehospitalization following an initial hospital stay, placing a significant burden on both patients and health care systems. Accurate predictive tools are crucial for guiding clinical decision-making and optimizing patient care. However, the effectiveness of existing models tailored specifically to the Chinese population is still limited.

Objective: This study aimed to formulate a predictive model for assessing the likelihood of readmission among patients diagnosed with heart failure.

Methods: In this study, we analyzed data from 1948 patients with heart failure in a hospital in Sichuan Province between 2016 and 2019. By applying 3 variable selection strategies, 29 relevant variables were identified. Subsequently, we constructed 6 predictive models using different algorithms: logistic regression, support vector machine, gradient boosting machine, Extreme Gradient Boosting, multilayer perception, and graph convolutional networks.

Results: The graph convolutional network model showed the highest prediction accuracy with an area under the receiver operating characteristic curve of 0.831, accuracy of 75%, sensitivity of 52.12%, and specificity of 90.25%.

Conclusions: The model crafted in this study proves its effectiveness in forecasting the likelihood of readmission among patients with heart failure, thus serving as a crucial reference for clinical decision-making.

Background: Chronic pain is widespread and carries a heavy disease burden, and there is a lack of effective outpatient pain management. As an emerging internet medical platform in China, internet hospitals have been successfully applied for the management of chronic diseases. There are also a certain number of patients with chronic pain that use internet hospitals for pain management. However, no studies have investigated the effectiveness of pain management via internet hospitals.

Objective: The aim of this retrospective cohort study was to explore the effectiveness of chronic pain management by internet hospitals and their advantages and disadvantages compared to traditional physical hospital visits.

Methods: This was a retrospective cohort study. Demographic information such as the patient's sex, age, and number of visits was obtained from the IT center. During the first and last patient visits, information on outcome variables such as the Brief Pain Inventory (BPI), medical satisfaction, medical costs, and adverse drug events was obtained through a telephone follow-up. All patients with chronic pain who had 3 or more visits (internet or offline) between September 2021, and February 2023, were included. The patients were divided into an internet hospital group and a physical hospital group, according to whether they had web-based or in-person consultations, respectively. To control for confounding variables, propensity score matching was used to match the two groups. Matching variables included age, sex, diagnosis, and number of clinic visits.

Results: A total of 122 people in the internet hospital group and 739 people in the physical hospital group met the inclusion criteria. After propensity score matching, 77 patients in each of the two groups were included in the analysis. There was not a significant difference in the quality of life (QOL; QOL assessment was part of the BPI scale) between the internet hospital group and the physical hospital group (P=.80), but the QOL of both groups of patients improved after pain management (internet hospital group: P<.001; physical hospital group: P=.001). There were no significant differences in the pain relief rate (P=.25) or the incidence of adverse events (P=.60) between the two groups. The total cost (P<.001) and treatment-related cost (P<.001) of the physical hospital group were higher than those of the internet hospital group. In addition, the degree of satisfaction in the internet hospital group was greater than that in the physical hospital group (P=.01).

Conclusions: Internet hospitals are an effective way of managing chronic pain. They can improve patients' QOL and satisfaction, reduce treatment costs, and can be used as part of a multimodal strategy for chronic pain self-management.

Background: Telehealth programs and wearable sensors that enable patients to monitor their vital signs have expanded due to the COVID-19 pandemic. The electronic National Early Warning Score (e-NEWS) system helps identify and respond to acute illness.

Objective: This study aimed to implement and evaluate a comprehensive telehealth system to monitor vital signs using e-NEWS for patients receiving integrated home-based medical care (iHBMC). The goal was to improve the early detection of patient deterioration and enhance care delivery in home settings. The system was deployed to optimize remote monitoring in iHBMC and reduce emergency visits and hospitalizations.

Methods: The study was conducted at a medical center and its affiliated home health agency in central Taiwan from November 1, 2022, to October 31, 2023. Patients eligible for iHBMC were enrolled, and sensor data from devices such as blood pressure monitors, thermometers, and pulse oximeters were transmitted to a cloud-based server for e-NEWS calculations at least twice per day over a 2-week period. Patients with e-NEWSs up to 4 received nursing or physician recommendations and interventions based on abnormal physiological data, with reassessment occurring after 2 hours.

Unlabelled: A total of 28 participants were enrolled, with a median age of 84.5 (IQR 79.3-90.8) years, and 32% (n=9) were male. All participants had caregivers, with only 5 out of 28 (18%) able to make decisions independently. The system was implemented across one medical center and its affiliated home health agency. Of the 28 participants, 27 completed the study, while 1 exited early due to low blood pressure and shortness of breath. The median e-NEWS value was 4 (IQR 3-6), with 397 abnormal readings recorded. Of the remaining 27 participants, 8 participants had earlier home visits due to abnormal readings, 6 required hypertension medication adjustments, and 9 received advice on oxygen supplementation. Overall, 24 out of 28 (86%) participants reported being satisfied with the system.

Conclusions: This study demonstrated the feasibility of implementing a telehealth system integrated with e-NEWS in iHBMC settings, potentially aiding in the early detection of clinical deterioration. Although caregivers receive training and resources for their tasks, the system may increase their workload, which could lead to higher stress levels. The small sample size, short monitoring duration, and regional focus in central Taiwan may further limit the applicability of the findings to areas with differing countries, regions, and health care infrastructures. Further research is required to confirm its impact.

Background: Delirium is common in hospitalized patients and correlated with increased morbidity and mortality. Despite this, delirium is underdiagnosed, and many institutions do not have sufficient resources to consistently apply effective screening and prevention.

Objective: To develop a machine learning algorithm to identify patients at highest risk of delirium in the hospital each day in an automated fashion based on data available in the electronic medical record, reducing the barrier to large-scale delirium screening.

Methods: We developed and compared multiple machine learning models on a retrospective dataset of all hospitalized adult patients with recorded Confusion Assessment Method (CAM) screens at a major academic medical center from April 2nd, 2016 to January 16th 2019, comprising 23006 patients. The patient's age, gender, and all available laboratory values, vital signs, prior CAM screens, and medication administrations were used as potential predictors. Four machine learning approaches were investigated: logistic regression with L1-regularization, multilayer perceptrons, random forests, and boosted trees. Model development used 80% of the patients; the remaining 20% were reserved for testing the final models. Lab values, vital signs, medications, gender, and age were used to predict a positive CAM screen in the next 24 hours.

Results: The boosted tree model achieved the greatest predictive power, with a 0.92 area under the receiver operator characteristic curve (AUROC) (95% Confidence Interval (CI) 0.913-9.22), followed by the random forest at 0.91 (95% CI 0.909-0.918), multilayer perceptron at 0.86 (95% CI 0.850-0.861), and logistic regression at 0.85 (95% CI 0.841-0.852). These AUROCs decreased to 0.78-0.82 and 0.74-0.80 when limited to patients not currently or never delirious, respectively.

Conclusions: A boosted tree machine learning model was able to identify hospitalized patients at elevated risk for delirium in the next 24 hours. This may allow for automated delirium risk screening and more precise targeting of proven and investigational interventions to prevent delirium.

Clinicaltrial: