JMIR Medical Informatics最新文献

Background: Artificial intelligence (AI) chatbots have recently gained use in medical practice by health care practitioners. Interestingly, the output of these AI chatbots was found to have varying degrees of hallucination in content and references. Such hallucinations generate doubts about their output and their implementation.

Objective: The aim of our study was to propose a reference hallucination score (RHS) to evaluate the authenticity of AI chatbots' citations.

Methods: Six AI chatbots were challenged with the same 10 medical prompts, requesting 10 references per prompt. The RHS is composed of 6 bibliographic items and the reference's relevance to prompts' keywords. RHS was calculated for each reference, prompt, and type of prompt (basic vs complex). The average RHS was calculated for each AI chatbot and compared across the different types of prompts and AI chatbots.

Results: Bard failed to generate any references. ChatGPT 3.5 and Bing generated the highest RHS (score=11), while Elicit and SciSpace generated the lowest RHS (score=1), and Perplexity generated a middle RHS (score=7). The highest degree of hallucination was observed for reference relevancy to the prompt keywords (308/500, 61.6%), while the lowest was for reference titles (169/500, 33.8%). ChatGPT and Bing had comparable RHS (β coefficient=-0.069; P=.32), while Perplexity had significantly lower RHS than ChatGPT (β coefficient=-0.345; P<.001). AI chatbots generally had significantly higher RHS when prompted with scenarios or complex format prompts (β coefficient=0.486; P<.001).

Conclusions: The variation in RHS underscores the necessity for a robust reference evaluation tool to improve the authenticity of AI chatbots. Further, the variations highlight the importance of verifying their output and citations. Elicit and SciSpace had negligible hallucination, while ChatGPT and Bing had critical hallucination levels. The proposed AI chatbots' RHS could contribute to ongoing efforts to enhance AI's general reliability in medical research.

Background: Although Ethiopia has made remarkable progress in the uptake of the District Health Information System version 2 (DHIS2) for national aggregate data reporting, there has been no comprehensive assessment of the maturity level of the system.

Objective: This study aims to assess the maturity level of DHIS2 implementation in Ethiopia and propose a road map that could guide the progress toward a higher level of maturity. We also aim to assess the current maturity status, implementation gaps, and future directions of DHIS2 implementation in Ethiopia. The assessment focused on digital health system governance, skilled human resources, information and communication technology (ICT) infrastructure, interoperability, and data quality and use.

Methods: A collaborative assessment was conducted with the engagement of key stakeholders through consultative workshops using the Stages of Continuous Improvement tool to measure maturity levels in 5 core domains, 13 components, and 39 subcomponents. A 5-point scale (1=emerging, 2=repeatable, 3=defined, 4=managed, and 5=optimized) was used to measure the DHIS2 implementation maturity level.

Results: The national DHIS2 implementation's maturity level is currently at the defined stage (score=2.81) and planned to move to the manageable stage (score=4.09) by 2025. The domain-wise maturity score indicated that except for ICT infrastructure, which is at the repeatable stage (score=2.14), the remaining 4 domains are at the defined stage (score=3). The development of a standardized and basic DHIS2 process at the national level, the development of a 10-year strategic plan to guide the implementation of digital health systems including DHIS2, and the presence of the required competencies at the facility level to accomplish specific DHIS2-related tasks are the major strength of the Ministry of Health of Ethiopia so far. The lack of workforce competency guidelines to support the implementation of DHIS2; the unavailability of core competencies (knowledge, skills, and abilities) required to accomplish DHIS2 tasks at all levels of the health system; and ICT infrastructures such as communication network and internet connectivity at the district, zonal, and regional levels are the major hindrances to effective DHIS2 implementation in the country.

Conclusions: On the basis of the Stages of Continuous Improvement maturity model toolkit, the implementation status of DHIS2 in Ethiopia is at the defined stage, with the ICT infrastructure domain being at the lowest stage as compared to the other 4 domains. By 2025, the maturity status is planned to move from the defined stage to the managed stage by improving the identified gaps. Various action points are suggested to address the identified gaps and reach the stated maturity level. The responsible body, necessary resources, and methods of verification required to reac

Background: Childhood tumors in the central nervous system (CNS) have longer diagnostic delays than other pediatric tumors. Vague presenting symptoms pose a challenge in the diagnostic process; it has been indicated that patients and parents may be hesitant to seek help, and health care professionals (HCPs) may lack awareness and knowledge about clinical presentation. To raise awareness among HCPs, the Danish CNS tumor awareness initiative hjernetegn.dk was launched.

Objective: This study aims to present the learnings from designing and implementing a decision support tool for HCPs to reduce diagnostic delay in childhood CNS tumors. The aims also include decisions regarding strategies for dissemination and use of social media, and an evaluation of the digital impact 6 months after launch.

Methods: The phases of developing and implementing the tool include participatory co-creation workshops, designing the website and digital platforms, and implementing a press and media strategy. The digital impact of hjernetegn.dk was evaluated through website analytics and social media engagement.

Implementation (results): hjernetegn.dk was launched in August 2023. The results after 6 months exceeded key performance indicators. The analysis showed a high number of website visitors and engagement, with a plateau reached 3 months after the initial launch. The LinkedIn campaign and Google Search strategy also generated a high number of impressions and clicks.

Conclusions: The findings suggest that the initiative has been successfully integrated, raising awareness and providing a valuable tool for HCPs in diagnosing childhood CNS tumors. The study highlights the importance of interdisciplinary collaboration, co-creation, and ongoing community management, as well as broad dissemination strategies when introducing a digital support tool.

Background: Interpretability and intuitive visualization facilitate medical knowledge generation through big data. In addition, robustness to high-dimensional and missing data is a requirement for statistical approaches in the medical domain. A method tailored to the needs of physicians must meet all the abovementioned criteria.

Objective: This study aims to develop an accessible tool for visual data exploration without the need for programming knowledge, adjusting complex parameterizations, or handling missing data. We sought to use statistical analysis using the setting of disease and control cohorts familiar to clinical researchers. We aimed to guide the user by identifying and highlighting data patterns associated with disease and reveal relations between attributes within the data set.

Methods: We introduce the attribute association graph, a novel graph structure designed for visual data exploration using robust statistical metrics. The nodes capture frequencies of participant attributes in disease and control cohorts as well as deviations between groups. The edges represent conditional relations between attributes. The graph is visualized using the Neo4j (Neo4j, Inc) data platform and can be interactively explored without the need for technical knowledge. Nodes with high deviations between cohorts and edges of noticeable conditional relationship are highlighted to guide the user during the exploration. The graph is accompanied by a dashboard visualizing variable distributions. For evaluation, we applied the graph and dashboard to the Hamburg City Health Study data set, a large cohort study conducted in the city of Hamburg, Germany. All data structures can be accessed freely by researchers, physicians, and patients. In addition, we developed a user test conducted with physicians incorporating the System Usability Scale, individual questions, and user tasks.

Results: We evaluated the attribute association graph and dashboard through an exemplary data analysis of participants with a general cardiovascular disease in the Hamburg City Health Study data set. All results extracted from the graph structure and dashboard are in accordance with findings from the literature, except for unusually low cholesterol levels in participants with cardiovascular disease, which could be induced by medication. In addition, 95% CIs of Pearson correlation coefficients were calculated for all associations identified during the data analysis, confirming the results. In addition, a user test with 10 physicians assessing the usability of the proposed methods was conducted. A System Usability Scale score of 70.5% and average successful task completion of 81.4% were reported.

Conclusions: The proposed attribute association graph and dashboard enable intuitive visual data exploration. They are robust to high-dimensional as well as missing data and require no pa

Background: The Hispanic community represents a sizeable community that experiences inequities in the US health care system. As the system has moved toward digital health platforms, evaluating the potential impact on Hispanic communities is critical.

Objective: The study aimed to investigate demographic, socioeconomic, and behavioral factors contributing to low telehealth use in Hispanic communities.

Methods: We used a retrospective observation study design to examine the study objectives. The COVID-19 Research Database Consortium provided the Analytics IQ PeopleCore consumer data and Office Alley claims data. The study period was from March 2020 to April 2021. Multiple logistic regression was used to determine the odds of using telehealth services.

Results: We examined 3,478,287 unique Hispanic patients, 16.6% (577,396) of whom used telehealth. Results suggested that patients aged between 18 and 44 years were more likely to use telehealth (odds ratio [OR] 1.07, 95% CI 1.05-1.1; P<.001) than patients aged older than 65 years. Across all age groups, patients with high incomes were at least 20% more likely to use telehealth than patients with lower incomes (P<.001); patients who had a primary care physician (P=.01), exhibited high medical usage (P<.001), or were interested in exercise (P=.03) were more likely to use telehealth; patients who had unhealthy behaviors such as smoking and alcohol consumption were less likely to use telehealth (P<.001). Male patients were less likely than female patients to use telehealth among patients aged 65 years and older (OR 0.94, 95% CI 0.93-0.95; P<.001), while male patients aged between 18 and 44 years were more likely to use telehealth (OR 1.05, 95% CI 1.03-1.07; P<.001). Among patients younger than 65 years, full-time employment was positively associated with telehealth use (P<.001). Patients aged between 18 and 44 years with high school or less education were 2% less likely to use telehealth (OR 0.98, 95% CI 0.97-0.99; P=.005). Results also revealed a positive association with using WebMD (WebMD LLC) among patients aged older than 44 years (P<.001), while there was a negative association with electronic prescriptions among those who were aged between 18 and 44 years (P=.009) and aged between 45 and 64 years (P=.004).

Conclusions: This study demonstrates that telehealth use among Hispanic communities is dependent upon factors such as age, gender, education, socioeconomic status, current health care engagement, and health behaviors. To address these challenges, we advocate for interdisciplinary approaches that involve medical professionals, insurance providers, and community-based services actively engaging with Hispanic communities and promoting telehealth use. We propose the following recommendations: enhance access to health insurance, improve access to primary care providers, and allocate fiscal a

Background: Early identification of impending in-hospital cardiac arrest (IHCA) improves clinical outcomes but remains elusive for practicing clinicians.

Objective: We aimed to develop a multimodal machine learning algorithm based on ensemble techniques to predict the occurrence of IHCA.

Methods: Our model was developed by the Multiparameter Intelligent Monitoring of Intensive Care (MIMIC)-IV database and validated in the Electronic Intensive Care Unit Collaborative Research Database (eICU-CRD). Baseline features consisting of patient demographics, presenting illness, and comorbidities were collected to train a random forest model. Next, vital signs were extracted to train a long short-term memory model. A support vector machine algorithm then stacked the results to form the final prediction model.

Results: Of 23,909 patients in the MIMIC-IV database and 10,049 patients in the eICU-CRD database, 452 and 85 patients, respectively, had IHCA. At 13 hours in advance of an IHCA event, our algorithm had already demonstrated an area under the receiver operating characteristic curve of 0.85 (95% CI 0.815-0.885) in the MIMIC-IV database. External validation with the eICU-CRD and National Taiwan University Hospital databases also presented satisfactory results, showing area under the receiver operating characteristic curve values of 0.81 (95% CI 0.763-0.851) and 0.945 (95% CI 0.934-0.956), respectively.

Conclusions: Using only vital signs and information available in the electronic medical record, our model demonstrates it is possible to detect a trajectory of clinical deterioration up to 13 hours in advance. This predictive tool, which has undergone external validation, could forewarn and help clinicians identify patients in need of assessment to improve their overall prognosis.

Background: Cross-institutional interoperability between health care providers remains a recurring challenge worldwide. The German Medical Informatics Initiative, a collaboration of 37 university hospitals in Germany, aims to enable interoperability between partner sites by defining Fast Healthcare Interoperability Resources (FHIR) profiles for the cross-institutional exchange of health care data, the Core Data Set (CDS). The current CDS and its extension modules define elements representing patients' health care records. All university hospitals in Germany have made significant progress in providing routine data in a standardized format based on the CDS. In addition, the central research platform for health, the German Portal for Medical Research Data feasibility tool, allows medical researchers to query the available CDS data items across many participating hospitals.

Objective: In this study, we aimed to evaluate a novel approach of combining the current top-down generated FHIR profiles with the bottom-up generated knowledge gained by the analysis of respective instance data. This allowed us to derive options for iteratively refining FHIR profiles using the information obtained from a discrepancy analysis.

Methods: We developed an FHIR validation pipeline and opted to derive more restrictive profiles from the original CDS profiles. This decision was driven by the need to align more closely with the specific assumptions and requirements of the central feasibility platform's search ontology. While the original CDS profiles offer a generic framework adaptable for a broad spectrum of medical informatics use cases, they lack the specificity to model the nuanced criteria essential for medical researchers. A key example of this is the necessity to represent specific laboratory codings and values interdependencies accurately. The validation results allow us to identify discrepancies between the instance data at the clinical sites and the profiles specified by the feasibility platform and addressed in the future.

Results: A total of 20 university hospitals participated in this study. Historical factors, lack of harmonization, a wide range of source systems, and case sensitivity of coding are some of the causes for the discrepancies identified. While in our case study, Conditions, Procedures, and Medications have a high degree of uniformity in the coding of instance data due to legislative requirements for billing in Germany, we found that laboratory values pose a significant data harmonization challenge due to their interdependency between coding and value.

Conclusions: While the CDS achieves interoperability, different challenges for federated data access arise, requiring more specificity in the profiles to make assumptions on the instance data. We further argue that further harmonization of the instance data can significantly lower required

Background: Medication safety in residential care facilities is a critical concern, particularly when nonmedical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on health care providers, underscores the need for effective incident analysis and preventive strategies. A thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents.

Objective: We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff.

Methods: We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F₁-score and exact match accuracy through 5-fold cross-validation.

Results: Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included "procedure adherence," "medicine," "resident," "resident family," "nonmedical staff," "medical staff," "team," "environment," and "organizational management," respectively. Owing to limited labels, "resident family" and "medical staff" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F₁-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy