Journal of the American Medical Informatics Association最新文献

Objective: Implement the 5-type health information technology (HIT) patient safety concern classification system for HIT patient safety issues reported to the Veterans Health Administration's Informatics Patient Safety Office.

Materials and methods: A team of informatics safety analysts retrospectively classified 1 year of HIT patient safety issues by type of HIT patient safety concern using consensus discussions. The processes established during retrospective classification were then applied to incoming HIT safety issues moving forward.

Results: Of 140 issues retrospectively reviewed, 124 met the classification criteria. The majority were HIT failures (eg, software defects) (33.1%) or configuration and implementation problems (29.8%). Unmet user needs and external system interactions accounted for 20.2% and 10.5%, respectively. Absence of HIT safety features accounted for 2.4% of issues, and 4% did not have enough information to classify.

Conclusion: The 5-type HIT safety concern classification framework generated actionable categories helping organizations effectively respond to HIT patient safety risks.

Objective: Predicting mortality after acute myocardial infarction (AMI) is crucial for timely prescription and treatment of AMI patients, but there are no appropriate AI systems for clinicians. Our primary goal is to develop a reliable and interpretable AI system and provide some valuable insights regarding short, and long-term mortality.

Materials and methods: We propose the RIAS framework, an end-to-end framework that is designed with reliability and interpretability at its core and automatically optimizes the given model. Using RIAS, clinicians get accurate and reliable predictions which can be used as likelihood, with global and local explanations, and "what if" scenarios to achieve desired outcomes as well.

Results: We apply RIAS to AMI prognosis prediction data which comes from the Korean Acute Myocardial Infarction Registry. We compared FT-Transformer with XGBoost and MLP and found that FT-Transformer has superiority in sensitivity and comparable performance in AUROC and F1 score to XGBoost. Furthermore, RIAS reveals the significance of statin-based medications, beta-blockers, and age on mortality regardless of time period. Lastly, we showcase reliable and interpretable results of RIAS with local explanations and counterfactual examples for several realistic scenarios.

Discussion: RIAS addresses the "black-box" issue in AI by providing both global and local explanations based on SHAP values and reliable predictions, interpretable as actual likelihoods. The system's "what if" counterfactual explanations enable clinicians to simulate patient-specific scenarios under various conditions, enhancing its practical utility.

Conclusion: The proposed framework provides reliable and interpretable predictions along with counterfactual examples.

Background: Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process.

Objectives: This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks.

Materials and methods: We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator.

Results: The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies.

Conclusion: The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Objectives: To illustrate the utility of the All of Us Research Program for studying military and veteran health.

Materials and methods: Results were derived from the All of Us Researcher Workbench Controlled Tier v7. Specific variables examined were family history of post-traumatic stress disorder (PTSD), medical encounters, and body mass index/body size.

Results: There are 37 363 military and veteran participants enrolled in the All of Us Research Program. The population is older (M = 63.3 years), White (71.3%), and male (83.2%), consistent with military and veteran populations. Participants reported a high prevalence of PTSD (13.4%), obesity (40.2%), and abdominal obesity (77.1%).

Discussion and conclusion: The breadth and depth of health data from service members and veterans enrolled in the All of Us Research Program allow researchers to address pressing health questions in these populations. Future enrollment and data releases will make this an increasingly powerful and useful study for understanding military and veteran health.

Background: There are significant disparities in access and utilization of patient portals by age, language, race, and ethnicity.

Materials and methods: We developed ambulatory and inpatient portal activation equity dashboards to understand disparities in initial portal activation, identify targets for improvement, and enable monitoring of interventions over time. We selected key metrics focused on episodes of care and filters to enable high-level overviews and granular data selection to meet the needs of health system leaders and individual clinical units.

Results: In addition to highlighting disparities by age, preferred language, race and ethnicity, and insurance payor, the dashboards enabled development and monitoring of interventions to improve portal activation and equity.

Discussion and conclusions: Data visualization tools that provide easily accessible, timely, and customizable data can enable a variety of stakeholders to understand and address healthcare disparities, such as patient portal activation. Further institutional efforts are needed to address the persistent inequities highlighted by these dashboards.

Objective: This case study describes how an All of Us engagement project returned value to community by strengthening high school students' capacity to serve as health advocates.

Materials and methods: Project activities included health literacy education and research projects on the influence of environmental, societal, and lifestyle factors on community health disparities. The research project involved use of the Photovoice method and All of Us data. At project's end, students presented their research to the community.

Results: The project's success was measured by students' participation in the research poster session and comparison of pre- and post-project scores from the Health Literacy Assessment Scale for Adolescent. Data analysis suggests the project succeeded in meeting its goal of increasing students' health literacy.

Discussion and conclusion: Through education and research activities, students learned about community health issues and the importance of participation in medical research programs, like All of Us, to address issues.

Objectives: To evaluate the efficacy of ChatGPT 4 (GPT-4) in delivering genetic information about BRCA1, HFE, and MLH1, building on previous findings with ChatGPT 3.5 (GPT-3.5). To focus on assessing the utility, limitations, and ethical implications of using ChatGPT in medical settings.

Materials and methods: A structured survey was developed to assess GPT-4's clinical value. An expert panel of genetic counselors and clinical geneticists evaluated GPT-4's responses to these questions. We also performed comparative analysis with GPT-3.5, utilizing descriptive statistics and using Prism 9 for data analysis.

Results: The findings indicate improved accuracy in GPT-4 over GPT-3.5 (P < .0001). However, notable errors in accuracy remained. The relevance of responses varied in GPT-4, but was generally favorable, with a mean in the "somewhat agree" range. There was no difference in performance by disease category. The 7-question subset of the Bot Usability Scale (BUS-15) showed no statistically significant difference between the groups but trended lower in the GPT-4 version.

Discussion and conclusion: The study underscores GPT-4's potential role in genetic education, showing notable progress yet facing challenges like outdated information and the necessity of ongoing refinement. Our results, while showing promise, emphasizes the importance of balancing technological innovation with ethical responsibility in healthcare information delivery.

Objective: Allow health professionals to monitor and anticipate demands for emergency care in the Île-de-France region of France.

Materials and methods: Data from emergency departments and emergency medical services are automatically processed on a daily basis and visualized through an interactive online dashboard. Forecasting methods are used to provide 7 days predictions.

Results: The dashboard displays data at regional and departmental levels or for five different age categories. It features summary statistics, historical values, predictions, comparisons to previous years, and monitoring of common reasons for care and outcomes.

Discussion: A large number of health professionals have already requested access to the dashboard (n = 606). Although the quality of data transmitted may vary slightly, the dashboard has already helped improve health situational awareness and anticipation.

Conclusions: The high access demand to the dashboard demonstrates the operational usefulness of real time visualization of multisource data coupled with advanced analytics.

Objective: This study experimentally evaluated how well lay individuals could interpret and use 4 types of electronic health record (EHR) patient-facing immunization visualizations.

Materials and methods: Participants (n = 69) completed the study using a secure online survey platform. Participants viewed the same immunization information in 1 of 4 EHR-based immunization visualizations: 2 different patient portals (Epic MyChart and eClinicWorks), a downloadable EHR record, and a clinic-generated electronic letter (eLetter). Participants completed a common task, created a standard vaccine schedule form, and answered questions about their perceived workload, subjective numeracy and health literacy, demographic variables, and familiarity with the task.

Results: The design of the immunization visualization significantly affected both task performance measures (time taken to complete the task and number of correct dates). In particular, those using Epic MyChart took significantly longer to complete the task than those using eLetter or eClinicWorks. Those using Epic MyChart entered fewer correct dates than those using the eLetter or eClinicWorks. There were no systematic statistically significant differences in task performance measures based on the numeracy, health literacy, demographic, and experience-related questions we asked.

Discussion: The 4 immunization visualizations had unique design elements that likely contributed to these performance differences.

Conclusion: Based on our findings, we provide practical guidance for the design of immunization visualizations, and future studies. Future research should focus on understanding the contexts of use and design elements that make tables an effective type of health data visualization.

Objectives: Medical practitioners analyze numerous types of data, often using archaic representations that do not meet their needs. Pneumologists who analyze lung function exams must often consult multiple exam records manually, making comparisons cumbersome. Such shortcomings can be addressed with interactive visualizations, but these must be designed carefully with practitioners' needs in mind.

Materials and methods: A workshop with experts was conducted to gather user requirements and common tasks. Based on the workshop results, we iteratively designed a web-based prototype, continuously consulting experts along the way. The resulting application was evaluated in a formative study via expert interviews with 3 medical practitioners.

Results: Participants in our study were able to solve all tasks in accordance with experts' expectations and generally viewed our system positively, though there were some usability and utility issues in the initial prototype. An improved version of our system solves these issues and includes additional customization functionalities.

Discussion: The study results showed that participants were able to use our system effectively to solve domain-relevant tasks, even though some shortcomings could be observed. Using a different framework with more fine-grained control over interactions and visual elements, we implemented design changes in an improved version of our prototype that needs to be evaluated in future work.

Conclusion: Employing a user-centered design approach, we developed a visual analytics system for lung function data that allows medical practitioners to more easily analyze the progression of several key parameters over time.