JAMIA Open最新文献

Objective: Medical examiners and coroners (ME/C) oversee medicolegal death investigations which determine causes of death and other contextual factors that may have influenced a death. We utilize open data releases from ME/C offices covering 6 different geographic areas to demonstrate the strengths and limitations of ME/C data for forensic epidemiology research.

Materials and methods: We use our novel geoPIPE tool to establish a pipeline that (a) automates ingesting open data releases, (b) geocodes records where possible to yield a spatial component, (c) enhances data with variables useful for overdose research, such as flagging substances contributing to each death, and (d) publishes the enriched data to our open repository. We use results from this pipeline to highlight similarities and differences of overdose data across different sources.

Results: Text processing to extract drugs contributing to each death yielded compatible data across all locations. Conversely, geospatial analyses are sometimes incompatible due to differences in available geographic resolution, which range from fine-grain latitude and longitude coordinates to larger regions identified by zip codes. Our pipeline pushes weekly results to an open repository.

Discussion: Open ME/C data are highly useful for research on substance use disorders; our visualizations demonstrate the ability to contextualize overdose data within and across specific geographic regions. Furthermore, the spatial component of our results enables clustering of overdose events and accessibility studies for resources related to preventing overdose deaths.

Conclusions: Given the utility to public health researchers, we advocate that other ME/C offices explore releasing open data and for policy makers to support and fund transparency efforts.

Background: Deprescribing is the clinically supervised process of stopping or reducing medications that are no longer beneficial. MedSafer is an electronic decision support tool that guides healthcare providers (HCPs) through the deprescribing process. We recently developed a novel patient-facing version of the software, allowing patients and caregivers to generate a personalized deprescribing report to bring to their prescriber.

Objective: The study aimed to evaluate the usability and acceptability of MedSafer among older adults, caregivers, and community HCPs (physicians, nurse practitioners and pharmacists).

Method: A mixed-methods feasibility study was conducted with a convenience sample of 100 older adults/caregivers, and 25 healthcare practitioners. Participants were invited to test MedSafer and answer telephone or electronic surveys via RedCap. The Extended Technology Acceptance Model (TAM2) and System Usability Scale (SUS) were used for evaluation. A semi-structured interview was also conducted with a subset of participants (5 per group) who were selected on a volunteer basis, and thematic analysis was used following Braun & Clarke's approach.

Results: Healthcare providers scored more favorably on TAM2 constructs such as perceived usefulness (PU) (median: 4.25 for HCPs; 3.75 for caregivers; 3.00 for patients), and SUS compared to patients and caregivers (mean: 79.50 for HCPs; 52.95 for caregivers; 55.75 for patients). Thematic analysis revealed that participants recognized MedSafer as an empowering tool but noted the need for some usability improvements.

Conclusion: MedSafer is a promising tool to support deprescribing conversations. Enhancing usability, accessibility, and patient education may improve adoption.

Introduction: Clinical trial data is still predominantly manually entered by site staff into Electronic Data Capture (EDC) systems. This process of abstracting and manually transcribing patient data is time-consuming, inefficient and error prone. Use of Electronic Health Record to Electronic Data Capture (EHR-To-EDC) technologies that digitize this process would improve these inefficiencies.

Objectives: This study measured the impact of EHR-To-EDC technology on the data entry workflow of clinical trial data managers. The primary objective was to compare the speed and accuracy of the EHR-To-EDC enabled data entry method to the traditional, manual method. The secondary objective was to measure end user satisfaction.

Materials and methods: Five data managers ranging in experience from 9 months to over 2 years, were assigned an investigator-initiated, Memorial Sloan Kettering-sponsored oncology study within their disease area of expertise. Each data manager performed one-hour of manual data entry, and a week later, one-hour of data entry using IgniteData's EHR-To-EDC solution, Archer, on a predetermined set of patients, timepoints and data domains (labs, vitals). The data entered into the EDC were compared side-by-side and used to evaluate the speed and accuracy of the EHR-To-EDC enabled method versus traditional, manual data entry. A user satisfaction survey using a 5-point Likert scale was used to collect feedback regarding the selected platform's learnability, ease of use, perceived time savings, perceived efficiency, and preference over the manual method.

Results: The EHR-To-EDC method resulted in 58% more data entered versus the manual method (difference, 1745 data points; manual, 3023 data points; EHR-To-EDC, 4768 data points). The number of data entry errors was reduced by 99% (manual, 100 data points; EHR-To-EDC, 1 data point). Regarding user satisfaction, data managers either agreed or strongly agreed that the EHR-To-EDC workflow was easy to learn (5/5), easy to use (4.6/5), saved time (5/5), was more efficient (4.8/5), and preferred it over the manual entry workflow (4/5).

Conclusion: EHR-To-EDC enabled data entry increases data manager productivity, reduces errors and is preferred by data managers over manual data entry.

Objective: To engage end-users to develop and evaluate an algorithm to convert electronic adherence monitoring device (EAMD) output into the adherence data required for analyses.

Materials and methods: This study included 4 phases. First, process mapping interviews and focus groups were conducted to identify rules for EAMD data processing and user needs. Second, algorithm parameters required to compute daily adherence values were defined and coded in an R package (OncMAP). Third, algorithm-produced data were compared to manually recoded data to evaluate the algorithm's sensitivity, specificity, and accuracy. Finally, pilot testing was conducted to obtain feedback on the perceived value/benefit of the algorithm and features that should be considered during software development.

Results: EAMD data processing rules were identified and coded in an R application. The algorithm correctly classified all complete observations with 100% sensitivity and specificity. The receiver operating characteristic curve analysis yielded an area under the curve of 1.00. All pilot testing participants expressed interest in using the algorithm (Net Promoter Score = 71%) but identified several features essential for inclusion in the software package to ensure widespread adoption.

Discussion: The decision rules implemented to process EAMD actuation data can be parameterized to develop an algorithm to automate this process. The algorithm demonstrated high sensitivity, specificity, and accuracy. End-users were enthusiastic about the product and provided insights to inform the development of a software package including the algorithm.

Conclusion: A rule-based algorithm can accurately process EAMD actuation data and has the potential to improve the rigor and pace of adherence science.

Objectives: Describing the development of a visual dashboard leveraging available tools for efficient recruitment for patient centered clinical trials in resource constrained settings.

Materials and methods: A real-time, visual dashboard was developed, facilitating interactive visualizations, detailed analyses, and data quality control. Daily automated REDCap data retrieval occurred via an R program using REDCap API and output was integrated into Power BI. An interrupted time series analysis was conducted evaluating effects of dashboard on clinical trial recruitment metrics.

Results: The visual dashboard displayed key recruitment metrics, including individual participant progression and recruitment trends over time. Interrupted time series analysis showed improvements in screening rates upon implementation. The mean time to study completion decreased by 19 days following implementation.

Discussion: Customizable metrics offer comprehensive view of recruitment data and granularity, identifying actionable issues, enhancing study timeliness and completion.

Conclusion: Clinical trials of all budgets can integrate dashboards for real-time monitoring and data driven improvements to promote more timely completion.

Objectives: As the use of artificial intelligence (AI) in healthcare is rapidly expanding, there is also growing recognition of the need for ongoing monitoring of AI after implementation, called algorithmovigilance. Yet, there remain few systems that support systematic monitoring and governance of AI used across a health system. In this study, we identify end-user needs for a novel AI monitoring system-the Vanderbilt Algorithmovigilance Monitoring and Operations System (VAMOS)-using human-centered design (HCD).

Materials and methods: We assembled a multidisciplinary team to plan AI monitoring and governance at Vanderbilt University Medical Center. We then conducted 9 participatory design sessions with diverse stakeholders to develop prototypes of VAMOS. Once we had a working prototype, we conducted 8 formative design interviews with key stakeholders to gather feedback on the system. We analyzed the interviews using a rapid qualitative analysis approach and revised the mock-ups. We then conducted a multidisciplinary heuristic evaluation to identify further improvements to the tool.

Results: Through an iterative, HCD process that engaged diverse end-users, we identified key components needed in AI monitoring systems. We identified specific data views and functionality required by end users across several user interfaces including a performance monitoring dashboard, accordion snapshots, and model-specific pages.

Discussion: We distilled general design requirements for systems to support AI monitoring throughout its lifecycle. One important consideration is how to support teams of health system leaders, clinical experts, and technical personnel that are distributed across the organization as they monitor and respond to algorithm deterioration.

Conclusion: VAMOS aims to support systematic and proactive monitoring of AI tools in healthcare organizations. Our findings and recommendations can support the design of AI monitoring systems to support health systems, improve quality of care, and ensure patient safety.

Objectives: Phone surveys are crucial for collecting health data but are expensive, time-consuming, and difficult to scale. To overcome these limitations, we propose a survey collection approach powered by conversational Large Language Models (LLMs).

Materials and methods: Our framework leverages an LLM-powered conversational agent to conduct surveys and transcribe conversations, along with an LLM (GPT-4o) to extract responses from the transcripts. We evaluated the framework's performance by analyzing transcription errors, the accuracy of inferred survey responses, and participant experiences across 40 survey responses collected from a convenience sample of 8 individuals, each adopting the role of five LLM-generated personas.

Results: GPT-4o extracted responses to survey questions with an average accuracy of 98%, despite an average transcription word error rate of 7.7%. Participants reported occasional errors by the conversational agent but praised its ability to demonstrate comprehension and maintain engaging conversations.

Discussion and conclusion: Our study showcases the potential of LLM agents to enable scalable, AI-powered phone surveys, reducing human effort and advancing healthcare data collection.

Objectives: To evaluate ChatGPT's ability to perform thematic analysis of medical Best Practice Advisory (BPA) free-text comments and identify prompt engineering strategies that optimize performance.

Materials and methods: We analyzed 778 BPA comments from a pilot AI-enabled clinical deterioration intervention at Stanford Hospital, categorized as reasons for deterioration (Category 1) and care team actions (Category 2). Prompt engineering strategies (role, context specification, stepwise instructions, few-shot prompting, and dialogue-based calibration) were tested on a 20% random subsample to determine the best-performing prompt. Using that prompt, ChatGPT conducted deductive coding on the full dataset followed by inductive analysis. Agreement with human coding was assessed as inter-rater reliability (IRR) using Cohen's Kappa (κ).

Results: With structured prompts and calibration, ChatGPT achieved substantial agreement with human coding (κ = 0.76 for Category 1; κ = 0.78 for Category 2). Baseline agreement was higher for Category 1 than Category 2, reflecting differences in comment type and complexity, but calibration improved both. Inductive analysis yielded 9 themes, with ChatGPT-generated themes closely aligning with human coding.

Discussion: ChatGPT can accelerate qualitative analysis, but its rigor depends heavily on prompt engineering. Key strategies included role and context specification, pulse-check calibration, and safeguard techniques, which enhanced reliability and reproducibility.

Conclusion: This study demonstrates the feasibility of ChatGPT-assisted thematic analysis and introduces a structured approach for applying LLMs to qualitative analysis of clinical free-text data, underscoring prompt engineering as a methodological lever.

Background: Health information exchanges (HIE), tools that electronically share clinical data across healthcare organizations, provide the opportunity to improve patient care. While widely available, HIE is utilized in only 2%-10% of patient encounters. Few studies have explored current barriers to use. The goal of this study was to evaluate current clinician perspectives on HIE and barriers to use at the point of care.

Methods: We conducted a population-based survey of internal medicine (IM) and emergency medicine (EM) physicians, physician assistants, and nurse practitioners at 8 health systems in the Atlanta area. Survey responses were analyzed overall and by specialty.

Results: Of 1239 clinicians who were invited to participate, 276 (22.3%) responded, with 65.6% of respondents working in inpatient IM and 32.6% in EM. 80.4% of respondents reported using HIE at least once a day, while 4.8% reported never using HIE. Most clinicians used HIE at least daily to access lab results (80.2%), clinical notes (81.9%), imaging reports (74.0%), and medication lists (71.2%). The most reported barriers to HIE utilization included unavailability of needed information (66.4%), adding time to patient care (45.5%), and ease of simply reordering tests (31.6%). HIE use and reported barriers to use were similar across IM and EM providers.

Conclusions: Of those responding to the survey, daily access of HIE was common. We identified several barriers to HIE use, which can be used to develop targeted interventions to improve utilization and patient care. Approaches to reach survey non-responders are also needed.

Objective: This study evaluates the performance and deployment feasibility of a machine learning (ML) model to identify adult-onset type 1 diabetes (T1D) initially coded as type 2 on electronic medical records (EMRs) from a health information exchange (HIE). To our knowledge, this is the first evaluation of such a model on real-world HIE data.

Materials and methods: An existing ML model, trained on national US EMR data, was tested on a regional HIE dataset, after several adjustments for compatibility. A localized model retrained on the regional dataset was compared to the national model. Discrepancies between the 2 datasets' features and cohorts were also investigated.

Results: The national model performed well on HIE data (AUROC = 0.751; precision at 5% recall [PR5] = 25.5%), and localization further improved performance (AUROC = 0.774; PR5 = 35.4%). Differences in the 2 models' top predictors reflected the discrepancies between the datasets and gaps in HIE data capture.

Discussion: The adjustments needed for testing on HIE data highlight the importance of aligning algorithm design with deployment needs. Moreover, localization increased precision, making it more appealing for patient screening, but added complexity and may impact scalability. Additionally, while HIEs offer opportunities for large-scale deployment, data inconsistencies across member organizations could undermine accuracy and providers' trust in ML-based tools.

Conclusion: Our findings offer valuable insights into the feasibility of at-scale deployment of ML models for high-risk patient identification. Although this work focuses on detecting potentially misclassified T1D, our learnings can also inform other applications.