Supplementing systematic review findings with healthcare system data: pilot projects from the Agency for Healthcare Research and Quality Evidence-based Practice Center program

Objectives

The US Agency for Healthcare Research and Quality, through the Evidence-based Practice Center (EPC) Program, aims to provide health system decision makers with the highest-quality evidence to inform clinical decisions. However, limitations in the literature may lead to inconclusive findings in EPC systematic reviews (SRs). The EPC Program conducted pilot projects to understand the feasibility, benefits, and challenges of utilizing health system data to augment SR findings to support confidence in healthcare decision-making based on real-world experiences.

Study Design and Setting

Three contractors (each an EPC located at a different health system) selected a recently completed SR conducted by their center and identified an evidence gap that electronic health record (EHR) data might address. All pilot project topics addressed clinical questions as opposed to care delivery, care organization, or care disparities topics that are common in EPC reports. Topic areas addressed by each EPC included infantile epilepsy, migraine, and hip fracture. EPCs also tracked additional resources needed to conduct supplemental analyses. The workgroup met monthly in 2022-2023 to discuss challenges and lessons learned from the pilot projects.

Results

Two supplemental data analyses filled an evidence gap identified in the SRs (raised certainty of evidence, improved applicability) and the third filled a health system knowledge gap. Project challenges fell under three themes: regulatory and logistical issues, data collection and analysis, and interpretation and presentation of findings. Limited ability to capture key clinical variables given inconsistent or missing data within the EHR was a major limitation. The workgroup found that conducting supplemental data analysis alongside an SR was feasible but adds considerable time and resources to the review process (estimated total hours to complete pilot projects ranged from 283 to 595 across EPCs), and that the increased effort and resources added limited incremental value.

Conclusion

Supplementing existing SRs with analyses of EHR data is resource intensive and requires specialized skillsets throughout the process. While using EHR data for research has immense potential to generate real-world evidence and fill knowledge gaps, these data may not yet be ready for routine use alongside SRs.