Implementing Accuracy, Completeness, and Traceability for Data Reliability.

Abstract

Importance: While it is well known that data quality underlies evidence validity, the measurement and impacts of data reliability are less well understood. The need has been highlighted in the 21st Century Cures Act of 2016 and US Food and Drug Administration (FDA) Real-World Evidence Program framework in 2018, draft guidance in 2021 and final guidance in 2024. Timely visibility into implementation may be provided by the Transforming Real-World Evidence With Unstructured and Structured Data to Advance Tailored Therapy (TRUST) study, a Verantos Inc-led FDA-funded demonstration project to explore data quality and inform regulatory decision-making.

Objective: To report early learnings from the TRUST study on distilling data reliability to practice including developing a practical approach to quantify accuracy, completeness, and traceability of real-world data (routinely collected patient health data) and comparing traditional to advanced data and technologies on these dimensions.

Design, setting, and participants: This quality improvement study was performed using data from 58 hospitals and more than 1180 associated outpatient clinics from academic and community settings in the US. Participants included patients with asthma treated between January 1, 2014, and December 31, 2022. Data were analyzed from January 1 to June 30, 2024.

Exposures: The traditional approach used medical and pharmacy claims as source documentation. The advanced approach used medical and pharmacy claims, electronic health records with unstructured data extracted using artificial intelligence methods, and mortality registry data.

Main outcomes and measures: Accuracy was assessed using the F1 score. Completeness was estimated as a weighted mean of available data sources during each calendar year under study for each patient. Traceability was estimated as the proportion of data elements identified in clinical source documentation.

Results: In total, 120 616 patients met the minimum data requirements (mean [SD] age, 43.2 [18.5] years; 41 011 male [34.0%]). For accuracy, traditional approaches had F1 scores of 59.5% and advanced approaches had scores of 93.4%. For completeness, traditional approaches yielded mean scores of 46.1% (95% CI, 38.2%-54.0%); advanced approaches, 96.6% (95% CI, 85.8%-1.1%). For traceability, traditional approaches had 11.5% (95% CI, 11.4%-11.5%) and advanced approaches had 77.3% (95% CI, 77.3%-77.3%) of data elements traceable to clinical source data.

Conclusions and relevance: In this study, practical implementation of data reliability measurement is described. Findings suggest the potential of using multiple data sources and applying advanced methods to increase real-world data reliability. The inclusion of data reliability standards when generating evidence from these sources has the potential to strengthen support for the use of real-world evidence in the prescription, reimbursement, and approval of medications.