Evaluating a learning health system initiative: Lessons learned during COVID-19 in Saskatchewan, Canada

Abstract

Introduction

Evaluating a learning health system (LHS) encourages continuous system improvement and collaboration within the healthcare system. Although LHS is a widely accepted concept, there is little knowledge about evaluating an LHS. To explore the outputs and outcomes of an LHS model, we evaluated the COVID-19 Evidence Support Team (CEST) in Saskatchewan, Canada, an initiative to rapidly review scientific evidence about COVID-19 for decision-making. By evaluating this program during its formation, we explored how and to what extent the CEST initiative was used by stakeholders. An additional study aim was to understand how CEST could be applied as a functional LHS and the value of similar knowledge-to-action cycles.

Methods

Using a formative evaluation design, we conducted qualitative interviews with key informants (KIs) who were involved with COVID-19 response strategies in Saskatchewan. Transcripts were analyzed using reflexive thematic analysis to identify key themes. A program logic model was created to represent the inputs, activities, outputs, and outcomes of the CEST initiative.

Results

Interview data from 11 KIs were collated under three overarching categories: (1) outputs, (2) short-term outcomes, and (3) long-term outcomes from the CEST initiative. Overall, participants found the CEST initiative improved speed and access to reliable information, supported and influenced decision-making and public health strategies, leveraged partnerships, increased confidence and reassurance, and challenged misinformation. Themes relating to the long-term outcomes of the initiative included improving coordination, awareness, and using good judgment and planning to integrate CEST sustainably into the health system.

Conclusion

This formative evaluation demonstrated that CEST was a valued program and a promising LHS model for Saskatchewan. The future direction involves addressing program recommendations to implement this model as a functional LHS in Saskatchewan.