‘Good’ evaluation: Methodological diversity may be empress, but sound methods remain queen

The appetite for evaluation of health professions education (HPE) programmes appears insatiable. Mandates to evaluate prevail and administrators persist in their quests for data to inform decision-making. Understandably, people want to know if their efforts ‘move the needle’ for today's learners and tomorrow's clinicians. Regulatory bodies want assurances that accreditation standards are met. And, communities deserve evidence-grounded accountability from those charged with training clinician workforces. Interestingly, however, the unabating appetite for evaluation exists in contrast to relatively modest scholarship or capacity-building on evaluation across HPE. Thus, it is encouraging to see contributions such as the one from Rees et al.¹ in this issue of Medical Education.

Rees et al.¹ present a comprehensive overview of the realist interview method. Following critical analysis of select realist evaluation studies, they outline best practice recommendations for applying realist interview techniques to realist evaluation. This work is timely, given the increasing popularity of realist evaluation across HPE, which may be a promising indicator of methodological expansion in the field's evaluation practices. Indeed, several scholars have critiqued the field's seeming preoccupation with simple outcome verification as the ‘go-to’ evaluation strategy.^2-6 Figuratively, the crux of this critique has been that when it comes to evaluation, HPE has focused on the trees—familiar, uniform trees—while neglecting the diversity of the forest. Indeed, I contend that embracing methodological diversity is the essence of ‘good’ evaluation. Not only does it enable the exploration of novel lines of inquiry, it also facilitates needed reflection on or redirection of how we are using evaluation to assign value (see Gates⁷). Conversely, asking similar questions of similar objects in routinised ways cannot satiate the appetite for new insights or innovative, evidence-informed, shifts away from a sub-optimal status quo across our academic healthcare systems.^8-10

As such, Rees et al.¹ present a strong value proposition for how realist evaluation expands possibilities for evaluative inquiry. Using examples, they illustrate the many avenues that emerge when a realist approach is used to explore how programme contexts may variably trigger change mechanisms and influence differences in outcomes. However, championing the merits of methodological diversity in evaluation is incidental to their aims. Rather, their core argument is best summarised as follows: The uptake of contemporary evaluation methodologies is of limited value if it is not accompanied by proper implementation of associated methods. Simply put, evaluators must adopt sound methods with fidelity to their espoused methodologies.

Additional points stand out.

One, not all interview methods are equal. Though interviews are a common data collection method, there are unique features of interview techniques across different research traditions and evaluation methodologies. For example, Rees et al.¹ explain how realist interviewing differs from interviewing within certain qualitative traditions where the focus is on eliciting participants' experiences on a given topic. In contrast, realist interview techniques are designed to elicit information to support theory building and testing of assumptions about programme functioning.

Two, realist interviewing requires balancing the tension of being both teacher and learner in the service of credible theory building. Rees et al.¹ describe the ‘teacher–learner cycle’ as a core tenet of realist interviewing. Realist interviewers must hold knowledge about both the intervention and about theoretical linkages that feasibly explain its functioning. Interviewers must share some of this insight with interviewees, while maintaining the flexibility and humility required to challenge their assumptions and refine theories in response to insights from interviewees' experiential knowledge.

Three, a diverse sampling frame across those who are conversant with the programme is desirable. Rees et al.¹ contend that whenever there is an indication that a programme may function differently for different stakeholder groups or across different settings, then it is particularly important that realist interviewers ensure participant diversity. At first glance, this guidance seems spot-on. Evaluators need to recruit a diverse participant pool by sampling those who have varied insights about programmes. Moreover, it is arguably an ethical imperative to recruit participants who would just as readily discuss ‘why the programme doesn't work’ and ‘how the programme could fail’ as they would share ‘how it works and for whom’. Upon a closer look, however, the sampling guidance surfaces a notable limitation of realist interviewing.

Qualifying for participation in the ways the authors' outline requires that the ideal target participant meet multiple criteria. These seemingly include (i) significant literacy about the programme, (ii) capacity to verbally explain how potentially nuanced experiences may trigger one outcome versus another, (iii) cognitive capacity to understand and accurately interpret interviewers' assumptions about programme functioning, (iv) availability and willingness to support refinement of potentially abstract theories about a complex intervention and (v) ability to identify realist interviewers' potentially erroneous assumptions about programme functioning, plus (vi) the confidence to challenge such assumptions in real-time. Alas! Realist interview method's appeals for participant diversity appear tough to reconcile with its seeming demand for erudite or otherwise socially assured participants with the gift of both time and articulation. Such sampling requirements could lead to equity gaps, inadvertently privileging certain perspectives over others. Indeed, this issue sorely highlights the necessity of their argument for improving evaluation reporting.

Following a review of select realist studies, Rees et al.¹ spotlight transparency in evaluation reporting—as both a gap—and best practice imperative. They note that common transparency gaps include scholars' failures to report sampling approaches or interview techniques. This message must not be discounted. Embracing methodological diversity is necessary but insufficient for high-quality evaluation. Essentially, it is inadequate to simply value evaluation methodology as a gauge that indicates whether our ‘needle’ is shifting. We must also take on the responsibility to monitor the quality and operations of the gauge itself. Rees et al. help us do that, by instructing us on how to ‘operate’ the realist interview method. Perhaps, next, scholars can explore how complementary methods can mitigate the approach's limitations.

Betty Onyura: Conceptualization; writing—review and editing; writing—original draft.