Whose voices are heard in health professions education validity arguments?

Abstract

The construction of validity arguments for assessments in health professions education (HPE) has been likened to a lawyer preparing for and presenting a case.¹ Like a lawyer curates a brief of evidence with the aim of convincing a judge or jury to make a particular decision about their client, so are health professions educators required to provide validity evidence that supports a defensible decision about a student being assessed.¹ Kane's argument-based validity framework,² now expanded by scholars in language testing and assessment (LTA), addresses challenges of prior conceptualisations of validity by providing a staged approach to building a validation argument according to ‘inferences’. Whereas Kane's original four inference model commences with scoring,² the expanded seven inference model as described in this issue by Dai et al.³ in LTA commences with a domain description.

The goal of the domain description is to ensure that the ‘selection, design and delivery of the test tasks takes the relevant target domain into account’.³ Described sources of backing for this inference include interviews or surveys of domain insiders. Starting with a domain description should provide a solid foundation for subsequent inferences made about the assessment but also begs the question of who are considered domain insiders. And whether insights from diverse groups with insider perspectives can together build a more robust and nuanced validity argument. Returning to the analogy of the lawyer's decision-making processes, there may be multiple witnesses with evidence to share, but which witnesses are called upon to provide evidence in court? Or alternatively, which witnesses are not selected out of concern that their differing perspectives may threaten the lawyer's plan for the case?

Domain insiders could be considered ‘expert witnesses’, that is, those with subject matter expertise typically built through education and professional experience, such as health professions educators with clinical and/or pedagogical expertise. While subject matter expertise is important to understanding whether assessment tasks sufficiently reflect the domain being assessed, potential differences between expert and novice (i.e., student) understandings of a domain could disrupt a validity argument. Consider assessments of uncertainty tolerance (UT): Commonly used UT scales intended to measure UT in healthcare contexts engaged expertise during scale development in the form of interviews with health professionals, reviews of construct literature and consultation with medical educator peers.⁴ One UT scale has been used by the Association of American Medical Colleges as part of routine matriculation and graduation surveys of medical students, with the intent that the results inform medical school programmatic evaluation.⁴ However, the perspectives of medical students were not included as part of the validity argument for this scale.⁴

Research on the UT construct more broadly highlights potential differences between expert and student conceptions of uncertainty, with students' conceptions of uncertainty focussed on individual knowledge gaps rather than uncertainty inherent within patient care more typical of clinical experts.⁵ By not including medical student perspectives as part of the validity argument, the meaning of scale results could potentially be misinterpreted. For example, an increase in UT scores from matriculation to graduation could represent that knowledge gaps have been filled and perceived uncertainty reduced, and not that students are better at ‘tolerating’ or managing uncertainty, an essential graduate skill needed to manage the dynamic, complex and unpredictable nature of healthcare. Hence, including test takers as domain insiders or ‘witnesses’ early in constructing a validity argument may enable those developing an assessment to proactively identify early threats to their argument before evaluating subsequent inferences in the framework.

An example of an assessment where diverse ‘witnesses’ enabled inferences to be made about the domain description, and where the field of LTA and HPE intersect, is the Occupational English Test (OET).⁶ The OET is an English-language test designed for health professionals originally developed by LTA experts and includes writing a letter of referral to assess written clinical communication.⁷ While updating and modifying the assessment criteria for the written component of the test, applied linguists, LTA experts and HPE researchers collaborated with health professionals (domain insiders) through an interview and focus group study with clinicians and other stakeholders to identify criteria indigenous to healthcare settings.^{6, 7}

The differing expertise of members of the research team enabled them to make recommendations about modifications to the assessment criteria based on the data.⁶ For instance, the health professionals were vital for ensuring the establishment of professionally relevant criteria, whereas the LTA experts were crucial for defining levels of performance.⁶ When analysing data, coding schemes were initially developed by the LTA experts and then refined through collaboration with the health professionals. The researchers in this study were transparent about how collaboration posed some challenges; however, they concluded that it ultimately enabled a greater depth and understanding of the complexity of the domain being assessed, and awareness of this complexity would not have been achieved without collaboration.⁶ Such insights identified through the process of domain description for the OET could in turn be used to enhance HPE (e.g., workshops on clinical note writing for final-year health professions students).

Collaborative approaches to understanding the perspectives of domain insiders may lead to richer and more robust validity arguments and assessments when compared with those centred on ‘expert witnesses’. Just as neglecting to call a key witness to the stand would be considered an unacceptable omission by a lawyer, the field of HPE should more broadly consider whose voices are valued and included in validity arguments so that important insights into domains are not overlooked. This can be achieved by inviting testimony from diverse stakeholders, including health professions students and patients (healthcare consumers), on what is important to and valued by them so that HPE aligns with community needs, values and expectations of healthcare. The cross-cutting edge paper by Dai et al.³ provides a timely reminder of how interdisciplinary dialogue (in this case, between HPE and LTA) can expand the epistemological landscape in HPE and research to generate new ways of thinking about validation practice in HPE.⁸ HPE in general is moving towards transformational approaches that integrate student and patient perspectives, as exemplified by the growing use of co-design methodology.^{9, 10} Co-design is underpinned by principles of inclusive, respectful, participative, iterative and outcomes-focussed consumer engagement.⁹ Although the use of co-design methodology in assessment is currently limited,¹⁰ co-design principles may be well suited to navigating how diverse assessment stakeholder perspectives can be explored and successfully integrated within a domain description and ultimately inform the development of curricula and assessments.

Returning to the opening analogy likening validity arguments to a lawyer constructing and presenting a case, it may be helpful for health professions researchers and educators to think more broadly about who is (or is not) given the opportunity to contribute to building validity evidence. For HPE, this may involve including ‘witnesses’ with diverse lived and learned expertise (e.g., students, patients and practicing clinicians) in collaboration with researchers with diverse backgrounds (e.g., interdisciplinary collaboration between HPE and LTA researchers). By broadly considering who is a domain insider, health professions educators and researchers may facilitate a more nuanced and holistic domain description, more robust arguments for the interpretation of test scores and ultimately the creation of richer and more relevant assessments with clear benefits for health professions educators, researchers, students and the wider community.

Georgina C. Stephens: Conceptualization; writing—original draft. Gabrielle Brand: Conceptualization; writing—original draft. Sharon Yahalom: Conceptualization; writing—original draft.