Influence of pairing in examiner leniency and stringency ('hawk-dove effect') in part II of the European Diploma of Anaesthesiology and Intensive Care: A cohort study.

Background: The European Diploma of Anaesthesiology and Intensive Care (EDAIC) Part II examination is a supranational examination for anaesthesiologists.

Objectives: We explore the impact of examiner pairing on leniency and stringency, commonly referred to as the 'hawk-dove effect'. We investigate the potential variations in grading approaches, resulting from different examiner pairs and their implications for candidate performance.

Design: Retrospective cohort, observational design.

Setting: EDAIC Part II examination data from 2021 to 2023.

Participants: Three hundred and twenty-five examiners across 122 EDAIC Part II single-day examination sessions.

Interventions: We analysed the influence of examiner leniency and examiner pairing on candidate performance in the EDAIC Part II using many-facet Rasch modelling.

Main outcome measures: The study's main outcome measure was determining a leniency score among the examiner population. The study also aimed to assess how examiner pairing influenced candidate performance, as measured by their scores in the examination.

Results: During the study period, the number of examiners who participated in 2021, 2022 and 2023 were 253, 242 and 247, respectively. The median [IQR] single-day sessions attended were 7.0 [3 to 10]. The examination data revealed a mean leniency score of 0 (95% confidence interval (CI) -0.046 to 0.046), with the standard deviation being one-third that of the candidates' ability scores. There were 1424 different pairs of examiners, with most pairs (97%) having only a one-point difference in marking. The mean leniency score for the pair of examiners was -0.053 (95% CI -0.069 to -0.037).

Conclusion: The variations in grading approaches associated with different pairings emphasise the potential for the 'hawk-dove effect' to influence candidate performance and outcomes. Understanding these variations can guide curriculum development, examiner training and coupling, ensuring a balanced and equitable assessment process.

Trial registration: None.