Refresh of a Clinical Skills Assessment for Physician Trainees

Purpose: The Royal Australasian College of Physicians (RACP) oversees physician training across Australia and Aotearoa New Zealand. Success in a written examination and clinical skills assessment (known as the clinical examination) at the mid-point of training is a requirement to progress from basic to advanced training. The clinical examination had evolved over many years without a review process. This paper describes the approach taken, the changes made and the evaluation undertaken as part of a formal review.
Methods: A working party that included education experts and examiners experienced in the assessment of clinical skills was established. The purpose of the clinical examination and competencies being assessed were clarified and were linked to learning objectives. Significant changes to the marking and scoring approaches resulted in a more holistic approach to the assessment of candidate performance with greater transparency of standards. Evaluation over a 2-year period was undertaken before the adoption of the new approach in 2019.
Results: In 2017 testing of a new marking rubric occurred during the annual examination cycle which confirmed feasibility and acceptability. The following year an extensive trial utilising the new marking rubric and a new scoring approach took place involving 1142 examiners, 880 candidates and 5280 scoresheets which led to some minor modifications to the scoring system. The final marking and scoring approaches resulted in unchanged pass rates and improved inter-rater reliability. Feedback from examiners confirmed that the new marking and scoring approaches were easier to use and enabled better feedback on performance for candidates.
Conclusion: The refresh of the RACP clinical examination has resulted in an assessment that has clarity of purpose, is linked to learning objectives, has greater transparency of expected standards, has improved inter-rater reliability, is well accepted by examiners and enables feedback on examination performance to candidates.

Keywords: physician assessment, reliability, clinical assessment, long case, short case