Reliability of the Reflective Essay Marking Scale (REMS): a scale for marking of students’ reflective essays

Abstract

Abstract Personal practice (PP) is widely practised and a requirement across major psychology and psychotherapy organizations and modalities. However, one of the challenges for training institutions is how to assess the quality of such PP. The Reflective Essay Marking Scale (REMS) was developed to improve standardization of marking reflective essays in cognitive behavioural psychotherapy (CBT) training. A small sample of 16 expert CBT participants recruited by email used the REMS to rate two mock reflective essays in a within-subjects design. The internal consistency of REMS was acceptable (Cronbach’s α=.73) with excellent inter-rater reliability. Across the raters, it sufficiently differentiated quality (t 12=4.91; p<.0001). Although these are the results of a preliminary and very small study with a small sample using mock essays, the REMS may be a useful scale, allowing CBT courses to account for students’ reflective work in a standardized way. A larger validation study is required in the future. Key learning aims (1) To improve the thinking about what raters should focus on when rating the reflective essays of trainee therapists. (2) To describe the development of the scale and how its reliability was tested. (3) To improve the transparency and objectivity in assessing and rating reflective practice.