How and how well do students reflect?: multi-dimensional automated reflection assessment in health professions education

Abstract

Reflection assessment is a critical component of health professions education that can be used for personalized learning support. However, reflection assessment at scale remains a challenge due to the demanding nature of tasks and the common use of simplified criteria of quality. This study addressed this issue by developing a multi-dimensional automated assessment that uses linguistic models to classify reflections by overall quality (depth) and the presence of six constituent elements denoting quality (description, analysis, feeling, perspective, evaluation, and outcome). 1500 reflections from 369 dental students were manually coded to establish ground truth. Classifiers for each of the six elements were trained and tested based on linguistic features extracted using the LIWC tool applying both single-label and multi-label classification approaches. Classifiers for depth were built both directly from linguistic features and based on the presence of the six elements. Results showed that linguistic modeling can be used to reliably detect the presence of reflection elements and the level of depth. However, the depth classifier showed a heavy reliance on cognitive elements (description, analysis, and evaluation) rather than the others. These findings indicate the feasibility of implementing multidimensional automated assessment in health professions education and the need to reconsider how quality of reflection is conceptualized.