A Difficulty-Informed Approach to Developing Language Assessment Literacy for Classroom Purposes

Abstract For pre-service teacher education, it would be helpful to know how difficult certain aspects of language testing and assessment are for students. Such information could serve different pedagogical purposes including program design and lesson sequencing. This paper, situated in the context of pre-service teacher education at the University of Vienna, Austria, presents an approach to developing language assessment literacy (LAL) which takes account of the difficulty of language assessment. To obtain empirical difficulty estimates, the items of a LAL test for pre-service teachers of English were converted into statements of ability, knowledge, and understanding and then calibrated by means of multi-facet Rasch analysis based on performance data from 420 students. Qualitative content analysis was used to identify clusters among the calibrations, thus characterizing the difficulty continuum. The findings show a clear progression which can offer a basis for the design of teacher education programs. I first report on the process of generating the difficulty estimates. Then I describe the resulting difficulty continuum with particular emphasis on classroom-based assessment. Finally, I suggest design principles for difficulty-informed assessment courses, illustrating how they could be implemented in the teacher education program at the University of Vienna.