Evaluating Language Assessments From an Ethics Perspective: Suggestions for a New Agenda

Abstract

Introduction The dominant 20th century approach to the evaluation of language assessments was the Standards-based approach. The Standards most evaluators referred to are the American Psychological Association (APA), American Educational Research Association (AERA), National Council on Measurement in Education (NCME) Standards (1999, 2014). These standards (mainly a list of test qualities such as validity and reliability, and of late, consequences and fairness) were developed from best practices at assessment institutions and had loose connections to theories of educational and psychological measurement. The “Test Usefulness” concept proposed by Bachman and Palmer (1996) was a popular example of the Standards approach. In the early part of the 21st century, Kane (1992) and Bachman and Palmer (2010) proposed an Argument-based approach using Toulmin’s way of structuring arguments with claims, warrants, backing and rebuttals. This approach provided a framework for evaluating language assessments. Bachman and Palmer’s (2010) “Assessment Use Argument” (AUA) is an example of this approach. While both approaches provide ways for researchers to conduct evaluations, they have a weakness, and that is they generally lack an articulated philosophical grounding. This lack of philosophical grounding can be seen in the Standards approach in which why the listed standards are important and not others is not articulated. In the Argument approach, what aspects are to be included as claims and warrants is left the assessment developer with the evaluator following them which is a critical problem. To remedy this situation, I am proposing an Ethics-based approach to assessment evaluation. The framework that implements the approach harnesses the dual concepts of fair assessments and just institutions leading to the Principle of Fairness and Principle of Justice, respectively.