Reconciling assessment quality standards and “double assessment” in competency-based higher education

High standards for assessment practices are essential in all institutions of learning. The role of assessment is arguably even more significant in competency-based education (CBE) institutions since credits and degrees are earned solely based on the demonstration of mastery of competencies through the assessments, and not, as in traditional schooling models, on an average that includes the accumulation of seat time (attendance) and points for activities that do not necessarily indicate competency (e.g., classwork, discussion participation) in addition to assessments.

CBE institutions making the claim that graduates are competent in stated competencies have a responsibility for making the quality of competency assessments a high priority in continual institutional improvement because “in CBE—unlike most traditional programs based on the credit hour—the institution must state with authority that its graduates have demonstrated the learning outcomes required for a degree” (Klein-Collins, 2013, p.7), and “the value of CBE credentials hinges on the reliability and validity of those assessments” in determining graduates' competence (McClarty & Gaertner, 2015, p. 3).

There are commonly accepted standards and best practices for the assessment of learning that apply to all learning models in general as well as assessment concepts that may be specific to the CBE model. One aspect of CBE assessment “best practices,” which was evident in assessment policies and anecdotally in conversations with colleagues at various CBE institutions, was the concept of “double assessment.”

Similar to how the “double jeopardy” clause in the Fifth Amendment of the US Constitution prevents a criminal defendant from being prosecuted more than once for the same crime, a prohibition against “double assessment” in CBE means that once a student has been assessed and has successfully demonstrated mastery of a competency on an assessment, that student should not be assessed on that competency again. “Double assessment” only applies to successful demonstration of mastery of a competency—it does not prohibit or preclude multiple attempts of an assessment when students fail to meet competence on the assessment. Allowing students multiple attempts to pass a competency assessment is a central tenant of CBE.

In addition, “double assessment” is only in reference to summative assessment, which is “conducted to help determine whether a student has attained a certain level of competency” (National Research Council, 2001, p. 40) or “to certify, report on, or evaluate learning” (Brookhart, McTighe, Stiggins, & Wiliam, 2019, p. 6). Using multiple types of formative assessment, or informal assessment that is used to monitor student progress and does not equate to a grade or credit, is common in higher education and viewed as best practice. There is, however, debate over whether using more than one summative assessment to assess students on the same content or learning outcomes is beneficial or whether it is unnecessary and may even inhibit student learning (Beagley & Capaldi, 2016; Domenech, Blazquez, de la Poza, & Munoz-Miquel, 2015; Lawrence, 2013).

The origin of “double assessment” in CBE is difficult to investigate because virtually no literature exists that defines it and explains what it is and what it is not. Literature about assessment best practice in CBE does not specifically and directly address “double assessment”; however, there is some evidence in CBE literature that allows us to infer the purpose of avoiding “double assessment” in CBE programs. For example, a key quality principle that is central to CBE philosophy is that “students advance upon demonstrated mastery” (Sturgis & Casey, 2018, p. 7). Assessing students again on a previously mastered competency could possibly be considered committing “double assessment” because it is preventing students from moving on to a new competency and might be considered the equivalent of seat time or just another hoop to jump through.

Given that CBE is founded on the rejection of seat time as a basis for earning academic credit in exchange for a focus on demonstrated proficiency, CBE program designers strive to eliminate activities that do little to measure proficiency and essentially equate to seat time. To many professionals at CBE institutions, repetition of a competency assessment would not serve the purpose of ensuring mastery of knowledge and skills if mastery has already been demonstrated on an assessment; it would only serve to add time and cost to the students' learning journey. Redundancies in curriculum and assessment that may occur accidentally in traditional, credit- or time-based institutions should be avoided in programs that are intentionally designed around student mastery of distinct competencies (Klein-Collins, 2012). Avoidance of “double assessment” in CBE, then, could be viewed as an effort to eliminate redundancy and reduce the cost of education for students and the institution.

Because “double assessment” is not well defined in literature, it can be interpreted in a variety of ways and perhaps misinterpreted, resulting in practices that hinder rather than promote high-quality competency assessment. For example, some have interpreted “double assessment” to mean that it is against assessment best practice to use more than one type of assessment to assess a single competency, even though using a variety of assessments and collecting multiple samples of evidence when drawing conclusions about students' knowledge are considered assessment best practices (Brookhart et al., 2019; McMillan, 2018; Suskie, 2018). This belief about “double assessment” can result in the use of a single high-stakes assessment to award credit for a competency or even a course when a combination of assessments might actually be needed to draw valid inferences about a particular competency.

The following scenario provides a situation in which more than one form of assessment is desired to draw valid inferences about student proficiency, but in which a misinterpretation of “double assessment” might prevent the best assessment strategy from being used.

According to the book Assessing Student Learning: A Common Sense Guide (3rd edition), an assessment is considered good quality “only if it is the right assessment for the learning goals you want to assess and the decisions you want the resulting evidence to inform” (Suskie, 2018, p. 23). The problem is that any one type of assessment has limitations and in many cases might not be entirely, on its own, the right assessment to provide the needed evidence to “certify” competence (National Research Council, 2001; Suskie, 2018). What if it is determined by experts working on a course that a combination of assessment types is actually needed to obtain the evidence necessary for making valid inferences about student mastery of a competency? “Using a variety of assessments … lets us infer more confidently how well students have achieved key learning goals” (Suskie, 2018, p.28).

Although there are many assessment formats, this paper will focus on two main forms of assessment, selected-response assessment and performance assessment, to compare their benefits and weaknesses.

Selected-response assessments such as multiple-choice, in which students select a correct answer to questions from provided choices, are commonly used because they are objective, and they have the advantage of being auto-graded, which makes them affordable and scalable since they do not require significant faculty time compared to performance assessments. They are also able to provide immediate, automatic feedback about students' performance and, when meaningful feedback is provided, can point students to the areas of the content in which they need remediation. In addition to selected-response assessments' practical advantages, a strategic advantage is that they “do a good job of assessing subject matter and procedural knowledge, and simple understanding, particularly when students must recognize or remember isolated facts, definitions, spellings, concepts, and principles” (McMillan, 2018, p. 77).

Selected-response assessments are, however, limited in their ability to measure a multitude of skills and abilities such as logical reasoning, critical thinking, ethical decision-making, interpersonal “soft” skills, and written communication, just to list a few. Higher education institutions are placing a greater emphasis on assessment activities that promote lifelong learning skills and allow students to tie their learning to real-world problems and contexts so that they can see how their learning will live beyond the classroom (Davidson, 2017).

Specifically in competency-based education, in which there is an inherent focus on application of knowledge in real-world contexts, “a multiple-choice, standardized test is likely inadequate to assess most competencies. Instead, what are required are assignments that present tasks or situations that students will encounter in life and in the workplace” (Klein-Collins, 2013, p.7). A promise of competency-based education is that students will leave the university more competent to enter the workforce because they are required to demonstrate mastery to earn a credential. Objective assessments do not provide students with the opportunity to leave with artifacts showing marketable skills that can be provided to employers as evidence of their competence, nor do they generally give students the opportunity to practice applying skills in a real-world context. They do, however, have some authentic value in programs when students need to pass assessments in a similar format for licensure or certification after graduation, as in teaching, nursing, and accounting, or to gain admittance into graduate school.

Performance assessment, or “open-ended tasks that call upon students to apply their knowledge and skills to create a product or solve a problem” (National Research Council, 2001, p. 29), is seemingly the preferred method of assessment in CBE. There is a big push in CBE institutions to use “authentic” performance assessment as reflected in the Quality Framework for Competency-Based Education Programs (2017) from the Competency-Based Education Network (C-BEN): “Authentic assessments and their corresponding rubrics are key components of CBE, which is anchored by the belief that progress toward a credential should be determined by what learners know and are able to do” (p. 17). But performance assessments also have limitations, such as subjectivity in the evaluation of student performance, and overuse can be taxing for students and faculty who must evaluate the assessments. “Because performance assessments are time intensive for teachers and students, they are usually not the best choice for assessing vast amounts of knowledge” (McMillan, 2018, p. 77). A wide range of facts and terminology, which would easily be assessed using an objective assessment, would not practically or authentically be assessed in a performance task.

This uncertainty is one reason why multiple samples of evidence are preferred to a single assessment for making inferences about student learning.

Standards that are specific to CBE also promote multiple forms of assessment, such as this example from iNACOL's Quality Principles for Competency-Based Education: “Students are empowered and engaged when the process of assessing learning is transparent, timely, draws upon multiple sources of evidence [emphasis added] and communicates progress” (Sturgis & Case y, 2018, p.17). C-BEN includes in its Quality Framework for Competency-Based Education Programs (2017) that CBE models “use a range of assessment types and modalities to measure the transfer of learning and mastery into varied contexts” and that assessments are “designed to provide learners with multiple opportunities and ways to demonstrate competency, including measures for both learning and the ability to apply (or transfer) that learning in novel settings and situations” (p.17). Neither of these sources of CBE assessment best practices state that only one method of assessment can be used to assess a competency.

Figures 1 and 2 below illustrate the concept of assessment instances as “snapshots of behavior” from which educators make estimates or inferences that are “bound to be at least somewhat inaccurate” (Suskie, 2018, p. 28).

During a conference session, the authors presented the photograph in Figure 1 to illustrate how assessment “snapshots” provide limited information (Tkatchov & Hugus, 2019). Participants were asked to make a judgment about where the photographer was located when taking the photograph in Figure 1. Responses included “in an airplane” and “in a field outside.”

[Correction added on July 11, 2020, after first online publication: The blinded text has been replaced with the reference citation (Tkatchov & Hugus, 2019).] Next, participants were shown a second photograph (Figure 2) that provides additional information. With new information from a different angle, participants were asked to make a judgment as to where the photographer was when taking the photograph. With new information to include evidence that situated the photographer inside a building, responses changed to “in a house or building looking out a window.” Having more information, or a second snapshot to complement the first one, allowed the participants to make a more accurate judgment as to where the photographer was when taking the photograph. The second photograph provided more breadth to give the viewer a better snapshot of the photographer's location, but it loses some depth and detail of the clouds that was apparent in the first photo.

As illustrated with the two photographs, a variety of assessment formats, such as a selected response in combination with a performance assessment, can be combined to complement each other or to supplement each other's deficiencies. They can also be used to address the different cognitive levels that are represented in a competency. “At lower levels of competence, multiple-choice and other tests of objective learning may be appropriate. At higher levels of competence, however, getting at more complex and analytical thinking requires different kinds of assessment such as student narratives, demonstrations, simulations, or performance-based assignments” (Klein-Collins, 2013, p. 12). A complementary assessment strategy for competencies that encompass theory and application will capture the breadth of knowledge (the recall level) as well as the depth of knowledge (the application level), and more than one assessment format might be combined to accomplish a more complete picture of learners' competency.

Despite the stated inadequacy of using only multiple-choice assessments for measuring most competencies, multiple-choice assessments are often the preferred assessment format because they are scalable at low cost. Once a multiple-choice assessment is developed, that assessment can be used to assess tens of thousands or, even, hundreds of thousands of students with very small incremental variable cost. Most of the cost of multiple-choice assessment is in the front-end development of the assessment itself. When contrasted against the high cost of evaluating task-based or performance assessments, this significant difference in per-student evaluation cost can bias budget-conscious institutions toward multiple-choice assessment.

Institutions must balance their own assessment cost considerations with how those considerations impact their students. Not all decisions that increase assessment cost are bad for students, and increases in the cost of assessment do not have to be passed on to students. Cost-conscious institutions can frequently lever cost reductions in other areas and maintain existing cost levels. Just because a decision is cost-conscious for the institution, if it degrades the quality of student learning or the quality of student assessment—increasing the likelihood of inaccurate assumptions regarding students' competence—then that cost-conscious decision represents a disservice to students, to the institution, and to all competency-based education. The proper balance of cost and assessment quality is a CBE institution's ethical responsibility. Failing to properly balance this ethical responsibility runs the risk that employers will lose confidence in competency-based degrees and credentials and be reluctant to hire CBE graduates.

A complementary assessment strategy that combines assessment formats might be necessary to make a valid judgment about the students' competency in conversing about the weather in French. Students would need to demonstrate their ability to have a conversation about the weather in French in a performance assessment task, especially to capture correct pronunciation (outcome 3) and the ability to give appropriate responses to questions and comments (outcome 4). The performance assessment would allow for the students to demonstrate depth of knowledge at the application level, but it would not be practical to expect the students to perform, or the faculty to evaluate, conversations in every conceivable weather-related situation. To assess the range of vocabulary and make an inference about students' ability to transfer their learning in a variety of situations, a selected-response, objective assessment might be used to capture the breadth of students' knowledge of weather-related vocabulary (outcome 1) and correct sentence structure (outcome 2) at the recall level.

In this case, at least two assessment formats would be used to create a single assessment strategy to assess a competency. Half of the competency would be assessed at the lower level but over a broad range of content, while the other half would be assessed at a higher level of application but over a narrower range of content. This strategy would be more scalable than giving the students a multitude of performance assessments, but it would be more dependable than relying on only one assessment.

Redundancy in assessment adds unnecessary time to degree completion, which also increases the cost of tuition. In addition, assessment practices that are overly burdensome for faculty can also place too much of a financial burden on an institution and, ultimately, the students. Therefore, CBE institutions are wise to avoid redundancy in assessment whenever possible and to prioritize the scalability of assessments. However, there must be a balance in competency-based higher education between minimizing the cost of education and maintaining high-quality assessment practices that give employers confidence in the legitimacy of competency-based credentials.

When literature resoundingly supports the use of multiple forms of assessment as a best practice, restricting assessment practices in CBE to only one form of assessment based on an ill-defined and unsubstantiated “double assessment” rule can have the opposite of its intended effect. It can reduce, not enhance, the quality of competency assessments and the validity of inferences about student learning derived from those assessments.

More discussion is needed among the CBE community about how higher education institutions can deliver on CBE's promise of providing an affordable and high-quality education that prepares students for life and work after graduation. Policies intended to manage cost and scalability in assessment should be counterbalanced by safeguards intended to ensure quality, such as allowances for exceptions when competencies call for a greater investment in assessment or are best assessed through multiple modalities. Further research and experimentation in scalable performance assessments and hybrid assessments that combine formats are important next steps in CBE innovation.