Developing errant paths in a simulation testing environment: A how to guide for assessment professionals

In competency-based education (CBE), students move through their degree programs one course at a time by demonstrating mastery through a variety of content and professional domains. Institutions of higher learning are beginning to realize the importance of real-world, performance-based assessments like computer simulations, where demonstrations of competence are key to success. The greatest advantage of this mode of testing is the real-world relevance that can be incorporated into the assessment, and technological innovations continue to expand these opportunities. However, without a standard-based approach, computer simulation examinations will remain challenging for those institutions.

Central to the development of computer simulations is the need for high-fidelity assessments of learner aptitudes and competences that include a significant emphasis on not only cognitive abilities (knowing what) but also the performance of those abilities (demonstrating how). As students work within an individual domain of knowledge and skill, they are assessed on one or more topics, each consisting of a series of competencies with associated test objectives and performance tasks (Gyll & Ragland, 2018). As the popularity of CBE programs continues to grow, its integrity will be scrutinized by students and employers alike; and its credibility is largely dependent upon the quality of the assessments that are used (McClarty & Gaertner, 2015).

A simulation is a mock representation of a real-world process or system that represents the key characteristics of the system, such as its behavior, functions, and physical properties. A simulation is like a case study (the examination of a problem used to illustrate a thesis or principle), but with the participants on the inside, not on the outside. Simulations are used across many disciplines. For example, stochastic (probabilistic) simulations are used when some variable or process is estimated based on a statistical technique, such as can be used to show the eventual real effects or forecasted conditions of weather patterns. The military uses for simulation can involve the use of aircraft simulators for training pilots or simulated armored fighting vehicles that imitate real-world combat conditions. Similarly, simulations are an essential part of today's medical education, as many universities have simulation centers (or skill laboratories), where students can practice diagnostics and procedures on life-like mannequins (Roterman-Konieczna, 2015).

In many instances, the differences between computer simulations and live environments seem trivial. In fact, many computer simulations have gotten so sophisticated, that, if optimized, they could serve as the actual workstation (i.e., a computer system designed for individual use). In some instances, however, it may neither be possible nor desirable to simulate the full functionality of an application. For example, when development costs increase in relationship to the number of simulation paths. Although there are a number of benefits to computer simulation testing including reduced development time and administration costs, less sophisticated scoring rubrics, and increased stability over testing occasions, several concerns remain unanswered.

The purpose of this paper was to discuss one of the primary dilemmas impeding the development of high-fidelity computer simulation examinations; chiefly, the determination of the appropriate number of errant paths that render a computer simulation examination valid. Here, validity refers to the type of external validity or generalizability that Mitchell and Jolley (2001) describe as the conclusions of a scientific study outside the context of that study. In other words, the extent to which the results of the computer simulation will generalize to and across real-world applications. This paper briefly explores the history of simulations as examinations and differentiates between low versus high-fidelity assessments in a simulation environment. End-user navigation requirements and its relationship to developing the appropriate number of errant paths within a computer simulation is also explored. Finally, several tools and templates are provided to aid assessment professionals in the development process.1

Since the early 2000s, there have been attempts to evaluate various aspects of human performance using simulation technology. Most of these studies focused on the cognitive, perceptual, and motor performance of end users, and paid little attention to simulation fidelity and its relationship to errant paths; that is, the look, feel, and functionality of a computer simulation and how end users navigate within the simulation environment (Nash, Edwards, Thompson, & Barfield, 2000). For the purpose of this paper, the definition of an errant path is operationalized as “any pathway leading an end user further away from a correct response in a computer simulation environment.”

Computer simulations are now reasonably well-established in the certification and educational fields as an assessment tool for learning. The technique fits well into educational philosophies that stress the importance of the learner as an interactive participant in the learning process. Some useful examples of how computer simulation examinations are being used at one competency-based institution include CompTIA's performance-based assessments (A⁺, Network⁺, Security⁺, Cloud⁺, and Cybersecurity Analyst⁺), which assess candidates' ability to solve problems in a simulated environment.

The simulation examination is often described by misleading labels, such as game, case study, or exercise. Perhaps the most useful approach, therefore, is to put forward a definition and description of the computer simulation followed by some examples of how and why it is used for assessment purposes both inside and outside of education. It is important to note that a simulated environment is very different from a virtual reality (VR) one. In VR environments, users react to certain simulated conditions within the environment (e.g., aircraft simulators), but the environment is completely imaginative and false. This “false” environment is created to appear very real, yet users can distinguish between reality and imagination (Keshav, 2017). In order for it to be a simulation, there can be no real interaction with a genuine environment. An information technology learner demonstrating the functionality of a software application through Power Point is not participating in a simulation. To be a computer simulation, the relevant functions of the software environment must be presented adequately through the computer equipment. Without reality of function, the activity is no more than an exercise or Power Point presentation.

Reality of function includes the acceptance of the relevant duties and responsibilities of the simulation. For example, if a simulation is about the work of news editors on social media then the materials will provide news items for the participants to work on within a social media platform. However, if a teacher tells the class, “pretend you are news editors on social media and invent your own news items,” then there would be no reality of function; the participants would be authors and inventors, not editors.

Developing high-fidelity simulations involves the understanding of navigational awareness and how learners acquire knowledge. Knowledge acquisition is typically measured at the task level (i.e., what the user is doing within an environment) by assessing speed and accuracy in participant response. As a rule of thumb, as knowledge increases both speed and accuracy also increase (Fitts, 1964). The pre-eminent method for assessing speed/accuracy performance, especially within highly complex tasks involving multiple decision paths, is click-stream analysis. Within this method of click-stream analysis, end-user response patterns are analyzed by measuring post hoc data from frequently travelled paths and locations to which the participant frequently returned. Since these methods often prove costly and time consuming, techniques that utilize qualitative procedures like training situation analysis (TSA) and subject matter expert (SME) judgments represent an alternative approach. The remaining sections of this paper focus on this TSA and SME approach.

Although not specifically designed to assess navigational tracking and errant paths in computer simulation testing, a technique that has proven useful and generalizable to errant path identification is TSA. TSA includes a systematic process for identifying tasks that comprise a job, describing in detail the operational components of each of those tasks, ranking the importance in cost-benefit terms of each task component, and describing the functional training requirements of the tasks (Jonassen, Tessmer, & Hannum, 1989).

Training situation analysis describes in detail the human performance for which systems design should be developed. These techniques were generalized to include test development processes performed by SMEs during the item writing and/or product task analysis phases. The following details the steps required to develop a computer simulation examination using a typical office product graphical user interface, comprised of menu, dropdown, and command prompt options. Appendices A and B include useful templates to aid in the development process.

Computer simulations as examinations represent a much-needed effort to move beyond the shortcomings of today's form-based assessments and an opportunity to reengineer the way in which we measure performance. Within computer simulation examinations, we assess for competency and problem-solving skills versus the content memorization typically supported by multiple-choice assessments. Certainly, competency-based institutions are looking for ways to add value to their credentials and all are unanimous in recognizing that asking test takers to do something is far more indicative of skills and ability than asking them to remember and recall something.

For any assessment program, construct validity is the paramount component of validity evidence. Put into context, this means that in order to support the inferences drawn from simulation examination scores, standard-based principles that focus on high fidelity must be maintained throughout the development life cycle. In this paper, several useful prototypes to aid assessment professionals in the development of computer simulation examinations are provided. Computer simulation examinations offer several benefits over traditional methods including reduced development time and administration costs, less sophisticated scoring rubrics, and increased stability over testing occasions.

An even more critical advantage of computer simulation examinations rests in its ability to assess higher-order cognitive skill within a simulation environment. Today, access to some kind of hands-on encounter with systems and software has become an essential part of the learning experience, especially within CBE programs where demonstrations of competence are required for success. Be it in the context of a diagnostic pre-assessment or simply while preparing to tackle examination objectives where test takers must go interactive, use of computer simulations as examinations are now both expected and routine.

Within CBE, a necessary part of taking responsibility for students' development is the ability to identify what standards should appropriately apply. Unless the expectation is set that raising questions about appropriate standards is a normal part of approaching any learning task, this is unlikely to be sufficiently developed. As discussed in this paper, it appears that the range, functionality, power, and utility of computer simulation examinations will continue to expand into the foreseeable future, especially as the popularity and legitimacy of CBE programs continue to grow. However, without an established set of standards, this loose structure provides little room for a consensus to emerge as to which method provides the best measure of validity evidence within a computer simulation environment.

No conflicts declared.