Increased accessibility of computer-based testing for residency application to a hospital in Brazil with item characteristics comparable to paper-based testing: a psychometric study.

Abstract

Purpose: With the COVID-19 pandemic, online high-stakes exams have become a viable alternative. This study evaluated the feasibility of computer-based testing (CBT) for medical residency applications in Brazil and its impacts on item quality and applicants' access compared to paper-based testing.

Methods: In 2020, an online CBT was conducted in a Ribeirao Preto Clinical Hospital in Brazil. In total, 120 multiple-choice question items were constructed. Two years later, the exam was performed as paper-based testing. Item construction processes were similar for both exams. Difficulty and discrimination indexes, point-biserial coefficient, difficulty, discrimination, guessing parameters, and Cronbach's alpha coefficient were measured based on the item response and classical test theories. Internet stability for applicants was monitored.

Results: In 2020, 4,846 individuals (57.1% female, mean age of 26.64 ± 3.37 years) applied to the residency program, versus 2,196 individuals (55.2% female, mean age of 26.47 ± 3.20 years) in 2022. For CBT, there was an increase of 2,650 (120.7%) applicants, albeit with significant differences in demographic characteristics. There was a significant increase in applicants from more distant and lower-income Brazilian regions, such as the North (5.6% vs. 2.7%) and Northeast (16.9% vs. 9.0%). No significant differences were found in difficulty and discrimination indexes, point-biserial coefficients, and Cronbach's alpha coefficients between the 2 exams.

Conclusion: Online CBT with multiple-choice questions was a viable format for a residency application exam, improving accessibility without compromising exam integrity and quality.