Flawed multiple-choice questions put on the scale: What is their impact on students′ achievement in a final undergraduate surgical examination?

Abstract

Background : Violation of item-writing guidelines is still frequently encountered in assessments in medical colleges. Flawed multiple-choice (MC) items affect students′ performance and distort examinations′ results. Aims: The aim was to assess the frequency and impact of flawed MC items on students′ achievement in our setting. Settings and Design: This is a quantitative descriptive study conducted at the Faculty of Medicine in the University of Tabuk, Saudi Arabia. Methods: We evaluated a 100 single-correct answer MC questions summative surgical examination administered to 44 6 th year final medical students in November 2014. MC items, which contain one or more violation of item-writing guidelines, were classified as flawed, those with no violation, were classified as standard. The passing rates and median scores of high- and low-achieving students were calculated on both standard and flawed test scales. Item performance parameters (difficulty index, discrimination power and internal consistency reliability (Kuder-Richardson formula 20) were calculated for standard and flawed items. Descriptive and comparative statistics with the relevant tests of significance were performed using the  SPSS (IBM SPSS Inc. Chicago, Illinois) computer software version 16. Results: Thirty-nine flawed items were identified (39%) which contain 49 violations of the item-writing guidelines. The passing rate was 93.2% and 91.8% on the total and standard scales, respectively. Flawed items benefited low-achieving students and disadvantaged the high-achieving students. Overall, flawed items were less difficult, less discriminating and less reliable than standard items. Conclusions: The frequency of flawed items in our examination was high and reflects the need for more training and faculty development programmes.