Peter Baldwin, Janet Mee, Victoria Yaneva, Miguel Paniagua, Jean D'Angelo, Kimberly Swygert, Brian E Clauser
{"title":"A Natural-Language-Processing-Based Procedure for Generating Distractors for Multiple-Choice Questions.","authors":"Peter Baldwin, Janet Mee, Victoria Yaneva, Miguel Paniagua, Jean D'Angelo, Kimberly Swygert, Brian E Clauser","doi":"10.1177/01632787211046981","DOIUrl":null,"url":null,"abstract":"<p><p>One of the most challenging aspects of writing multiple-choice test questions is identifying plausible incorrect response options-i.e., distractors. To help with this task, a procedure is introduced that can mine existing item banks for potential distractors by considering the similarities between a new item's stem and answer and the stems and response options for items in the bank. This approach uses natural language processing to measure similarity and requires a substantial pool of items for constructing the generating model. The procedure is demonstrated with data from the United States Medical Licensing Examination (USMLE®). For about half the items in the study, at least one of the top three system-produced candidates matched a human-produced distractor exactly; and for about one quarter of the items, two of the top three candidates matched human-produced distractors. A study was conducted in which a sample of system-produced candidates were shown to 10 experienced item writers. Overall, participants thought about 81% of the candidates were on topic and 56% would help human item writers with the task of writing distractors.</p>","PeriodicalId":12315,"journal":{"name":"Evaluation & the Health Professions","volume":null,"pages":null},"PeriodicalIF":2.2000,"publicationDate":"2022-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Evaluation & the Health Professions","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1177/01632787211046981","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0
Abstract
One of the most challenging aspects of writing multiple-choice test questions is identifying plausible incorrect response options-i.e., distractors. To help with this task, a procedure is introduced that can mine existing item banks for potential distractors by considering the similarities between a new item's stem and answer and the stems and response options for items in the bank. This approach uses natural language processing to measure similarity and requires a substantial pool of items for constructing the generating model. The procedure is demonstrated with data from the United States Medical Licensing Examination (USMLE®). For about half the items in the study, at least one of the top three system-produced candidates matched a human-produced distractor exactly; and for about one quarter of the items, two of the top three candidates matched human-produced distractors. A study was conducted in which a sample of system-produced candidates were shown to 10 experienced item writers. Overall, participants thought about 81% of the candidates were on topic and 56% would help human item writers with the task of writing distractors.
期刊介绍:
Evaluation & the Health Professions is a peer-reviewed, quarterly journal that provides health-related professionals with state-of-the-art methodological, measurement, and statistical tools for conceptualizing the etiology of health promotion and problems, and developing, implementing, and evaluating health programs, teaching and training services, and products that pertain to a myriad of health dimensions. This journal is a member of the Committee on Publication Ethics (COPE). Average time from submission to first decision: 31 days