Canadian medical education journal最新文献

OSCEai Dermatology demonstrates how large language models (LLMs) like GPT-4 can be integrated into medical education to enhance trainees' history taking and management skills in an OSCE-like format, including in visual-based specialties like dermatology. By generating diverse, realistic skin cancer role-play scenarios across different skin tones alongside the integration of pre-existing, evidence-based images, the app provides learners with valuable, personalized feedback. This innovation offers a novel, interactive learning tool that supplements traditional teaching methods and can be applied across various specialties. Institutions can adopt or adapt similar LLM-driven educational tools to introduce trainees to a wider range of clinical cases, fostering improved diagnostic skills and patient-centred, culturally sensitive care.

Background: Personal statements play a large role in pediatric residency applications, providing insights into candidates' motivations, experiences, and fit for the program. With large language models (LLMs) such as Chat Generative Pre-trained Transformer (ChatGPT), concerns have arisen regarding how this may influence the authenticity of statements in evaluating candidates. This study investigates the efficacy and perceived authenticity of LLM-generated personal statements compared to human-generated statements in residency applications.

Methods: We conducted a blinded study comparing 30 ChatGPT-generated personal statements with 30 human-written statements. Four pediatric faculty raters assessed each statement using a standardized 10-point rubric. We analyzed the data using linear mixed-effects models, a chi-square sensitivity analysis, an evaluation of rater accuracy in identifying statement origin as well as consistency of scores amongst raters using intraclass correlation coefficients (ICC).

Results: There was no significant difference in mean scores between AI and human-written statements. Raters could only identify the source of a letter (AI or human) with 59% accuracy. There was considerable disagreement in scores between raters as indicated by negative ICCs.

Conclusions: AI-generated statements were rated similarly to human-authored statements and were indistinguishable by reviewers, highlighting the sophistication of these LLM models and the challenge in detecting their use. Furthermore, scores varied substantially between reviewers. As AI becomes increasingly used in application processes, it is imperative to examine its implications in the overall evaluation of applicants.

Background: This study aims to evaluate the effectiveness of the OSCEai, a large language model-based platform that simulates clinical encounters, in enhancing undergraduate medical education.

Methods: A web-based application, OSCEai, was developed to bridge theoretical and practical learning. Following use, medical students from the University of Calgary Class of 2026 completed an anonymized survey on the usability, utility, and overall experience of OSCEai.

Results: A total of 37 respondents answered the anonymized survey. The OSCEai platform was highly valued for its ability to provide data on demand (33/37), support self-paced learning (30/37), and offer realistic patient interactions (29/37). The ease of use and medical content quality were rated at 4.73 (95% CI: 4.58 to 4.88) and 4.70 (95% CI: 4.55 to 4.86) out of 5, respectively. Some participants (8/37) commented that few cases were not representative and needed clarification about app functionality. Despite these limitations, OSCEai was favorably compared to lecture-based teaching methods, with an overall reception rating of 4.62 (95% CI: 4.46 to 4.79) out of 5.

Interpretation: The OSCEai platform fills a gap in medical training through its scalable, interactive, and personalized design. The findings suggest that integrating technologies, like OSCEai, into medical curricula can enhance the quality and efficacy of medical education.

Background: Medical school applications often require short written essays or personal statements, which are purportedly used to assess professional qualities related to the practice of medicine. With generative artificial intelligence (AI) tools capable of supplementing or replacing inputs by human applicants, concerns about how these tools impact written assessments are growing. This study explores how AI influences the ratings of essays used for medical school admissions.

Methods: A within-subject experimental design was employed. Eight participants (academic clinicians, faculty researchers, medical students, and a community member) rated essays written by 24 undergraduate students and recent graduates from McMaster University. The students were divided into four groups: medical school aspirants with AI assistance (ASP-AI), aspirants without AI assistance (ASP), non-aspirants with AI assistance (NASP-AI), and essays generated solely by ChatGPT 3.5 (AI-ONLY). Participants were provided training in the application of single Likert scale tool before rating. Differences in ratings by writer group were determined via one-way between group ANOVA.

Results: Analyses revealed no statistically significant differences in ratings across the four writer groups (p = .358). The intraclass correlation coefficient was .147.

Conclusion: The proliferation of AI adds to prevailing questions about the value personal statements and essays have in supporting applicant selection. We speculate that these assessments hold less value than ever in providing authentic insight into applicant attributes. In this context, we suggest that medical schools move away from the use of essays in their admissions processes.