Carl Ehrett, Sudeep Hegde, Kwame Andre, Dixizi Liu, Timothy Wilson
Background: Generative large language models (LLMs) have the potential to revolutionize medical education by generating tailored learning materials, enhancing teaching efficiency, and improving learner engagement. However, the application of LLMs in health care settings, particularly for augmenting small datasets in text classification tasks, remains underexplored, particularly for cost- and privacy-conscious applications that do not permit the use of third-party services such as OpenAI's ChatGPT.
Objective: This study aims to explore the use of open-source LLMs, such as Large Language Model Meta AI (LLaMA) and Alpaca models, for data augmentation in a specific text classification task related to hospital staff surveys.
Methods: The surveys were designed to elicit narratives of everyday adaptation by frontline radiology staff during the initial phase of the COVID-19 pandemic. A 2-step process of data augmentation and text classification was conducted. The study generated synthetic data similar to the survey reports using 4 generative LLMs for data augmentation. A different set of 3 classifier LLMs was then used to classify the augmented text for thematic categories. The study evaluated performance on the classification task.
Results: The overall best-performing combination of LLMs, temperature, classifier, and number of synthetic data cases is via augmentation with LLaMA 7B at temperature 0.7 with 100 augments, using Robustly Optimized BERT Pretraining Approach (RoBERTa) for the classification task, achieving an average area under the receiver operating characteristic (AUC) curve of 0.87 (SD 0.02; ie, 1 SD). The results demonstrate that open-source LLMs can enhance text classifiers' performance for small datasets in health care contexts, providing promising pathways for improving medical education processes and patient care practices.
Conclusions: The study demonstrates the value of data augmentation with open-source LLMs, highlights the importance of privacy and ethical considerations when using LLMs, and suggests future directions for research in this field.
{"title":"Leveraging Open-Source Large Language Models for Data Augmentation in Hospital Staff Surveys: Mixed Methods Study.","authors":"Carl Ehrett, Sudeep Hegde, Kwame Andre, Dixizi Liu, Timothy Wilson","doi":"10.2196/51433","DOIUrl":"10.2196/51433","url":null,"abstract":"<p><strong>Background: </strong>Generative large language models (LLMs) have the potential to revolutionize medical education by generating tailored learning materials, enhancing teaching efficiency, and improving learner engagement. However, the application of LLMs in health care settings, particularly for augmenting small datasets in text classification tasks, remains underexplored, particularly for cost- and privacy-conscious applications that do not permit the use of third-party services such as OpenAI's ChatGPT.</p><p><strong>Objective: </strong>This study aims to explore the use of open-source LLMs, such as Large Language Model Meta AI (LLaMA) and Alpaca models, for data augmentation in a specific text classification task related to hospital staff surveys.</p><p><strong>Methods: </strong>The surveys were designed to elicit narratives of everyday adaptation by frontline radiology staff during the initial phase of the COVID-19 pandemic. A 2-step process of data augmentation and text classification was conducted. The study generated synthetic data similar to the survey reports using 4 generative LLMs for data augmentation. A different set of 3 classifier LLMs was then used to classify the augmented text for thematic categories. The study evaluated performance on the classification task.</p><p><strong>Results: </strong>The overall best-performing combination of LLMs, temperature, classifier, and number of synthetic data cases is via augmentation with LLaMA 7B at temperature 0.7 with 100 augments, using Robustly Optimized BERT Pretraining Approach (RoBERTa) for the classification task, achieving an average area under the receiver operating characteristic (AUC) curve of 0.87 (SD 0.02; ie, 1 SD). The results demonstrate that open-source LLMs can enhance text classifiers' performance for small datasets in health care contexts, providing promising pathways for improving medical education processes and patient care practices.</p><p><strong>Conclusions: </strong>The study demonstrates the value of data augmentation with open-source LLMs, highlights the importance of privacy and ethical considerations when using LLMs, and suggests future directions for research in this field.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e51433"},"PeriodicalIF":3.2,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142669265","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gry Mørk, Tore Bonsaksen, Ole Sønnik Larsen, Hans Martin Kunnikoff, Silje Stangeland Lie
Background: Virtual reality (VR) is increasingly being used in higher education for clinical skills training and role-playing among health care students. Using 360° videos in VR headsets, followed by peer debrief and group discussions, may strengthen students' social and emotional learning.
Objective: This study aimed to explore student-perceived usability of VR simulation in three health care education programs in Norway.
Methods: Students from one university participated in a VR simulation program. Of these, students in social education (n=74), nursing (n=45), and occupational therapy (n=27) completed a questionnaire asking about their perceptions of the usability of the VR simulation and the related learning activities. Differences between groups of students were examined with Pearson chi-square tests and with 1-way ANOVA. Qualitative content analysis was used to analyze data from open-ended questions.
Results: The nursing students were most satisfied with the usability of the VR simulation, while the occupational therapy students were least satisfied. The nursing students had more often prior experience from using VR technology (60%), while occupational therapy students less often had prior experience (37%). Nevertheless, high mean scores indicated that the students experienced the VR simulation and the related learning activities as very useful. The results also showed that by using realistic scenarios in VR simulation, health care students can be prepared for complex clinical situations in a safe environment. Also, group debriefing sessions are a vital part of the learning process that enhance active involvement with peers.
Conclusions: VR simulation has promise and potential as a pedagogical tool in health care education, especially for training soft skills relevant for clinical practice, such as communication, decision-making, time management, and critical thinking.
{"title":"Virtual Reality Simulation in Undergraduate Health Care Education Programs: Usability Study.","authors":"Gry Mørk, Tore Bonsaksen, Ole Sønnik Larsen, Hans Martin Kunnikoff, Silje Stangeland Lie","doi":"10.2196/56844","DOIUrl":"10.2196/56844","url":null,"abstract":"<p><strong>Background: </strong>Virtual reality (VR) is increasingly being used in higher education for clinical skills training and role-playing among health care students. Using 360° videos in VR headsets, followed by peer debrief and group discussions, may strengthen students' social and emotional learning.</p><p><strong>Objective: </strong>This study aimed to explore student-perceived usability of VR simulation in three health care education programs in Norway.</p><p><strong>Methods: </strong>Students from one university participated in a VR simulation program. Of these, students in social education (n=74), nursing (n=45), and occupational therapy (n=27) completed a questionnaire asking about their perceptions of the usability of the VR simulation and the related learning activities. Differences between groups of students were examined with Pearson chi-square tests and with 1-way ANOVA. Qualitative content analysis was used to analyze data from open-ended questions.</p><p><strong>Results: </strong>The nursing students were most satisfied with the usability of the VR simulation, while the occupational therapy students were least satisfied. The nursing students had more often prior experience from using VR technology (60%), while occupational therapy students less often had prior experience (37%). Nevertheless, high mean scores indicated that the students experienced the VR simulation and the related learning activities as very useful. The results also showed that by using realistic scenarios in VR simulation, health care students can be prepared for complex clinical situations in a safe environment. Also, group debriefing sessions are a vital part of the learning process that enhance active involvement with peers.</p><p><strong>Conclusions: </strong>VR simulation has promise and potential as a pedagogical tool in health care education, especially for training soft skills relevant for clinical practice, such as communication, decision-making, time management, and critical thinking.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e56844"},"PeriodicalIF":3.2,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142669267","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Irene Carrillo, Ivana Skoumalová, Ireen Bruus, Victoria Klemm, Sofia Guerra-Paiva, Bojana Knežević, Augustina Jankauskiene, Dragana Jocic, Susanna Tella, Sandra C Buttigieg, Einav Srulovici, Andrea Madarasová Gecková, Kaja Põlluste, Reinhard Strametz, Paulo Sousa, Marina Odalovic, José Joaquín Mira
[This corrects the article DOI: 10.2196/64125.].
[此处更正了文章 DOI:10.2196/64125]。
{"title":"Correction: Psychological Safety Competency Training During the Clinical Internship From the Perspective of Health Care Trainee Mentors in 11 Pan-European Countries: Mixed Methods Observational Study.","authors":"Irene Carrillo, Ivana Skoumalová, Ireen Bruus, Victoria Klemm, Sofia Guerra-Paiva, Bojana Knežević, Augustina Jankauskiene, Dragana Jocic, Susanna Tella, Sandra C Buttigieg, Einav Srulovici, Andrea Madarasová Gecková, Kaja Põlluste, Reinhard Strametz, Paulo Sousa, Marina Odalovic, José Joaquín Mira","doi":"10.2196/68503","DOIUrl":"10.2196/68503","url":null,"abstract":"<p><p>[This corrects the article DOI: 10.2196/64125.].</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e68503"},"PeriodicalIF":3.2,"publicationDate":"2024-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142639999","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Vasudha L Bhavaraju, Sarada Panchanathan, Brigham C Willis, Pamela Garcia-Filion
Background: Competence-based medical education requires robust data to link competence with clinical experiences. The SARS-CoV-2 (COVID-19) pandemic abruptly altered the standard trajectory of clinical exposure in medical training programs. Residency program directors were tasked with identifying and addressing the resultant gaps in each trainee's experiences using existing tools.
Objective: This study aims to demonstrate a feasible and efficient method to capture electronic health record (EHR) data that measure the volume and variety of pediatric resident clinical experiences from a continuity clinic; generate individual-, class-, and graduate-level benchmark data; and create a visualization for learners to quickly identify gaps in clinical experiences.
Methods: This pilot was conducted in a large, urban pediatric residency program from 2016 to 2022. Through consensus, 5 pediatric faculty identified diagnostic groups that pediatric residents should see to be competent in outpatient pediatrics. Information technology consultants used International Classification of Diseases, Tenth Revision (ICD-10) codes corresponding with each diagnostic group to extract EHR patient encounter data as an indicator of exposure to the specific diagnosis. The frequency (volume) and diagnosis types (variety) seen by active residents (classes of 2020-2022) were compared with class and graduated resident (classes of 2016-2019) averages. These data were converted to percentages and translated to a radar chart visualization for residents to quickly compare their current clinical experiences with peers and graduates. Residents were surveyed on the use of these data and the visualization to identify training gaps.
Results: Patient encounter data about clinical experiences for 102 residents (N=52 graduates) were extracted. Active residents (n=50) received data reports with radar graphs biannually: 3 for the classes of 2020 and 2021 and 2 for the class of 2022. Radar charts distinctly demonstrated gaps in diagnoses exposure compared with classmates and graduates. Residents found the visualization useful in setting clinical and learning goals.
Conclusions: This pilot describes an innovative method of capturing and presenting data about resident clinical experiences, compared with peer and graduate benchmarks, to identify learning gaps that may result from disruptions or modifications in medical training. This methodology can be aggregated across specialties and institutions and potentially inform competence-based medical education.
{"title":"Leveraging the Electronic Health Record to Measure Resident Clinical Experiences and Identify Training Gaps: Development and Usability Study.","authors":"Vasudha L Bhavaraju, Sarada Panchanathan, Brigham C Willis, Pamela Garcia-Filion","doi":"10.2196/53337","DOIUrl":"10.2196/53337","url":null,"abstract":"<p><strong>Background: </strong>Competence-based medical education requires robust data to link competence with clinical experiences. The SARS-CoV-2 (COVID-19) pandemic abruptly altered the standard trajectory of clinical exposure in medical training programs. Residency program directors were tasked with identifying and addressing the resultant gaps in each trainee's experiences using existing tools.</p><p><strong>Objective: </strong>This study aims to demonstrate a feasible and efficient method to capture electronic health record (EHR) data that measure the volume and variety of pediatric resident clinical experiences from a continuity clinic; generate individual-, class-, and graduate-level benchmark data; and create a visualization for learners to quickly identify gaps in clinical experiences.</p><p><strong>Methods: </strong>This pilot was conducted in a large, urban pediatric residency program from 2016 to 2022. Through consensus, 5 pediatric faculty identified diagnostic groups that pediatric residents should see to be competent in outpatient pediatrics. Information technology consultants used International Classification of Diseases, Tenth Revision (ICD-10) codes corresponding with each diagnostic group to extract EHR patient encounter data as an indicator of exposure to the specific diagnosis. The frequency (volume) and diagnosis types (variety) seen by active residents (classes of 2020-2022) were compared with class and graduated resident (classes of 2016-2019) averages. These data were converted to percentages and translated to a radar chart visualization for residents to quickly compare their current clinical experiences with peers and graduates. Residents were surveyed on the use of these data and the visualization to identify training gaps.</p><p><strong>Results: </strong>Patient encounter data about clinical experiences for 102 residents (N=52 graduates) were extracted. Active residents (n=50) received data reports with radar graphs biannually: 3 for the classes of 2020 and 2021 and 2 for the class of 2022. Radar charts distinctly demonstrated gaps in diagnoses exposure compared with classmates and graduates. Residents found the visualization useful in setting clinical and learning goals.</p><p><strong>Conclusions: </strong>This pilot describes an innovative method of capturing and presenting data about resident clinical experiences, compared with peer and graduate benchmarks, to identify learning gaps that may result from disruptions or modifications in medical training. This methodology can be aggregated across specialties and institutions and potentially inform competence-based medical education.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e53337"},"PeriodicalIF":3.2,"publicationDate":"2024-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11559912/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142591681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Brenton T Bicknell, Danner Butler, Sydney Whalen, James Ricks, Cory J Dixon, Abigail B Clark, Olivia Spaedy, Adam Skelton, Neel Edupuganti, Lance Dzubinski, Hudson Tate, Garrett Dyess, Brenessa Lindeman, Lisa Soleymani Lehmann
Background: Recent studies, including those by the National Board of Medical Examiners, have highlighted the remarkable capabilities of recent large language models (LLMs) such as ChatGPT in passing the United States Medical Licensing Examination (USMLE). However, there is a gap in detailed analysis of LLM performance in specific medical content areas, thus limiting an assessment of their potential utility in medical education.
Objective: This study aimed to assess and compare the accuracy of successive ChatGPT versions (GPT-3.5, GPT-4, and GPT-4 Omni) in USMLE disciplines, clinical clerkships, and the clinical skills of diagnostics and management.
Methods: This study used 750 clinical vignette-based multiple-choice questions to characterize the performance of successive ChatGPT versions (ChatGPT 3.5 [GPT-3.5], ChatGPT 4 [GPT-4], and ChatGPT 4 Omni [GPT-4o]) across USMLE disciplines, clinical clerkships, and in clinical skills (diagnostics and management). Accuracy was assessed using a standardized protocol, with statistical analyses conducted to compare the models' performances.
Results: GPT-4o achieved the highest accuracy across 750 multiple-choice questions at 90.4%, outperforming GPT-4 and GPT-3.5, which scored 81.1% and 60.0%, respectively. GPT-4o's highest performances were in social sciences (95.5%), behavioral and neuroscience (94.2%), and pharmacology (93.2%). In clinical skills, GPT-4o's diagnostic accuracy was 92.7% and management accuracy was 88.8%, significantly higher than its predecessors. Notably, both GPT-4o and GPT-4 significantly outperformed the medical student average accuracy of 59.3% (95% CI 58.3-60.3).
Conclusions: GPT-4o's performance in USMLE disciplines, clinical clerkships, and clinical skills indicates substantial improvements over its predecessors, suggesting significant potential for the use of this technology as an educational aid for medical students. These findings underscore the need for careful consideration when integrating LLMs into medical education, emphasizing the importance of structured curricula to guide their appropriate use and the need for ongoing critical analyses to ensure their reliability and effectiveness.
背景:最近的研究,包括美国国家医学考试委员会(National Board of Medical Examiners)的研究,都强调了最近的大型语言模型(LLM),如 ChatGPT,在通过美国医学执照考试(USMLE)方面的卓越能力。然而,在详细分析 LLM 在特定医学内容领域的表现方面还存在空白,从而限制了对其在医学教育中潜在用途的评估:本研究旨在评估和比较历代 ChatGPT 版本(GPT-3.5、GPT-4 和 GPT-4 Omni)在 USMLE 学科、临床实习以及诊断和管理临床技能方面的准确性:本研究使用了 750 道基于临床小故事的选择题,以描述历代 ChatGPT 版本(ChatGPT 3.5 [GPT-3.5]、ChatGPT 4 [GPT-4]和 ChatGPT 4 Omni [GPT-4o])在 USMLE 学科、临床实习和临床技能(诊断和管理)方面的表现。采用标准化方案评估准确性,并进行统计分析以比较模型的性能:结果:在750道选择题中,GPT-4o的准确率最高,达到90.4%,超过了分别为81.1%和60.0%的GPT-4和GPT-3.5。GPT-4o 在社会科学(95.5%)、行为与神经科学(94.2%)和药理学(93.2%)方面表现最佳。在临床技能方面,GPT-4o 的诊断准确率为 92.7%,管理准确率为 88.8%,明显高于其前身。值得注意的是,GPT-4o和GPT-4的准确率均明显高于医学生59.3%(95% CI 58.3-60.3)的平均准确率:结论:GPT-4o 在 USMLE 学科、临床实习和临床技能方面的表现比其前代产品有了大幅提高,这表明该技术作为医学生教育辅助工具的巨大潜力。这些发现强调了在将 LLMs 纳入医学教育时需要慎重考虑的问题,强调了结构化课程对指导其合理使用的重要性,以及持续进行关键分析以确保其可靠性和有效性的必要性。
{"title":"ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis.","authors":"Brenton T Bicknell, Danner Butler, Sydney Whalen, James Ricks, Cory J Dixon, Abigail B Clark, Olivia Spaedy, Adam Skelton, Neel Edupuganti, Lance Dzubinski, Hudson Tate, Garrett Dyess, Brenessa Lindeman, Lisa Soleymani Lehmann","doi":"10.2196/63430","DOIUrl":"https://doi.org/10.2196/63430","url":null,"abstract":"<p><strong>Background: </strong>Recent studies, including those by the National Board of Medical Examiners, have highlighted the remarkable capabilities of recent large language models (LLMs) such as ChatGPT in passing the United States Medical Licensing Examination (USMLE). However, there is a gap in detailed analysis of LLM performance in specific medical content areas, thus limiting an assessment of their potential utility in medical education.</p><p><strong>Objective: </strong>This study aimed to assess and compare the accuracy of successive ChatGPT versions (GPT-3.5, GPT-4, and GPT-4 Omni) in USMLE disciplines, clinical clerkships, and the clinical skills of diagnostics and management.</p><p><strong>Methods: </strong>This study used 750 clinical vignette-based multiple-choice questions to characterize the performance of successive ChatGPT versions (ChatGPT 3.5 [GPT-3.5], ChatGPT 4 [GPT-4], and ChatGPT 4 Omni [GPT-4o]) across USMLE disciplines, clinical clerkships, and in clinical skills (diagnostics and management). Accuracy was assessed using a standardized protocol, with statistical analyses conducted to compare the models' performances.</p><p><strong>Results: </strong>GPT-4o achieved the highest accuracy across 750 multiple-choice questions at 90.4%, outperforming GPT-4 and GPT-3.5, which scored 81.1% and 60.0%, respectively. GPT-4o's highest performances were in social sciences (95.5%), behavioral and neuroscience (94.2%), and pharmacology (93.2%). In clinical skills, GPT-4o's diagnostic accuracy was 92.7% and management accuracy was 88.8%, significantly higher than its predecessors. Notably, both GPT-4o and GPT-4 significantly outperformed the medical student average accuracy of 59.3% (95% CI 58.3-60.3).</p><p><strong>Conclusions: </strong>GPT-4o's performance in USMLE disciplines, clinical clerkships, and clinical skills indicates substantial improvements over its predecessors, suggesting significant potential for the use of this technology as an educational aid for medical students. These findings underscore the need for careful consideration when integrating LLMs into medical education, emphasizing the importance of structured curricula to guide their appropriate use and the need for ongoing critical analyses to ensure their reliability and effectiveness.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e63430"},"PeriodicalIF":3.2,"publicationDate":"2024-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142591666","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sauliha Rabia Alli, Soaad Qahhār Hossain, Sunit Das, Ross Upshur
Unlabelled: In the field of medicine, uncertainty is inherent. Physicians are asked to make decisions on a daily basis without complete certainty, whether it is in understanding the patient's problem, performing the physical examination, interpreting the findings of diagnostic tests, or proposing a management plan. The reasons for this uncertainty are widespread, including the lack of knowledge about the patient, individual physician limitations, and the limited predictive power of objective diagnostic tools. This uncertainty poses significant problems in providing competent patient care. Research efforts and teaching are attempts to reduce uncertainty that have now become inherent to medicine. Despite this, uncertainty is rampant. Artificial intelligence (AI) tools, which are being rapidly developed and integrated into practice, may change the way we navigate uncertainty. In their strongest forms, AI tools may have the ability to improve data collection on diseases, patient beliefs, values, and preferences, thereby allowing more time for physician-patient communication. By using methods not previously considered, these tools hold the potential to reduce the uncertainty in medicine, such as those arising due to the lack of clinical information and provider skill and bias. Despite this possibility, there has been considerable resistance to the implementation of AI tools in medical practice. In this viewpoint article, we discuss the impact of AI on medical uncertainty and discuss practical approaches to teaching the use of AI tools in medical schools and residency training programs, including AI ethics, real-world skills, and technological aptitude.
{"title":"The Potential of Artificial Intelligence Tools for Reducing Uncertainty in Medicine and Directions for Medical Education.","authors":"Sauliha Rabia Alli, Soaad Qahhār Hossain, Sunit Das, Ross Upshur","doi":"10.2196/51446","DOIUrl":"10.2196/51446","url":null,"abstract":"<p><strong>Unlabelled: </strong>In the field of medicine, uncertainty is inherent. Physicians are asked to make decisions on a daily basis without complete certainty, whether it is in understanding the patient's problem, performing the physical examination, interpreting the findings of diagnostic tests, or proposing a management plan. The reasons for this uncertainty are widespread, including the lack of knowledge about the patient, individual physician limitations, and the limited predictive power of objective diagnostic tools. This uncertainty poses significant problems in providing competent patient care. Research efforts and teaching are attempts to reduce uncertainty that have now become inherent to medicine. Despite this, uncertainty is rampant. Artificial intelligence (AI) tools, which are being rapidly developed and integrated into practice, may change the way we navigate uncertainty. In their strongest forms, AI tools may have the ability to improve data collection on diseases, patient beliefs, values, and preferences, thereby allowing more time for physician-patient communication. By using methods not previously considered, these tools hold the potential to reduce the uncertainty in medicine, such as those arising due to the lack of clinical information and provider skill and bias. Despite this possibility, there has been considerable resistance to the implementation of AI tools in medical practice. In this viewpoint article, we discuss the impact of AI on medical uncertainty and discuss practical approaches to teaching the use of AI tools in medical schools and residency training programs, including AI ethics, real-world skills, and technological aptitude.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e51446"},"PeriodicalIF":3.2,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11554287/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142575636","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Unlabelled: Digital transformation has disrupted many industries but is yet to revolutionize health care. Educational programs must be aligned with the reality that goes beyond developing individuals in their own professions, professionals wishing to make an impact in digital health will need a multidisciplinary understanding of how business models, organizational processes, stakeholder relationships, and workforce dynamics across the health care ecosystem may be disrupted by digital health technology. This paper describes the redesign of an existing postgraduate program, ensuring that core digital health content is relevant, pedagogically sound, and evidence-based, and that the program provides learning and practical application of concepts of the digital transformation of health. Existing subjects were mapped to the American Medical Informatics Association Clinical Informatics Core Competencies, followed by consultation with leadership to further identify gaps or opportunities to revise the course structure. New additions of core and elective subjects were proposed to align with the competencies. Suitable electives were chosen based on stakeholder feedback and a review of subjects in fields relevant to digital transformation of health. The program was revised with a new title, course overview, course intended learning outcomes, reorganizing of core subjects, and approval of new electives, adding to a suite of professional development offerings and forming a structured pathway to further qualification. Programs in digital health must move beyond purely informatics-based competencies toward enabling transformational change. Postgraduate program development in this field is possible within a short time frame with the use of established competency frameworks and expert and student consultation.
{"title":"Transforming the Future of Digital Health Education: Redesign of a Graduate Program Using Competency Mapping.","authors":"Michelle Mun, Sonia Chanchlani, Kayley Lyons, Kathleen Gray","doi":"10.2196/54112","DOIUrl":"10.2196/54112","url":null,"abstract":"<p><strong>Unlabelled: </strong>Digital transformation has disrupted many industries but is yet to revolutionize health care. Educational programs must be aligned with the reality that goes beyond developing individuals in their own professions, professionals wishing to make an impact in digital health will need a multidisciplinary understanding of how business models, organizational processes, stakeholder relationships, and workforce dynamics across the health care ecosystem may be disrupted by digital health technology. This paper describes the redesign of an existing postgraduate program, ensuring that core digital health content is relevant, pedagogically sound, and evidence-based, and that the program provides learning and practical application of concepts of the digital transformation of health. Existing subjects were mapped to the American Medical Informatics Association Clinical Informatics Core Competencies, followed by consultation with leadership to further identify gaps or opportunities to revise the course structure. New additions of core and elective subjects were proposed to align with the competencies. Suitable electives were chosen based on stakeholder feedback and a review of subjects in fields relevant to digital transformation of health. The program was revised with a new title, course overview, course intended learning outcomes, reorganizing of core subjects, and approval of new electives, adding to a suite of professional development offerings and forming a structured pathway to further qualification. Programs in digital health must move beyond purely informatics-based competencies toward enabling transformational change. Postgraduate program development in this field is possible within a short time frame with the use of established competency frameworks and expert and student consultation.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e54112"},"PeriodicalIF":3.2,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11542907/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142559031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mert Karabacak, Zeynep Ozcan, Burak Berksu Ozkara, Zeynep Sude Furkan, Sotirios Bisdas
Background: Undergraduate medical students often lack hands-on research experience and fundamental scientific research skills, limiting their exposure to the practical aspects of scientific investigation. The Cerrahpasa Neuroscience Society introduced a program to address this deficiency and facilitate student-led research.
Objective: The primary goal of this initiative was to enhance medical students' research output by enabling them to generate and publish peer-reviewed papers within the framework of this pilot project. The project aimed to provide an accessible, global model for research training through structured journal clubs, mentorship from experienced peers, and resource access.
Methods: In January 2022, a total of 30 volunteer students from various Turkish medical schools participated in this course-based undergraduate research experience program. Students self-organized into 2 groups according to their preferred study type: original research or systematic review. Two final-year students with prior research experience led the project, developing training modules using selected materials. The project was implemented entirely online, with participants completing training modules before using their newly acquired theoretical knowledge to perform assigned tasks.
Results: Based on student feedback, the project timeline was adjusted to allow for greater flexibility in meeting deadlines. Despite these adjustments, participants successfully completed their tasks, applying the theoretical knowledge they had gained to their respective assignments. As of April 2024, the initiative has culminated in 3 published papers and 3 more under peer review. The project has also seen an increase in student interest in further involvement and self-paced learning.
Conclusions: This initiative leverages globally accessible resources for research training, effectively fostering research competency among participants. It has successfully demonstrated the potential for undergraduates to contribute to medical research output and paved the way for a self-sustaining, student-led research program. Despite some logistical challenges, the project provided valuable insights for future implementations, showcasing the potential for students to engage in meaningful, publishable research.
{"title":"A Pilot Project to Promote Research Competency in Medical Students Through Journal Clubs: Mixed Methods Study.","authors":"Mert Karabacak, Zeynep Ozcan, Burak Berksu Ozkara, Zeynep Sude Furkan, Sotirios Bisdas","doi":"10.2196/51173","DOIUrl":"10.2196/51173","url":null,"abstract":"<p><strong>Background: </strong>Undergraduate medical students often lack hands-on research experience and fundamental scientific research skills, limiting their exposure to the practical aspects of scientific investigation. The Cerrahpasa Neuroscience Society introduced a program to address this deficiency and facilitate student-led research.</p><p><strong>Objective: </strong>The primary goal of this initiative was to enhance medical students' research output by enabling them to generate and publish peer-reviewed papers within the framework of this pilot project. The project aimed to provide an accessible, global model for research training through structured journal clubs, mentorship from experienced peers, and resource access.</p><p><strong>Methods: </strong>In January 2022, a total of 30 volunteer students from various Turkish medical schools participated in this course-based undergraduate research experience program. Students self-organized into 2 groups according to their preferred study type: original research or systematic review. Two final-year students with prior research experience led the project, developing training modules using selected materials. The project was implemented entirely online, with participants completing training modules before using their newly acquired theoretical knowledge to perform assigned tasks.</p><p><strong>Results: </strong>Based on student feedback, the project timeline was adjusted to allow for greater flexibility in meeting deadlines. Despite these adjustments, participants successfully completed their tasks, applying the theoretical knowledge they had gained to their respective assignments. As of April 2024, the initiative has culminated in 3 published papers and 3 more under peer review. The project has also seen an increase in student interest in further involvement and self-paced learning.</p><p><strong>Conclusions: </strong>This initiative leverages globally accessible resources for research training, effectively fostering research competency among participants. It has successfully demonstrated the potential for undergraduates to contribute to medical research output and paved the way for a self-sustaining, student-led research program. Despite some logistical challenges, the project provided valuable insights for future implementations, showcasing the potential for students to engage in meaningful, publishable research.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e51173"},"PeriodicalIF":3.2,"publicationDate":"2024-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11542906/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142559030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Maiar Elhariry, Kashish Malhotra, Kashish Goyal, Marco Bardus, Punith Kempegowda
<p><strong>Background: </strong>Social media is a powerful platform for disseminating health information, yet it is often riddled with misinformation. Further, few guidelines exist for producing reliable, peer-reviewed content. This study describes a framework for creating and disseminating evidence-based videos on polycystic ovary syndrome (PCOS) and thyroid conditions to improve health literacy and tackle misinformation.</p><p><strong>Objective: </strong>The study aims to evaluate the creation, dissemination, and impact of evidence-based, peer-reviewed short videos on PCOS and thyroid disorders across social media. It also explores the experiences of content creators and assesses audience engagement.</p><p><strong>Methods: </strong>This mixed methods prospective study was conducted between December 2022 and May 2023 and comprised five phases: (1) script generation, (2) video creation, (3) cross-platform publication, (4) process evaluation, and (5) impact evaluation. The SIMBA-CoMICs (Simulation via Instant Messaging for Bedside Application-Combined Medical Information Cines) initiative provides a structured process where medical concepts are simplified and converted to visually engaging videos. The initiative recruited medical students interested in making visually appealing and scientifically accurate videos for social media. The students were then guided to create video scripts based on frequently searched PCOS- and thyroid-related topics. Once experts confirmed the accuracy of the scripts, the medical students produced the videos. The videos were checked by clinical experts and experts with lived experience to ensure clarity and engagement. The SIMBA-CoMICs team then guided the students in editing these videos to fit platform requirements before posting them on TikTok, Instagram, YouTube, and Twitter. Engagement metrics were tracked over 2 months. Content creators were interviewed, and thematic analysis was performed to explore their experiences.</p><p><strong>Results: </strong>The 20 videos received 718 likes, 120 shares, and 54,686 views across all platforms, with TikTok (19,458 views) and Twitter (19,678 views) being the most popular. Engagement increased significantly, with follower growth ranging from 5% on Twitter to 89% on TikTok. Thematic analysis of interviews with 8 out of 38 participants revealed 4 key themes: views on social media, advice for using social media, reasons for participating, and reflections on the project. Content creators highlighted the advantages of social media, such as large outreach (12 references), convenience (10 references), and accessibility to opportunities (7 references). Participants appreciated the nonrestrictive participation criteria, convenience (8 references), and the ability to record from home using prewritten scripts (6 references). Further recommendations to improve the content creation experience included awareness of audience demographics (9 references), sharing content on multiple platforms
{"title":"A SIMBA CoMICs Initiative to Cocreating and Disseminating Evidence-Based, Peer-Reviewed Short Videos on Social Media: Mixed Methods Prospective Study.","authors":"Maiar Elhariry, Kashish Malhotra, Kashish Goyal, Marco Bardus, Punith Kempegowda","doi":"10.2196/52924","DOIUrl":"10.2196/52924","url":null,"abstract":"<p><strong>Background: </strong>Social media is a powerful platform for disseminating health information, yet it is often riddled with misinformation. Further, few guidelines exist for producing reliable, peer-reviewed content. This study describes a framework for creating and disseminating evidence-based videos on polycystic ovary syndrome (PCOS) and thyroid conditions to improve health literacy and tackle misinformation.</p><p><strong>Objective: </strong>The study aims to evaluate the creation, dissemination, and impact of evidence-based, peer-reviewed short videos on PCOS and thyroid disorders across social media. It also explores the experiences of content creators and assesses audience engagement.</p><p><strong>Methods: </strong>This mixed methods prospective study was conducted between December 2022 and May 2023 and comprised five phases: (1) script generation, (2) video creation, (3) cross-platform publication, (4) process evaluation, and (5) impact evaluation. The SIMBA-CoMICs (Simulation via Instant Messaging for Bedside Application-Combined Medical Information Cines) initiative provides a structured process where medical concepts are simplified and converted to visually engaging videos. The initiative recruited medical students interested in making visually appealing and scientifically accurate videos for social media. The students were then guided to create video scripts based on frequently searched PCOS- and thyroid-related topics. Once experts confirmed the accuracy of the scripts, the medical students produced the videos. The videos were checked by clinical experts and experts with lived experience to ensure clarity and engagement. The SIMBA-CoMICs team then guided the students in editing these videos to fit platform requirements before posting them on TikTok, Instagram, YouTube, and Twitter. Engagement metrics were tracked over 2 months. Content creators were interviewed, and thematic analysis was performed to explore their experiences.</p><p><strong>Results: </strong>The 20 videos received 718 likes, 120 shares, and 54,686 views across all platforms, with TikTok (19,458 views) and Twitter (19,678 views) being the most popular. Engagement increased significantly, with follower growth ranging from 5% on Twitter to 89% on TikTok. Thematic analysis of interviews with 8 out of 38 participants revealed 4 key themes: views on social media, advice for using social media, reasons for participating, and reflections on the project. Content creators highlighted the advantages of social media, such as large outreach (12 references), convenience (10 references), and accessibility to opportunities (7 references). Participants appreciated the nonrestrictive participation criteria, convenience (8 references), and the ability to record from home using prewritten scripts (6 references). Further recommendations to improve the content creation experience included awareness of audience demographics (9 references), sharing content on multiple platforms ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e52924"},"PeriodicalIF":4.3,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11561432/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142548074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Background: Artificial intelligence (AI) chatbots are poised to have a profound impact on medical education. Medical students, as early adopters of technology and future health care providers, play a crucial role in shaping the future of health care. However, little is known about the utilization of, perceptions on, and intention to use AI chatbots among medical students in China.
Objective: This study aims to explore the utilization of, perceptions on, and intention to use generative AI chatbots among medical students in China, using the Unified Theory of Acceptance and Use of Technology (UTAUT) framework. By conducting a national cross-sectional survey, we sought to identify the key determinants that influence medical students' acceptance of AI chatbots, thereby providing a basis for enhancing their integration into medical education. Understanding these factors is crucial for educators, policy makers, and technology developers to design and implement effective AI-driven educational tools that align with the needs and expectations of future health care professionals.
Methods: A web-based electronic survey questionnaire was developed and distributed via social media to medical students across the country. The UTAUT was used as a theoretical framework to design the questionnaire and analyze the data. The relationship between behavioral intention to use AI chatbots and UTAUT predictors was examined using multivariable regression.
Results: A total of 693 participants were from 57 universities covering 21 provinces or municipalities in China. Only a minority (199/693, 28.72%) reported using AI chatbots for studying, with ChatGPT (129/693, 18.61%) being the most commonly used. Most of the participants used AI chatbots for quickly obtaining medical information and knowledge (631/693, 91.05%) and increasing learning efficiency (594/693, 85.71%). Utilization behavior, social influence, facilitating conditions, perceived risk, and personal innovativeness showed significant positive associations with the behavioral intention to use AI chatbots (all P values were <.05).
Conclusions: Chinese medical students hold positive perceptions toward and high intentions to use AI chatbots, but there are gaps between intention and actual adoption. This highlights the need for strategies to improve access, training, and support and provide peer usage examples to fully harness the potential benefits of chatbot technology.
背景:人工智能(AI)聊天机器人将对医学教育产生深远影响:人工智能(AI)聊天机器人将对医学教育产生深远影响。医学生作为技术的早期采用者和未来的医疗服务提供者,在塑造医疗保健的未来方面发挥着至关重要的作用。然而,人们对中国医学生使用人工智能聊天机器人的情况、看法和意向知之甚少:本研究旨在采用技术接受和使用统一理论(UTAUT)框架,探讨中国医学生对生成式人工智能聊天机器人的使用情况、看法和使用意向。通过开展全国横断面调查,我们试图找出影响医学生接受人工智能聊天机器人的关键决定因素,从而为加强人工智能聊天机器人与医学教育的结合提供依据。了解这些因素对于教育者、政策制定者和技术开发者设计和实施有效的人工智能驱动的教育工具至关重要,这些工具应符合未来医疗专业人员的需求和期望:方法:我们开发了一个基于网络的电子调查问卷,并通过社交媒体向全国各地的医学生发放。UTAUT作为设计问卷和分析数据的理论框架。使用多元回归法研究了使用人工智能聊天机器人的行为意向与UTAUT预测因素之间的关系:共有 693 名参与者来自中国 21 个省市的 57 所高校。只有少数人(199/693,28.72%)表示在学习中使用了人工智能聊天机器人,其中最常用的是 ChatGPT(129/693,18.61%)。大多数参与者使用人工智能聊天机器人来快速获取医疗信息和知识(631/693,91.05%)以及提高学习效率(594/693,85.71%)。使用行为、社会影响、便利条件、感知风险和个人创新性与使用人工智能聊天机器人的行为意向呈显著正相关(所有 P 值均为结论):中国医学生对人工智能聊天机器人持有积极的看法和较高的使用意愿,但在意愿和实际采用之间存在差距。这凸显出需要制定策略来改善获取、培训和支持,并提供同行使用范例,以充分利用聊天机器人技术的潜在优势。
{"title":"Utilization of, Perceptions on, and Intention to Use AI Chatbots Among Medical Students in China: National Cross-Sectional Study.","authors":"Wenjuan Tao, Jinming Yang, Xing Qu","doi":"10.2196/57132","DOIUrl":"10.2196/57132","url":null,"abstract":"<p><strong>Background: </strong>Artificial intelligence (AI) chatbots are poised to have a profound impact on medical education. Medical students, as early adopters of technology and future health care providers, play a crucial role in shaping the future of health care. However, little is known about the utilization of, perceptions on, and intention to use AI chatbots among medical students in China.</p><p><strong>Objective: </strong>This study aims to explore the utilization of, perceptions on, and intention to use generative AI chatbots among medical students in China, using the Unified Theory of Acceptance and Use of Technology (UTAUT) framework. By conducting a national cross-sectional survey, we sought to identify the key determinants that influence medical students' acceptance of AI chatbots, thereby providing a basis for enhancing their integration into medical education. Understanding these factors is crucial for educators, policy makers, and technology developers to design and implement effective AI-driven educational tools that align with the needs and expectations of future health care professionals.</p><p><strong>Methods: </strong>A web-based electronic survey questionnaire was developed and distributed via social media to medical students across the country. The UTAUT was used as a theoretical framework to design the questionnaire and analyze the data. The relationship between behavioral intention to use AI chatbots and UTAUT predictors was examined using multivariable regression.</p><p><strong>Results: </strong>A total of 693 participants were from 57 universities covering 21 provinces or municipalities in China. Only a minority (199/693, 28.72%) reported using AI chatbots for studying, with ChatGPT (129/693, 18.61%) being the most commonly used. Most of the participants used AI chatbots for quickly obtaining medical information and knowledge (631/693, 91.05%) and increasing learning efficiency (594/693, 85.71%). Utilization behavior, social influence, facilitating conditions, perceived risk, and personal innovativeness showed significant positive associations with the behavioral intention to use AI chatbots (all P values were <.05).</p><p><strong>Conclusions: </strong>Chinese medical students hold positive perceptions toward and high intentions to use AI chatbots, but there are gaps between intention and actual adoption. This highlights the need for strategies to improve access, training, and support and provide peer usage examples to fully harness the potential benefits of chatbot technology.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"10 ","pages":"e57132"},"PeriodicalIF":3.2,"publicationDate":"2024-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11533383/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142509604","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}