Journal of Engineering Education最新文献_第2页

Intersections between ethics and diversity, equity, and inclusion in engineering practice: Insights from practitioners' mental models 工程实践中伦理与多样性、公平和包容的交叉点：来自实践者心智模型的见解

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-11-10 DOI: 10.1002/jee.70042

Justin L. Hess, Sowmya Panuganti, Isil Anakok

Background

Ethics and diversity, equity, and inclusion (DEI) are critical aspects of engineering, and connections between ethics/DEI are becoming increasingly common place in engineering education and practice. For example, many engineering professional organizations have integrated DEI into their codes of ethics.

Purpose

Given the rise of ethics/DEI connections, we address the research question, “How are engineering ethics and DEI related based on mental models elicited from a diverse cross-section of industrial practitioners who bring expertise in ethics and/or DEI?”

Methodology/Approach

We conducted semi-structured interviews with 25 engineering practitioners in 2022 and 2023. Participants depicted how they perceived ethics and DEI to connect in engineering through semi-structured interviews that included a drawing activity. We thematically analyzed participants' drawings regarding how ethics/DEI connect in engineering and participants' associated verbal descriptions.

Findings

We generated seven themes: (1) DEI is an ethical value in engineering; (2) DEI promotes ethical practice in engineering; (3) ethics/DEI shape interpersonal encounters; (4) ethics/DEI shape engineering processes and outcomes; (5) engineering ethics is increasingly and explicitly integrating DEI; (6) context informs how ethics/DEI connect in engineering; and (7) promoting DEI in engineering is a challenge.

Implications

This study adds to the body of knowledge regarding ethics and DEI in engineering and can enrich future research and practice in both domains by explicating interrelations between the two. This work provides empirical support for engineers, engineering educators, and engineering organizations regarding why it is important to elevate both ethics and DEI in engineering and how to do so.

伦理和多样性、公平和包容（DEI）是工程的关键方面，伦理和DEI之间的联系在工程教育和实践中变得越来越普遍。例如，许多工程专业组织已经将DEI整合到他们的道德准则中。鉴于伦理/DEI联系的兴起，我们提出了一个研究问题，“基于从具有伦理和/或DEI专业知识的不同行业从业者中得出的心理模型，工程伦理和DEI是如何相关的？”我们在2022年和2023年对25名工程从业者进行了半结构化访谈。参与者通过包括绘图活动的半结构化访谈描述了他们如何感知道德和DEI在工程中的联系。我们按主题分析了参与者关于工程中伦理/DEI如何联系的图纸以及参与者相关的口头描述。我们得出了七个主题：(1)DEI是工程中的伦理价值；(2) DEI促进工程伦理实践；(3)伦理/DEI塑造人际交往；(4)伦理/DEI塑造工程过程和结果；(5)工程伦理越来越明确地整合DEI；(6)环境决定了伦理与DEI在工程中的联系；(7)在工程中推广DEI是一个挑战。本研究增加了关于工程伦理和DEI的知识体系，并可以通过解释两者之间的相互关系来丰富这两个领域的未来研究和实践。这项工作为工程师、工程教育工作者和工程组织提供了经验支持，说明为什么在工程中提高道德和DEI是重要的，以及如何做到这一点。

{"title":"Intersections between ethics and diversity, equity, and inclusion in engineering practice: Insights from practitioners' mental models","authors":"Justin L. Hess, Sowmya Panuganti, Isil Anakok","doi":"10.1002/jee.70042","DOIUrl":"https://doi.org/10.1002/jee.70042","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Ethics and diversity, equity, and inclusion (DEI) are critical aspects of engineering, and connections between ethics/DEI are becoming increasingly common place in engineering education and practice. For example, many engineering professional organizations have integrated DEI into their codes of ethics.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>Given the rise of ethics/DEI connections, we address the research question, “How are engineering ethics and DEI related based on mental models elicited from a diverse cross-section of industrial practitioners who bring expertise in ethics and/or DEI?”</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methodology/Approach</h3>\u0000 \u0000 <p>We conducted semi-structured interviews with 25 engineering practitioners in 2022 and 2023. Participants depicted how they perceived ethics and DEI to connect in engineering through semi-structured interviews that included a drawing activity. We thematically analyzed participants' drawings regarding how ethics/DEI connect in engineering and participants' associated verbal descriptions.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Findings</h3>\u0000 \u0000 <p>We generated seven themes: (1) DEI is an ethical value in engineering; (2) DEI promotes ethical practice in engineering; (3) ethics/DEI shape interpersonal encounters; (4) ethics/DEI shape engineering processes and outcomes; (5) engineering ethics is increasingly and explicitly integrating DEI; (6) context informs how ethics/DEI connect in engineering; and (7) promoting DEI in engineering is a challenge.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Implications</h3>\u0000 \u0000 <p>This study adds to the body of knowledge regarding ethics and DEI in engineering and can enrich future research and practice in both domains by explicating interrelations between the two. This work provides empirical support for engineers, engineering educators, and engineering organizations regarding why it is important to elevate both ethics and DEI in engineering and how to do so.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"115 1","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jee.70042","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145521618","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Isolation, stressors, and resiliency: Examining the experiences of a transgender woman in engineering 孤立，压力源和弹性：检查工程中变性女性的经历

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-11-06 DOI: 10.1002/jee.70044

Brandon Bakka, Jill Castle, Flora Makowka, Maura Borrego

Background

Recent anti-trans legislation reflects societal discrimination that extends to college campuses, negatively impacting the mental health and persistence of transgender students. Engineering is documented to be particularly hostile to trans and other LGBTQ+ students, but the experiences of transgender engineering students are severely understudied.

Purpose

The purpose of this work is to develop a greater understanding of the unique challenges and stressors that TGNC (transgender, gender non-conforming) students face within engineering by analyzing the experiences of a transgender engineering student.

Method

This reflexive thematic analysis focuses on one student's experiences of coming out and social transition experiences while an engineering student. The student, who transitioned socially in the summer prior to the academic year she was interviewed, discussed how transitioning impacted her experiences within engineering. Gender minority stress theory frames the findings.

Results

The student experienced a hostile political climate in a state that did not value her safety or happiness. This broader context, combined with a masculine and heteronormative engineering climate, subtle anti-trans sentiments expressed by peers, and frequent misgendering led to isolation, depression, and pressure to conceal her identity. After transitioning, she felt more othered by men students but experienced positive mental health impacts and closer relationships with women engineering students.

Conclusions

The burden of persisting in engineering should not fall entirely to TGNC students who may be struggling with belonging and mental health. Even in challenging political environments, engineering faculty can signal their support of trans students through use of pronouns and inclusive examples that do not reinforce binary gender and heteronormativity.

最近的反跨性别立法反映了社会歧视延伸到大学校园，对跨性别学生的心理健康和持久性产生了负面影响。据记载，工程学对跨性别学生和其他LGBTQ+学生尤其不友好，但对跨性别工程学学生的经历却严重缺乏研究。本研究的目的是通过分析一名跨性别工程学生的经历，更好地理解TGNC（跨性别，性别不符合标准）学生在工程领域面临的独特挑战和压力源。方法对一名工科学生的出柜经历和社会转型经历进行反身性专题分析。这名学生在接受采访的学年之前的夏天进行了社会转型，她讨论了转型如何影响了她在工程领域的经历。性别少数群体压力理论构成了研究结果的框架。结果这名学生在一个不重视她的安全或幸福的国家经历了一个充满敌意的政治气候。这种更广泛的背景，加上男性化和异性恋规范的工程氛围，同龄人表达的微妙的反跨性别情绪，以及频繁的性别错误，导致了她的孤立、抑郁和隐藏身份的压力。变性后，她感受到男学生更多的关爱，但心理健康也受到了积极的影响，与女工科学生的关系也更密切了。结论坚持工程的负担不应该完全落在TGNC学生身上，他们可能正在与归属感和心理健康作斗争。即使在充满挑战的政治环境中，工程学院也可以通过使用代词和包容性的例子来表达他们对跨性别学生的支持，这些例子不会强化二元性别和异性恋规范。

{"title":"Isolation, stressors, and resiliency: Examining the experiences of a transgender woman in engineering","authors":"Brandon Bakka, Jill Castle, Flora Makowka, Maura Borrego","doi":"10.1002/jee.70044","DOIUrl":"https://doi.org/10.1002/jee.70044","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Recent anti-trans legislation reflects societal discrimination that extends to college campuses, negatively impacting the mental health and persistence of transgender students. Engineering is documented to be particularly hostile to trans and other LGBTQ+ students, but the experiences of transgender engineering students are severely understudied.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>The purpose of this work is to develop a greater understanding of the unique challenges and stressors that TGNC (transgender, gender non-conforming) students face within engineering by analyzing the experiences of a transgender engineering student.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Method</h3>\u0000 \u0000 <p>This reflexive thematic analysis focuses on one student's experiences of coming out and social transition experiences while an engineering student. The student, who transitioned socially in the summer prior to the academic year she was interviewed, discussed how transitioning impacted her experiences within engineering. Gender minority stress theory frames the findings.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The student experienced a hostile political climate in a state that did not value her safety or happiness. This broader context, combined with a masculine and heteronormative engineering climate, subtle anti-trans sentiments expressed by peers, and frequent misgendering led to isolation, depression, and pressure to conceal her identity. After transitioning, she felt more othered by men students but experienced positive mental health impacts and closer relationships with women engineering students.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>The burden of persisting in engineering should not fall entirely to TGNC students who may be struggling with belonging and mental health. Even in challenging political environments, engineering faculty can signal their support of trans students through use of pronouns and inclusive examples that do not reinforce binary gender and heteronormativity.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"115 1","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jee.70044","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145469931","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Difficulties in understanding the transition between database design stages: An experiment from a semantic distance perspective 理解数据库设计阶段之间转换的困难：从语义距离角度的实验

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-10-21 DOI: 10.1002/jee.70038

Shin-Shing Shin, Yu-Shan Lin, Yi-Cheng Chen, Wei-Ru Chiou

Background

Learners of database courses usually encounter difficulties in building entity-relationship (ER) models and relational models for database problems. These difficulties may arise because of semantic gaps between the stages of database design. To investigate this issue, we employed semantic network theory—particularly the concept of semantic distance—as an analytical framework.

Hypothesis

We hypothesized that (i) the transition from business requirements to ER models involves a greater semantic distance than the ER-to-relational conversion, and (ii) this greater distance results in imprecise semantic elaboration, leading to lower learning performance.

Method

An experiment was designed in which participants, drawn from a database course, completed two sequential translation tasks: (a) business requirements to ER models, and (b) ER models to relational models. Participants' performances were assessed through problem-solving effectiveness and efficiency measures.

Results

The problem-solving effectiveness and efficiency of task (b) surpassed those of task (a), suggesting that task (a) entailed greater semantic distance and less precise elaboration of semantic relationships than task (b).

Conclusion

Our findings suggest that semantic distance and semantic elaboration are critical factors in database design education, aligning with semantic network theory. Instruction should reduce semantic distance and pay more attention to the teaching of the business requirement to ER model transition. This may prompt scholars to develop more effective teaching methods for database design learning from the perspective of semantic distance.

数据库课程的学习者通常在为数据库问题建立实体关系模型和关系模型时遇到困难。这些困难可能是由于数据库设计阶段之间的语义差距造成的。为了研究这个问题，我们采用语义网络理论——特别是语义距离的概念——作为分析框架。我们假设：(i)从业务需求到ER模型的转换比从ER到关系模型的转换涉及更大的语义距离，（ii）这种更大的距离导致不精确的语义细化，从而导致较低的学习性能。方法设计了一项实验，参与者从数据库课程中抽取，完成两个顺序翻译任务：(a)业务需求到ER模型，(b) ER模型到关系模型。参与者的表现通过解决问题的有效性和效率指标进行评估。结果任务(b)的问题解决效果和效率超过任务(a)，说明任务(a)比任务(b)需要更大的语义距离和更少的语义关系的精确阐述。结论语义距离和语义精细化是数据库设计教育的关键因素，与语义网络理论一致。教学中应减少语义距离，更加注重业务需求向ER模型转换的教学。这可能会促使学者从语义距离的角度开发更有效的数据库设计学习教学方法。

{"title":"Difficulties in understanding the transition between database design stages: An experiment from a semantic distance perspective","authors":"Shin-Shing Shin, Yu-Shan Lin, Yi-Cheng Chen, Wei-Ru Chiou","doi":"10.1002/jee.70038","DOIUrl":"https://doi.org/10.1002/jee.70038","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Learners of database courses usually encounter difficulties in building entity-relationship (ER) models and relational models for database problems. These difficulties may arise because of semantic gaps between the stages of database design. To investigate this issue, we employed semantic network theory—particularly the concept of semantic distance—as an analytical framework.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Hypothesis</h3>\u0000 \u0000 <p>We hypothesized that (i) the transition from business requirements to ER models involves a greater semantic distance than the ER-to-relational conversion, and (ii) this greater distance results in imprecise semantic elaboration, leading to lower learning performance.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Method</h3>\u0000 \u0000 <p>An experiment was designed in which participants, drawn from a database course, completed two sequential translation tasks: (a) business requirements to ER models, and (b) ER models to relational models. Participants' performances were assessed through problem-solving effectiveness and efficiency measures.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>The problem-solving effectiveness and efficiency of task (b) surpassed those of task (a), suggesting that task (a) entailed greater semantic distance and less precise elaboration of semantic relationships than task (b).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Our findings suggest that semantic distance and semantic elaboration are critical factors in database design education, aligning with semantic network theory. Instruction should reduce semantic distance and pay more attention to the teaching of the business requirement to ER model transition. This may prompt scholars to develop more effective teaching methods for database design learning from the perspective of semantic distance.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145366569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Comparing forward and reverse engineering design approaches: Lag sequential and epistemic network analyses of student learning 正向与逆向工程设计方法之比较：学生学习的滞后序贯与认知网络分析

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-10-20 DOI: 10.1002/jee.70039

Yu-Hung Chien, Chia-Yu Liu, Cheng-Shiun Tsai

Background

There is a growing emphasis on integrating engineering design into K-12 science, technology, engineering, and mathematics (STEM) education. Prior studies have primarily examined the impact of engineering design on student learning outcomes or employed frequency-based static statistical analyses, rather than focusing on student learning processes—that is, what students actually did and thought during engineering design courses.

Purpose/Hypotheses

This study explored the effects of two prominent engineering design approaches, forward and reverse, on learning outcomes and learning processes.

Design/Method

A mixed-methods quasi-experiment was conducted with a purposive sample of 52 11th-grade students enrolled in a mechatronics engineering design course. One class engaged in the forward engineering teaching condition (F-class; n = 28) and the other in the reverse engineering teaching condition (R-class; n = 24).

Results

Mann–Whitney U tests indicated that the two conditions exhibited no significant differences in overall design solutions. However, the R-class surpassed the F-class in understandability and organic qualities, using the creative product analysis matrix. The Mann–Whitney U test showed that the R-class scored significantly higher on mechatronics content knowledge than the F-class. Lag sequential analysis and epistemic network analysis revealed distinct differences in engineering design behavior, engineering design reflection, and cognitive action between the conditions.

Conclusions

These findings underscore the importance of applying multiple analytical lenses to examine outcomes and processes in engineering design learning. They offer practical implications for developing secondary-level engineering design curricula that support students' design solutions, conceptual understanding, design behavior, design reflection, and higher order cognitive action.

人们越来越强调将工程设计融入K-12科学、技术、工程和数学（STEM）教育。之前的研究主要是考察工程设计对学生学习成果的影响，或者采用基于频率的静态统计分析，而不是关注学生的学习过程——也就是说，学生在工程设计课程中实际做了什么和想了什么。目的/假设本研究探讨了正向和反向两种主要的工程设计方法对学习结果和学习过程的影响。设计/方法采用混合方法进行准实验，目的样本为52名11年级机电一体化工程设计专业学生。一个班从事正向工程教学条件（f班，n = 28），另一个班从事逆向工程教学条件（r班，n = 24）。结果Mann-Whitney U检验表明，两种情况在总体设计方案上没有显著差异。然而，使用创意产品分析矩阵，r级在可理解性和有机品质上超过了f级。Mann-Whitney U检验显示，r级学生的机电一体化内容知识得分显著高于f级学生。滞后序列分析和认知网络分析揭示了不同条件下工程设计行为、工程设计反思和认知行为的显著差异。这些发现强调了应用多种分析视角来检查工程设计学习的结果和过程的重要性。它们为开发二级工程设计课程提供了实际意义，这些课程支持学生的设计解决方案、概念理解、设计行为、设计反思和更高层次的认知行动。

{"title":"Comparing forward and reverse engineering design approaches: Lag sequential and epistemic network analyses of student learning","authors":"Yu-Hung Chien, Chia-Yu Liu, Cheng-Shiun Tsai","doi":"10.1002/jee.70039","DOIUrl":"https://doi.org/10.1002/jee.70039","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>There is a growing emphasis on integrating engineering design into K-12 science, technology, engineering, and mathematics (STEM) education. Prior studies have primarily examined the impact of engineering design on student learning outcomes or employed frequency-based static statistical analyses, rather than focusing on student learning processes—that is, what students actually did and thought during engineering design courses.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose/Hypotheses</h3>\u0000 \u0000 <p>This study explored the effects of two prominent engineering design approaches, forward and reverse, on learning outcomes and learning processes.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Design/Method</h3>\u0000 \u0000 <p>A mixed-methods quasi-experiment was conducted with a purposive sample of 52 11th-grade students enrolled in a mechatronics engineering design course. One class engaged in the forward engineering teaching condition (F-class; <i>n</i> = 28) and the other in the reverse engineering teaching condition (R-class; <i>n</i> = 24).</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Mann–Whitney <i>U</i> tests indicated that the two conditions exhibited no significant differences in overall design solutions. However, the R-class surpassed the F-class in understandability and organic qualities, using the creative product analysis matrix. The Mann–Whitney U test showed that the R-class scored significantly higher on mechatronics content knowledge than the F-class. Lag sequential analysis and epistemic network analysis revealed distinct differences in engineering design behavior, engineering design reflection, and cognitive action between the conditions.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>These findings underscore the importance of applying multiple analytical lenses to examine outcomes and processes in engineering design learning. They offer practical implications for developing secondary-level engineering design curricula that support students' design solutions, conceptual understanding, design behavior, design reflection, and higher order cognitive action.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-10-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145317626","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Using generative AI for large-scale qualitative analysis of social media posts to understand why people leave computer science 使用生成式人工智能对社交媒体帖子进行大规模定性分析，以了解人们离开计算机科学的原因

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-09-30 DOI: 10.1002/jee.70036

Amanda Ross, Andrew Katz

Background

Computer science faces a persistent attrition problem, with people leaving the field at a rate that exceeds new entrants. Given the increasing demand for computing jobs, it is essential to focus on reducing the number of individuals exiting the field.

Purpose

This study investigates why individuals leave the computer science field across various stages and contexts, addressing two questions: (1) What are the reasons for leaving? (2) What external factors influence these decisions?

Method

This large-scale qualitative study collected over 10,000 Reddit posts using keyword-based scraping. Using generative AI, we refined the dataset, filtering it down to 263 relevant posts. Generative AI was then used for thematic analysis on this subset of posts, utilizing the established GATOS method. We extend this approach by integrating a human-in-the-loop process to contextualize the identified themes within social cognitive career theory.

Results

Findings reveal diverse reasons for leaving, including job dissatisfaction, interests in other fields, psychological factors, academic challenges, health concerns, and industry issues. Influential factors include background, transition requirements, alternative field characteristics, and personal circumstances. Although the extent varied, all of these reasons and factors were observed at every departure stage.

Conclusions

These findings provide important insights that can help inform industry and academic policies and practices. Additionally, we contribute to the development of more efficient, scalable workflows for future qualitative research using generative AI.

计算机科学面临着一个持续的人员流失问题，人们离开这个领域的速度超过了新进入者。鉴于对计算机工作的需求不断增加，有必要将重点放在减少退出该领域的个人数量上。本研究调查了个人在不同阶段和背景下离开计算机科学领域的原因，解决了两个问题：(1)离开的原因是什么？(2)哪些外部因素影响这些决策？方法本研究采用基于关键词的抓取技术，收集了1万多条Reddit帖子。使用生成式人工智能，我们改进了数据集，将其过滤到263个相关帖子。然后使用生成式人工智能对该帖子子集进行主题分析，利用已建立的GATOS方法。我们通过整合人在循环过程来扩展这种方法，将社会认知职业理论中确定的主题置于背景中。研究结果显示，学生离职的原因多种多样，包括对工作不满意、对其他领域的兴趣、心理因素、学业挑战、健康问题和行业问题。影响因素包括背景、过渡要求、替代领域特征和个人情况。虽然程度不同，但在每个出发阶段都观察到所有这些原因和因素。这些发现提供了重要的见解，可以帮助告知行业和学术政策和实践。此外，我们还利用生成式人工智能为未来的定性研究开发更高效、可扩展的工作流程。

{"title":"Using generative AI for large-scale qualitative analysis of social media posts to understand why people leave computer science","authors":"Amanda Ross, Andrew Katz","doi":"10.1002/jee.70036","DOIUrl":"https://doi.org/10.1002/jee.70036","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Computer science faces a persistent attrition problem, with people leaving the field at a rate that exceeds new entrants. Given the increasing demand for computing jobs, it is essential to focus on reducing the number of individuals exiting the field.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>This study investigates why individuals leave the computer science field across various stages and contexts, addressing two questions: (1) What are the reasons for leaving? (2) What external factors influence these decisions?</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Method</h3>\u0000 \u0000 <p>This large-scale qualitative study collected over 10,000 Reddit posts using keyword-based scraping. Using generative AI, we refined the dataset, filtering it down to 263 relevant posts. Generative AI was then used for thematic analysis on this subset of posts, utilizing the established GATOS method. We extend this approach by integrating a human-in-the-loop process to contextualize the identified themes within social cognitive career theory.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Findings reveal diverse reasons for leaving, including job dissatisfaction, interests in other fields, psychological factors, academic challenges, health concerns, and industry issues. Influential factors include background, transition requirements, alternative field characteristics, and personal circumstances. Although the extent varied, all of these reasons and factors were observed at every departure stage.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>These findings provide important insights that can help inform industry and academic policies and practices. Additionally, we contribute to the development of more efficient, scalable workflows for future qualitative research using generative AI.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jee.70036","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145224466","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Artificial intelligence in engineering education research: Using machine learning models to predict undergraduate engineering students' persistence to graduation 人工智能在工程教育研究中的应用：利用机器学习模型预测工程本科学生的毕业坚持性

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-09-25 DOI: 10.1002/jee.70034

Ibukun Osunbunmi, Taiwo Feyijimi, Stephanie Cutler, Yashin Brijmohan, Lexy Arinze, Viyon Dansu, Bolaji Bamidele, Jennifer Wu, Robert Rabb

Background

Attrition of engineering students continues to be a concern in higher education. Despite indications that students who opt to leave engineering programs may go on to make meaningful contributions in other fields more aligned to their interests, it remains essential to support those who choose to stay in engineering with the necessary resources, mentorship, and enabling environments to thrive.

Purpose

This study explores predictors of persistence to graduation for students in a College of Engineering (CoE), examining pre-college preparation (SAT scores), academic performance in core courses, demographic factors, and engagement in co-curricular activities.

Methods

We analyzed a 10-year dataset (fall 2007 to fall 2016) from a US R1 university's CoE, comprising 16,292 observations. Machine learning techniques, including dimensionality reduction (forward, backward, and unidirectional stepwise regression), explainable artificial intelligence, and predictive modeling (K-nearest neighbors, logistic regression, decision trees, artificial neural networks, and gradient boosting), were applied to identify significant predictors of persistence.

Results

Key predictors of persistence included students' GPAs in their first two years and SAT math. Additional factors, although not consistently ranked highly by all models, include performance in PHYS 211, CHM 110, and MAT 140 (Physics 1, Chemistry 1, and Calculus 1, respectively). Demographics and engaging in co-curricular activities also contribute to persistence, although not as significantly as academic factors.

Conclusion

Findings from the machine learning models extend Tinto's theory of persistence, and identify key factors that predict engineering students' persistence to graduation. We recommend that institutions engage in strategic planning and policymaking as part of their collective effort to reduce engineering student attrition.

工科学生的流失一直是高等教育中一个令人担忧的问题。尽管有迹象表明，选择离开工程专业的学生可能会在其他更符合他们兴趣的领域做出有意义的贡献，但仍然有必要为那些选择留在工程专业的学生提供必要的资源、指导和有利的环境，让他们茁壮成长。本研究探讨了工程学院（CoE）学生坚持毕业的预测因素，考察了大学前准备（SAT分数）、核心课程的学习成绩、人口统计学因素和参与课外活动。我们分析了来自美国R1大学CoE的10年数据集（2007年秋季至2016年秋季），包括16,292个观测值。机器学习技术，包括降维（前向、后向和单向逐步回归）、可解释的人工智能和预测建模（k近邻、逻辑回归、决策树、人工神经网络和梯度增强），被用于识别持久性的重要预测因子。结果坚持的关键预测因素包括学生前两年的gpa和SAT数学。其他因素，虽然不是所有模型都排在很高的位置，包括物理211，CHM 110和MAT 140（分别是物理1，化学1和微积分1）的表现。人口统计和参与课外活动也有助于坚持，尽管没有学术因素那么重要。机器学习模型的发现扩展了Tinto的坚持理论，并确定了预测工程专业学生坚持到毕业的关键因素。我们建议各院校参与战略规划和政策制定，作为减少工程专业学生流失的集体努力的一部分。

{"title":"Artificial intelligence in engineering education research: Using machine learning models to predict undergraduate engineering students' persistence to graduation","authors":"Ibukun Osunbunmi, Taiwo Feyijimi, Stephanie Cutler, Yashin Brijmohan, Lexy Arinze, Viyon Dansu, Bolaji Bamidele, Jennifer Wu, Robert Rabb","doi":"10.1002/jee.70034","DOIUrl":"https://doi.org/10.1002/jee.70034","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Attrition of engineering students continues to be a concern in higher education. Despite indications that students who opt to leave engineering programs may go on to make meaningful contributions in other fields more aligned to their interests, it remains essential to support those who choose to stay in engineering with the necessary resources, mentorship, and enabling environments to thrive.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>This study explores predictors of persistence to graduation for students in a College of Engineering (CoE), examining pre-college preparation (SAT scores), academic performance in core courses, demographic factors, and engagement in co-curricular activities.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>We analyzed a 10-year dataset (fall 2007 to fall 2016) from a US R1 university's CoE, comprising 16,292 observations. Machine learning techniques, including dimensionality reduction (forward, backward, and unidirectional stepwise regression), explainable artificial intelligence, and predictive modeling (K-nearest neighbors, logistic regression, decision trees, artificial neural networks, and gradient boosting), were applied to identify significant predictors of persistence.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Key predictors of persistence included students' GPAs in their first two years and SAT math. Additional factors, although not consistently ranked highly by all models, include performance in PHYS 211, CHM 110, and MAT 140 (Physics 1, Chemistry 1, and Calculus 1, respectively). Demographics and engaging in co-curricular activities also contribute to persistence, although not as significantly as academic factors.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Findings from the machine learning models extend Tinto's theory of persistence, and identify key factors that predict engineering students' persistence to graduation. We recommend that institutions engage in strategic planning and policymaking as part of their collective effort to reduce engineering student attrition.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jee.70034","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145146571","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

“More conceptual than actual”: Epistemic metacognition in response to a non-numerical statics question “概念多于实际”：对一个非数值静态问题的认知元认知

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-09-16 DOI: 10.1002/jee.70035

Lorena S. Grundy, Milo D. Koretsky

Background

Metacognitive processes have been linked to the development of conceptual knowledge in STEM courses, but previous work has centered on the regulatory aspects of metacognition.

Purpose

We interrogated the relationship between epistemic metacognition and conceptual knowledge in engineering statics courses across six universities by asking students a difficult concept question with concurrent reflection prompts that elicited their metacognitive thinking.

Method

We used a mixed-methods design containing an embedded phase followed by an explanatory phase. This design allowed us to both prompt and measure student epistemic metacognition within the learning context. The embedded phase consisted of quantitative and qualitative analyses of student responses. The explanatory phase consisted of an analysis of six instructor interviews.

Results

Analysis of 267 student responses showed greater variation in students' epistemic metacognition than in their ability to answer correctly. Students used different kinds of epistemic metacognitive resources about the nature and origin of knowledge, epistemological forms, epistemological activities, and stances toward knowledge. These resources generally assembled into one of two frames: a constructed knowledge framing valuing conceptual knowledge and sense-making, and an authoritative knowledge framing foregrounding numerical, algorithmic problem-solving. All six instructors interviewed described resources that align with both frames, and none explicitly considered student epistemic metacognition.

Conclusions

Instructors' explicit attention to epistemic metacognition can potentially shift students to more productive frames for engineering learning. Findings here also inform two broader issues in STEM instruction: student resistance to active learning, and the direct instruction versus inquiry-based learning debate.

元认知过程与STEM课程中概念知识的发展有关，但之前的工作主要集中在元认知的调节方面。目的通过向六所大学的工程静力学课程的学生提问一个复杂的概念问题，并同时提出反思提示，以激发他们的元认知思维，探讨认知元认知与概念知识之间的关系。方法采用混合方法设计，包括一个嵌入阶段和一个解释阶段。这种设计使我们能够在学习情境中提示和测量学生的认知元认知。嵌入阶段包括对学生反应的定量和定性分析。解释阶段包括对六个教师访谈的分析。结果对267名学生的回答进行分析，发现学生的认识论元认知差异大于正确回答能力差异。学生对知识的性质和起源、认识论形式、认识论活动和对知识的立场使用了不同种类的认识论元认知资源。这些资源通常组合成两个框架之一：一个是重视概念知识和意义构建的构建知识框架，另一个是重视数值、算法问题解决的权威知识框架。所有接受采访的六名教师都描述了与这两个框架一致的资源，没有一个明确考虑到学生的认知元认知。结论：教师对认识论元认知的明确关注可以潜在地将学生转变为更有效的工程学习框架。这里的研究结果还揭示了STEM教学中的两个更广泛的问题：学生对主动学习的抵制，以及直接教学与基于探究的学习的争论。

{"title":"“More conceptual than actual”: Epistemic metacognition in response to a non-numerical statics question","authors":"Lorena S. Grundy, Milo D. Koretsky","doi":"10.1002/jee.70035","DOIUrl":"https://doi.org/10.1002/jee.70035","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Metacognitive processes have been linked to the development of conceptual knowledge in STEM courses, but previous work has centered on the regulatory aspects of metacognition.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>We interrogated the relationship between epistemic metacognition and conceptual knowledge in engineering statics courses across six universities by asking students a difficult concept question with concurrent reflection prompts that elicited their metacognitive thinking.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Method</h3>\u0000 \u0000 <p>We used a mixed-methods design containing an embedded phase followed by an explanatory phase. This design allowed us to both prompt and measure student epistemic metacognition within the learning context. The embedded phase consisted of quantitative and qualitative analyses of student responses. The explanatory phase consisted of an analysis of six instructor interviews.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Analysis of 267 student responses showed greater variation in students' epistemic metacognition than in their ability to answer correctly. Students used different kinds of epistemic metacognitive resources about the nature and origin of knowledge, epistemological forms, epistemological activities, and stances toward knowledge. These resources generally assembled into one of two frames: a <i>constructed knowledge framing</i> valuing conceptual knowledge and sense-making, and an <i>authoritative knowledge framing</i> foregrounding numerical, algorithmic problem-solving. All six instructors interviewed described resources that align with both frames, and none explicitly considered student epistemic metacognition.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>Instructors' explicit attention to epistemic metacognition can potentially shift students to more productive frames for engineering learning. Findings here also inform two broader issues in STEM instruction: student resistance to active learning, and the direct instruction versus inquiry-based learning debate.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145101405","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Factors affecting students' sense of inclusion in the undergraduate engineering program at Waipapa Taumata Rau (The University of Auckland) 影响奥克兰大学本科工程专业学生融入感的因素

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-09-16 DOI: 10.1002/jee.70029

Priyanka Dhopade, James Tizard, Penelope Watson, Ashleigh Fox, Tom Allen, Hazim Namik, Aryan Karan, Rituparna Roy, Kelly Blincoe

Background

Women, ethnic minorities, and LGBTQIA+ people have historically been excluded from the engineering profession. When they do pursue engineering, they often face challenges within both education and industry. Retention is a growing issue; for example, women in industry have significantly higher turnover rates than men.

Purpose/Hypothesis

Feelings of belonging, satisfaction, and perceptions of one's future career are important for retention in engineering education. However, little is known about the factors that impact these constructs in tertiary education—where foundational engineering experiences occur—for a range of potentially intersectional social identities in contexts other than the United States.

Methods

We designed an online questionnaire (n = 379) and a series of focus groups (n = 17) with engineering students at Waipapa Taumata Rau (The University of Auckland) in Aotearoa (New Zealand). We applied thematic analysis to extract a list of common factors that influenced students' experiences in this unique context.

Results

Students who were unsure of or did not want to disclose parts of their identity reported the lowest sense of belonging and satisfaction. The factors that specifically impacted historically excluded groups included unsupportive working environments, not being respected academically, and exclusionary course content.

Conclusion

Our findings identify factors that contributed to students' experiences that may impact retention in Aotearoa but have implications for other contexts. Finally, we make recommendations to engineering education practitioners on how to support (and retain) students from historically excluded groups, including dedicated learning and social environments, inclusive course content, and awareness education on inclusivity.

历史上，女性、少数民族和LGBTQIA+人群一直被排除在工程专业之外。当他们追求工程时，他们经常面临教育和行业的挑战。留存率是一个日益严重的问题；例如，工业界女性的离职率明显高于男性。目的/假设归属感、满足感和对未来职业的认知对工程教育的保留很重要。然而，对于影响高等教育（基础工程经验发生的地方）中这些结构的因素，我们所知甚少，因为在美国以外的环境中，有一系列潜在的交叉社会身份。方法我们设计了一份在线问卷（n = 379）和一系列焦点小组（n = 17），调查对象是新西兰奥特罗阿的Waipapa Taumata Rau（奥克兰大学）的工科学生。我们应用主题分析来提取影响学生在这一独特背景下经历的共同因素列表。结果不确定或不愿透露部分身份的学生的归属感和满意度最低。特别影响历史上被排斥群体的因素包括不支持的工作环境，不受学术尊重，以及排他性的课程内容。结论：我们的研究结果确定了影响学生在Aotearoa中记忆的因素，但对其他情况也有影响。最后，我们就如何支持（和留住）历史上被排斥群体的学生向工程教育从业者提出建议，包括专门的学习和社会环境、包容性课程内容和包容性意识教育。

{"title":"Factors affecting students' sense of inclusion in the undergraduate engineering program at Waipapa Taumata Rau (The University of Auckland)","authors":"Priyanka Dhopade, James Tizard, Penelope Watson, Ashleigh Fox, Tom Allen, Hazim Namik, Aryan Karan, Rituparna Roy, Kelly Blincoe","doi":"10.1002/jee.70029","DOIUrl":"https://doi.org/10.1002/jee.70029","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Women, ethnic minorities, and LGBTQIA+ people have historically been excluded from the engineering profession. When they do pursue engineering, they often face challenges within both education and industry. Retention is a growing issue; for example, women in industry have significantly higher turnover rates than men.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose/Hypothesis</h3>\u0000 \u0000 <p>Feelings of belonging, satisfaction, and perceptions of one's future career are important for retention in engineering education. However, little is known about the factors that impact these constructs in tertiary education—where foundational engineering experiences occur—for a range of potentially intersectional social identities in contexts other than the United States.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>We designed an online questionnaire (<i>n</i> = 379) and a series of focus groups (<i>n</i> = 17) with engineering students at Waipapa Taumata Rau (The University of Auckland) in Aotearoa (New Zealand). We applied thematic analysis to extract a list of common factors that influenced students' experiences in this unique context.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Students who were unsure of or did not want to disclose parts of their identity reported the lowest sense of belonging and satisfaction. The factors that specifically impacted historically excluded groups included unsupportive working environments, not being respected academically, and exclusionary course content.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusion</h3>\u0000 \u0000 <p>Our findings identify factors that contributed to students' experiences that may impact retention in Aotearoa but have implications for other contexts. Finally, we make recommendations to engineering education practitioners on how to support (and retain) students from historically excluded groups, including dedicated learning and social environments, inclusive course content, and awareness education on inclusivity.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/jee.70029","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145101723","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Analysis of student understanding in short-answer explanations to concept questions using a human-centered AI approach 使用以人为本的人工智能方法分析学生对概念问题的简答解释的理解

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-08-31 DOI: 10.1002/jee.70032

Harpreet Auby, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky, Milo D. Koretsky

Background

Analyzing student short-answer written justifications to conceptually challenging questions has proven helpful to understand student thinking and improve conceptual understanding. However, qualitative analyses are limited by the burden of analyzing large amounts of text.

Purpose

We apply dense and sparse Large Language Models (LLMs) to explore how machine learning can automate coding for responses in engineering mechanics and thermodynamics.

Design/Method

We first identify the cognitive resources students use through human coding of seven questions. We then compare the performance of four dense LLMs and a sparse Mixture of Experts (Mixtral) model to automate coding. Finally, we investigate the extent to which domain-specific training is necessary for accurate coding.

Findings

In a sample question, we analyze 904 responses to identify 48 unique cognitive resources, which we then organize into six themes. In contrast to recommendations in the literature, students who activate molecular resources were less likely to answer correctly. This example illustrates the usefulness of qualitatively analyzing large datasets. Of the LLMs, Mixtral and Llama-3 performed best at within the same-dataset, in-domain coding tasks, especially as the training set size increases. Phi-3.5-mini, while effective in mechanics, shows inconsistent improvements with additional data and struggles in thermodynamics. In contrast, GPT-4 and GPT-4o-mini stand out for their robust generalization across in- and cross-domain tasks.

Conclusions

Open-source models like Mixtral have the potential to perform well when coding short-answer justifications to challenging concept questions. However, more fine-tuning is needed so that they can be robust enough to be utilized with a resources-based framing.

分析学生对概念性挑战性问题的简短回答，有助于理解学生的思维，提高对概念的理解。然而，定性分析受到分析大量文本的负担的限制。我们应用密集和稀疏的大型语言模型（llm）来探索机器学习如何自动编码工程力学和热力学中的响应。设计/方法我们首先通过对七个问题进行人工编码来确定学生使用的认知资源。然后，我们比较了四个密集llm和一个稀疏的混合专家（Mixtral）模型在自动编码方面的性能。最后，我们研究了特定领域的训练在多大程度上是准确编码所必需的。在一个样本问题中，我们分析了904个回答，确定了48个独特的认知资源，然后我们将其组织成六个主题。与文献中的建议相反，激活分子资源的学生不太可能回答正确。这个例子说明了定性分析大型数据集的有用性。在llm中，Mixtral和lama-3在相同数据集的域内编码任务中表现最好，特别是当训练集大小增加时。phil -3.5-mini虽然在力学方面很有效，但在热力学方面表现出不一致的改进和额外数据的挣扎。相比之下，GPT-4和gpt - 40 -mini在跨域和跨域任务中具有强大的泛化能力。像Mixtral这样的开源模型在编写具有挑战性的概念问题的简短答案时具有良好的表现。但是，需要进行更多的微调，以便它们足够健壮，能够与基于资源的框架一起使用。

{"title":"Analysis of student understanding in short-answer explanations to concept questions using a human-centered AI approach","authors":"Harpreet Auby, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky, Milo D. Koretsky","doi":"10.1002/jee.70032","DOIUrl":"https://doi.org/10.1002/jee.70032","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>Analyzing student short-answer written justifications to conceptually challenging questions has proven helpful to understand student thinking and improve conceptual understanding. However, qualitative analyses are limited by the burden of analyzing large amounts of text.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>We apply dense and sparse Large Language Models (LLMs) to explore how machine learning can automate coding for responses in engineering mechanics and thermodynamics.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Design/Method</h3>\u0000 \u0000 <p>We first identify the cognitive resources students use through human coding of seven questions. We then compare the performance of four dense LLMs and a sparse Mixture of Experts (Mixtral) model to automate coding. Finally, we investigate the extent to which domain-specific training is necessary for accurate coding.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Findings</h3>\u0000 \u0000 <p>In a sample question, we analyze 904 responses to identify 48 unique cognitive resources, which we then organize into six themes. In contrast to recommendations in the literature, students who activate molecular resources were less likely to answer correctly. This example illustrates the usefulness of qualitatively analyzing large datasets. Of the LLMs, Mixtral and Llama-3 performed best at within the same-dataset, in-domain coding tasks, especially as the training set size increases. Phi-3.5-mini, while effective in mechanics, shows inconsistent improvements with additional data and struggles in thermodynamics. In contrast, GPT-4 and GPT-4o-mini stand out for their robust generalization across in- and cross-domain tasks.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>Open-source models like Mixtral have the potential to perform well when coding short-answer justifications to challenging concept questions. However, more fine-tuning is needed so that they can be robust enough to be utilized with a resources-based framing.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144923504","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Leveraging AI-generated synthetic data to train natural language processing models for qualitative feedback analysis 利用人工智能生成的合成数据训练自然语言处理模型进行定性反馈分析

IF 3.4 2区工程技术 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Engineering Education

Pub Date : 2025-08-31 DOI: 10.1002/jee.70033

Stephanie Fuchs, Alexandra Werth, Cristóbal Méndez, Jonathan Butcher

Background

High-quality feedback is crucial for academic success, driving student motivation and engagement while research explores effective delivery and student interactions. Advances in artificial intelligence (AI), particularly natural language processing (NLP), offer innovative methods for analyzing complex qualitative data such as feedback interactions.

Purpose

We developed a framework to train sentence transformers using generative AI–created synthetic data to categorize student-feedback interactions in engineering studios. We compared traditional thematic analysis with modern methods to evaluate the realism of synthetic datasets and their effectiveness in training NLP models by exploring how generative AI can aid qualitative coding.

Methods

We deidentified and transcribed eight audio recordings from engineering studios. Synthetic feedback transcripts were generated using three locally hosted large language models: Llama 3.1, Gemma 2.0, and Mistral NeMo, adjusting parameters to produce datasets mimicking the real transcripts. We assessed the quality of synthetic transcripts using our framework and used a sentence transformer model (trained on both real and synthetic data) to compare changes in the model's percent accuracy when qualitatively coding feedback interactions.

Results

Synthetic data improved the NLP model's performance in classifying feedback interactions, boosting the average accuracy from 68.4% to 81% with Llama 3.1. Although incorporating synthetic data improved classification, all models produced transcripts that occasionally included extraneous details and failed to capture instructor-dominant discourse.

Conclusions

Synthetic data offers an opportunity to expand qualitative research, particularly in contexts where real data for NLP training is limited or hard to obtain; however, transparency in its use is paramount to maintain research integrity.

高质量的反馈对学术成功至关重要，在研究探索有效的交付和学生互动的同时，它能推动学生的积极性和参与度。人工智能（AI）的进步，特别是自然语言处理（NLP），为分析复杂的定性数据（如反馈交互）提供了创新方法。我们开发了一个框架，使用生成式人工智能创建的合成数据来训练句子转换器，以对工程工作室的学生反馈交互进行分类。我们将传统的主题分析与现代方法进行比较，通过探索生成式人工智能如何帮助定性编码，评估合成数据集的真实性及其在训练NLP模型中的有效性。方法对来自工程工作室的8段录音进行鉴定和转录。使用三个本地托管的大型语言模型：Llama 3.1、Gemma 2.0和Mistral NeMo生成合成反馈转录本，调整参数以生成模拟真实转录本的数据集。我们使用我们的框架评估合成转录本的质量，并使用句子转换模型（在真实和合成数据上进行训练）来比较定性编码反馈交互时模型百分比准确性的变化。结果综合数据提高了NLP模型在反馈交互分类方面的性能，将Llama 3.1的平均准确率从68.4%提高到81%。虽然合并合成数据改进了分类，但所有模型产生的转录本偶尔会包含无关的细节，并且无法捕捉到教师主导的话语。综合数据为扩展定性研究提供了机会，特别是在NLP训练的真实数据有限或难以获得的情况下；然而，其使用的透明度对于保持研究的完整性至关重要。

{"title":"Leveraging AI-generated synthetic data to train natural language processing models for qualitative feedback analysis","authors":"Stephanie Fuchs, Alexandra Werth, Cristóbal Méndez, Jonathan Butcher","doi":"10.1002/jee.70033","DOIUrl":"https://doi.org/10.1002/jee.70033","url":null,"abstract":"<div>\u0000 \u0000 \u0000 <section>\u0000 \u0000 <h3> Background</h3>\u0000 \u0000 <p>High-quality feedback is crucial for academic success, driving student motivation and engagement while research explores effective delivery and student interactions. Advances in artificial intelligence (AI), particularly natural language processing (NLP), offer innovative methods for analyzing complex qualitative data such as feedback interactions.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Purpose</h3>\u0000 \u0000 <p>We developed a framework to train sentence transformers using generative AI–created synthetic data to categorize student-feedback interactions in engineering studios. We compared traditional thematic analysis with modern methods to evaluate the realism of synthetic datasets and their effectiveness in training NLP models by exploring how generative AI can aid qualitative coding.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Methods</h3>\u0000 \u0000 <p>We deidentified and transcribed eight audio recordings from engineering studios. Synthetic feedback transcripts were generated using three locally hosted large language models: Llama 3.1, Gemma 2.0, and Mistral NeMo, adjusting parameters to produce datasets mimicking the real transcripts. We assessed the quality of synthetic transcripts using our framework and used a sentence transformer model (trained on both real and synthetic data) to compare changes in the model's percent accuracy when qualitatively coding feedback interactions.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Results</h3>\u0000 \u0000 <p>Synthetic data improved the NLP model's performance in classifying feedback interactions, boosting the average accuracy from 68.4% to 81% with Llama 3.1. Although incorporating synthetic data improved classification, all models produced transcripts that occasionally included extraneous details and failed to capture instructor-dominant discourse.</p>\u0000 </section>\u0000 \u0000 <section>\u0000 \u0000 <h3> Conclusions</h3>\u0000 \u0000 <p>Synthetic data offers an opportunity to expand qualitative research, particularly in contexts where real data for NLP training is limited or hard to obtain; however, transparency in its use is paramount to maintain research integrity.</p>\u0000 </section>\u0000 </div>","PeriodicalId":50206,"journal":{"name":"Journal of Engineering Education","volume":"114 4","pages":""},"PeriodicalIF":3.4,"publicationDate":"2025-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144923477","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0