Research Methods in Applied Linguistics最新文献_第8页

Applying a polytomous Rasch model to investigate Likert scale functioning and L2 writing strategy use 运用多同体Rasch模型研究李克特量表功能和二语写作策略的使用

Research Methods in Applied Linguistics

Pub Date : 2025-07-11 DOI: 10.1016/j.rmal.2025.100240

Apichat Khamboonruang

While Rasch models have been increasingly employed in applied linguistics research, their use remains underexplored in L2 writing strategy research, which has relied primarily on statistical methods that assume continuous data. This study aimed to address this methodological gap by applying a polytomous Rasch modelling approach to investigate Likert scale functioning in the context of L2 writing strategy use. Participants were 172 Thai EFL English-major undergraduates who completed a 26-item, 5-category Likert-type scale designed to measure five strategy domains: metacognitive, effort-regulation, cognitive, social, and affective strategies. The data were analysed using a Rasch rating scale model (RSM) implemented in Winsteps and Facets software programmes. The main results indicated that the RSM analysis provided sound evidence of appropriate item and category functioning, while revealing specific areas for refinement, such as limited item coverage, item redundancy, and category disordering. The RSM analysis also revealed systematic trends in Thai EFL students’ writing strategy use across domains and proficiency levels: metacognitive strategies were used most often and clearly differentiated higher- and lower-achieving students, while social strategies were less common and more frequently used by lower achievers. These findings highlight the value of a polytomous Rasch modelling approach in examining not only rating scale functioning but also writing strategy use. The present findings have implications for rating scale validation and L2 writing strategy instruction.

虽然Rasch模型在应用语言学研究中的应用越来越多，但在二语写作策略研究中的应用仍未得到充分探索，这主要依赖于假设连续数据的统计方法。本研究旨在解决这一方法学上的差距，采用多分体Rasch建模方法来调查二语写作策略使用背景下的李克特量表功能。参与者是172名泰国英语专业的本科生，他们完成了一份26项5类李克特式量表，该量表旨在测量五个策略领域：元认知策略、努力调节策略、认知策略、社会策略和情感策略。使用在Winsteps和Facets软件程序中实现的Rasch评定量表模型（RSM）对数据进行分析。主要结果表明，RSM分析为适当的项目和类别功能提供了可靠的证据，同时揭示了需要改进的特定领域，如有限的项目覆盖，项目冗余和类别无序。RSM分析还揭示了泰国英语学生在不同领域和熟练程度上使用写作策略的系统性趋势：元认知策略的使用频率最高，并且明显区分了成绩较高和较差的学生，而社会策略的使用频率较低，但成绩较差的学生使用频率较高。这些发现突出了多元拉赫建模方法的价值，不仅检查评分量表功能，而且还检查写作策略的使用。本研究结果对评定量表的验证和二语写作策略的指导具有启示意义。

{"title":"Applying a polytomous Rasch model to investigate Likert scale functioning and L2 writing strategy use","authors":"Apichat Khamboonruang","doi":"10.1016/j.rmal.2025.100240","DOIUrl":"10.1016/j.rmal.2025.100240","url":null,"abstract":"<div><div>While Rasch models have been increasingly employed in applied linguistics research, their use remains underexplored in L2 writing strategy research, which has relied primarily on statistical methods that assume continuous data. This study aimed to address this methodological gap by applying a polytomous Rasch modelling approach to investigate Likert scale functioning in the context of L2 writing strategy use. Participants were 172 Thai EFL English-major undergraduates who completed a 26-item, 5-category Likert-type scale designed to measure five strategy domains: metacognitive, effort-regulation, cognitive, social, and affective strategies. The data were analysed using a Rasch rating scale model (RSM) implemented in Winsteps and Facets software programmes. The main results indicated that the RSM analysis provided sound evidence of appropriate item and category functioning, while revealing specific areas for refinement, such as limited item coverage, item redundancy, and category disordering. The RSM analysis also revealed systematic trends in Thai EFL students’ writing strategy use across domains and proficiency levels: metacognitive strategies were used most often and clearly differentiated higher- and lower-achieving students, while social strategies were less common and more frequently used by lower achievers. These findings highlight the value of a polytomous Rasch modelling approach in examining not only rating scale functioning but also writing strategy use. The present findings have implications for rating scale validation and L2 writing strategy instruction.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100240"},"PeriodicalIF":0.0,"publicationDate":"2025-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144597529","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Indonesian cross-linguistic named entity recognition 印尼语跨语言命名实体识别

Research Methods in Applied Linguistics

Pub Date : 2025-07-08 DOI: 10.1016/j.rmal.2025.100236

Danang Arbian Sulistyo , Aji Prasetya Wibawa , Didik Dwi Prasetya , Fadhli Almu’iini Ahda

This study examines the potential of Named Entity Recognition (NER) in translating cross-biblical texts of Indonesian, Madurese, and Javanese. The goal is to enhance translation precision by incorporating entity categorization. The approach involves training an NER model using Conditional Random Fields (CRF) and evaluating its performance on the Book of Joshua. The annotated dataset includes features such as word identity, shape, part-of-speech identifiers, and semantic information. Tagging the data with labels such as Person, Location, and Organization reveals variations in effectiveness across languages. Indonesian yields the highest F1 score (78.69), reflecting consistent performance across all parameters. Although Madurese achieves a high recall for Location entities (82.16), its precision is lower (74.99). Javanese demonstrates strong precision in identifying locations (77.46), but a slightly lower recall score (77.21). The findings suggest the need to tailor the NER model to suit the specific characteristics of low-resource languages for improved translation quality.

本研究探讨了命名实体识别（NER）在印尼语、马杜雷语和爪哇语跨圣经文本翻译中的潜力。目标是通过结合实体分类来提高翻译精度。该方法包括使用条件随机场（Conditional Random Fields， CRF）训练一个NER模型，并评估其在约书亚记上的表现。带注释的数据集包括单词标识、形状、词性标识符和语义信息等特征。用诸如Person、Location和Organization之类的标签标记数据，揭示了不同语言之间有效性的差异。印度尼西亚获得了最高的F1分数（78.69），反映了在所有参数上的一致表现。虽然Madurese对Location实体的查全率较高（82.16），但准确率较低（74.99）。爪哇语在识别位置方面表现出很高的精确度（77.46），但召回率略低（77.21）。研究结果表明，为了提高翻译质量，需要调整NER模型以适应低资源语言的具体特征。

引用次数: 0

Investigating LID in four test conditions - Do instructions, test formats and item positioning matter? 在四种测试条件下调查LID -说明，测试格式和项目定位重要吗？

Research Methods in Applied Linguistics

Pub Date : 2025-07-04 DOI: 10.1016/j.rmal.2025.100233

Hung Tan Ha , Duyen Thi Bich Nguyen , Tim Stoeckel

Recent research has found the Updated Vocabulary Levels Test (UVLT) to have Local Item Dependence (LID), a violation to the central assumption of all Rasch and Item Response Theory models. LID in the UVLT is hypothesized to be caused by a feature of matching tasks: once an option is selected for one target word, it will not be selected for another. It is also hypothesized that if this feature is removed, LID will be reduced. The present study investigated the effects of LID in four test conditions. The first employed the 3:6 matching format of the UVLT with no instruction concerning option recycling. The second used the same format but with instructions encouraging option recycling. The third utilized a multiple-choice format, with items belonging to the same UVLT cluster using identical sets of 6 options and placed adjacently. The fourth also used a multiple-choice, 6-option format, but items sharing identical options were far apart, making them less “local”. Data from 231 Vietnamese EFL learners were analyzed using Rasch unidimensional modelling and Rasch Testlet Modelling (RTM). Person estimates from the unidimensional models and the general dimensions from the RTMs were compared and correlated. Substantial LID was present in Conditions 1–3. Significant distortions of person estimates were found in all test conditions. However, the findings showed that LID had a negligible impact on person ordering in all test conditions.

最近的研究发现，更新词汇水平测试（UVLT）具有局部项目依赖（LID），这违反了所有Rasch和项目反应理论模型的中心假设。假设UVLT中的LID是由匹配任务的一个特征引起的：一旦为一个目标单词选择了一个选项，就不会为另一个目标单词选择它。也有假设认为，如果这个特征被移除，LID将会减少。本研究考察了四种测试条件下LID的影响。第一个采用了UVLT的3:6匹配格式，没有关于选项回收的说明。第二组使用了相同的格式，但附带了鼓励选项回收的说明。第三个使用多项选择格式，属于同一UVLT集群的项目使用相同的6个选项集并相邻放置。第四个实验也采用了6个选项的多选题形式，但共享相同选项的项目相隔甚远，使其不那么“本地化”。采用Rasch一维模型和Rasch测试模型（RTM）对231名越南英语学习者的数据进行了分析。对单维模型的人员估计与rtm的一般维度进行了比较和关联。在条件1-3中存在大量LID。在所有测试条件下，对人的估计都存在显著的扭曲。然而，研究结果表明，在所有测试条件下，LID对人员订购的影响可以忽略不计。

{"title":"Investigating LID in four test conditions - Do instructions, test formats and item positioning matter?","authors":"Hung Tan Ha , Duyen Thi Bich Nguyen , Tim Stoeckel","doi":"10.1016/j.rmal.2025.100233","DOIUrl":"10.1016/j.rmal.2025.100233","url":null,"abstract":"<div><div>Recent research has found the Updated Vocabulary Levels Test (UVLT) to have Local Item Dependence (LID), a violation to the central assumption of all Rasch and Item Response Theory models. LID in the UVLT is hypothesized to be caused by a feature of matching tasks: once an option is selected for one target word, it will not be selected for another. It is also hypothesized that if this feature is removed, LID will be reduced. The present study investigated the effects of LID in four test conditions. The first employed the 3:6 matching format of the UVLT with no instruction concerning option recycling. The second used the same format but with instructions encouraging option recycling. The third utilized a multiple-choice format, with items belonging to the same UVLT cluster using identical sets of 6 options and placed adjacently. The fourth also used a multiple-choice, 6-option format, but items sharing identical options were far apart, making them less “local”. Data from 231 Vietnamese EFL learners were analyzed using Rasch unidimensional modelling and Rasch Testlet Modelling (RTM). Person estimates from the unidimensional models and the general dimensions from the RTMs were compared and correlated. Substantial LID was present in Conditions 1–3. Significant distortions of person estimates were found in all test conditions. However, the findings showed that LID had a negligible impact on person ordering in all test conditions.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100233"},"PeriodicalIF":0.0,"publicationDate":"2025-07-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144556765","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Do data collection methods matter for self-reported L2 individual differences questionnaires? In-person vs crowdsourced data 数据收集方法对自我报告的第二语言个体差异问卷有影响吗？面对面vs众包数据

Research Methods in Applied Linguistics

Pub Date : 2025-07-03 DOI: 10.1016/j.rmal.2025.100235

Ruirui Jia , Ekaterina Sudina , Kejun Du

Crowdsourcing offers great advantages in data collection by enabling researchers to recruit a large number of participants across geographical boundaries within a short period of time. Despite the benefits of crowdsourcing, no study has explored its validity in collecting self-reported individual differences (ID) data in second language (L2) research. The present study aims to address this gap by examining crowdsourcing as a viable alternative or complementary tool to traditional in-person data collection. We recruited a total of 209 in-person and 209 crowdsourced participants for comparison. Both groups completed the short versions of the Foreign Language Classroom Anxiety Scale and the Foreign Language Enjoyment Scale, provided their demographic and language learning background information, and completed the LexTALE test. Measurement invariance testing revealed that most (sub)constructs exhibited partial or full invariance, indicating stability in the measurement systems across both data collection settings. However, crowdsourced participants reported higher enjoyment and lower anxiety than in-person participants. These differences can be attributed to the more relaxed mental state of the crowdsourced participants who completed the survey outside of the classroom. Moreover, some crowdsourced participants tended to overrate their English proficiency and exhibited potentially dishonest behavior during the LexTALE test. These findings suggest that although crowdsourcing offers valuable opportunities for data collection in L2 ID research, the potential for inflated self-assessments and questionable behavior in an unsupervised online testing environment must be considered. Thus, the use of crowdsourcing platforms to collect self-reported L2 ID data requires caution and careful preparation.

众包在数据收集方面具有很大的优势，它使研究人员能够在短时间内招募到大量跨越地理边界的参与者。尽管众包有很多好处，但还没有研究探讨它在第二语言研究中收集自我报告的个体差异（ID）数据的有效性。本研究旨在通过检验众包作为传统现场数据收集的可行替代或补充工具来解决这一差距。我们共招募了209名现场参与者和众包参与者进行比较。两组都完成了简短版的外语课堂焦虑量表和外语享受量表，提供了他们的人口统计和语言学习背景信息，并完成了LexTALE测试。测量不变性测试显示，大多数（子）结构表现出部分或完全不变性，表明测量系统在两种数据收集设置中的稳定性。然而，与面对面的参与者相比，众包参与者报告了更高的享受和更低的焦虑。这些差异可以归因于在课堂外完成调查的众包参与者更放松的精神状态。此外，一些众包参与者倾向于高估自己的英语水平，并在LexTALE测试中表现出潜在的不诚实行为。这些发现表明，尽管众包为L2 ID研究中的数据收集提供了宝贵的机会，但必须考虑在无监督的在线测试环境中夸大自我评估和可疑行为的可能性。因此，使用众包平台收集自我报告的L2 ID数据需要谨慎和仔细的准备。

{"title":"Do data collection methods matter for self-reported L2 individual differences questionnaires? In-person vs crowdsourced data","authors":"Ruirui Jia , Ekaterina Sudina , Kejun Du","doi":"10.1016/j.rmal.2025.100235","DOIUrl":"10.1016/j.rmal.2025.100235","url":null,"abstract":"<div><div>Crowdsourcing offers great advantages in data collection by enabling researchers to recruit a large number of participants across geographical boundaries within a short period of time. Despite the benefits of crowdsourcing, no study has explored its validity in collecting self-reported individual differences (ID) data in second language (L2) research. The present study aims to address this gap by examining crowdsourcing as a viable alternative or complementary tool to traditional in-person data collection. We recruited a total of 209 in-person and 209 crowdsourced participants for comparison. Both groups completed the short versions of the Foreign Language Classroom Anxiety Scale and the Foreign Language Enjoyment Scale, provided their demographic and language learning background information, and completed the LexTALE test. Measurement invariance testing revealed that most (sub)constructs exhibited partial or full invariance, indicating stability in the measurement systems across both data collection settings. However, crowdsourced participants reported higher enjoyment and lower anxiety than in-person participants. These differences can be attributed to the more relaxed mental state of the crowdsourced participants who completed the survey outside of the classroom. Moreover, some crowdsourced participants tended to overrate their English proficiency and exhibited potentially dishonest behavior during the LexTALE test. These findings suggest that although crowdsourcing offers valuable opportunities for data collection in L2 ID research, the potential for inflated self-assessments and questionable behavior in an unsupervised online testing environment must be considered. Thus, the use of crowdsourcing platforms to collect self-reported L2 ID data requires caution and careful preparation.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100235"},"PeriodicalIF":0.0,"publicationDate":"2025-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144549574","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Oral task repetition research via videoconferencing 基于视频会议的口语任务重复研究

Research Methods in Applied Linguistics

Pub Date : 2025-07-03 DOI: 10.1016/j.rmal.2025.100232

Joe Kakitani

A substantial body of research has demonstrated the benefits of oral task repetition in enhancing second language (L2) performance. However, empirical studies investigating its effects on L2 development through longitudinal designs remain limited. This limitation may be partly due to the methodological challenges of traditional classroom- and laboratory-based research, such as participant attrition and scheduling difficulties. This paper explores the potential of online oral experimentation via videoconferencing—experiments conducted through synchronous computer-mediated communication using platforms like Zoom and Microsoft Teams—to advance L2 oral task repetition research. After reviewing research on task repetition and the methodological characteristics of conventional classroom- and laboratory-based studies that may present challenges within this domain, this article discusses the advantages of online experiments conducted via videoconferencing, including greater convenience and flexibility, increased efficiency, improved control of extraneous factors, and automated speech transcription. In addition, it examines the ecological validity of online video-based oral experiments. Methodological recommendations are also provided to help researchers address some of the challenges associated with conducting experiments via videoconferencing.

大量的研究已经证明口语任务重复在提高第二语言（L2）表现方面的好处。然而，通过纵向设计调查其对第二语言发展影响的实证研究仍然有限。这种限制可能部分是由于传统的基于课堂和实验室的研究在方法上的挑战，例如参与者流失和安排困难。本文探讨了通过视频会议进行在线口语实验的潜力——使用Zoom和Microsoft teams等平台通过同步计算机媒介通信进行的实验——以推进第二语言口语任务重复研究。在回顾了关于任务重复的研究以及传统的基于课堂和实验室的研究的方法特征后，本文讨论了通过视频会议进行在线实验的优势，包括更大的便利性和灵活性，提高效率，改进对外来因素的控制，以及自动语音转录。此外，它还检验了基于在线视频的口腔实验的生态有效性。还提供了方法建议，以帮助研究人员解决与通过视频会议进行实验相关的一些挑战。

{"title":"Oral task repetition research via videoconferencing","authors":"Joe Kakitani","doi":"10.1016/j.rmal.2025.100232","DOIUrl":"10.1016/j.rmal.2025.100232","url":null,"abstract":"<div><div>A substantial body of research has demonstrated the benefits of oral task repetition in enhancing second language (L2) performance. However, empirical studies investigating its effects on L2 development through longitudinal designs remain limited. This limitation may be partly due to the methodological challenges of traditional classroom- and laboratory-based research, such as participant attrition and scheduling difficulties. This paper explores the potential of online oral experimentation via videoconferencing—experiments conducted through synchronous computer-mediated communication using platforms like Zoom and Microsoft Teams—to advance L2 oral task repetition research. After reviewing research on task repetition and the methodological characteristics of conventional classroom- and laboratory-based studies that may present challenges within this domain, this article discusses the advantages of online experiments conducted via videoconferencing, including greater convenience and flexibility, increased efficiency, improved control of extraneous factors, and automated speech transcription. In addition, it examines the ecological validity of online video-based oral experiments. Methodological recommendations are also provided to help researchers address some of the challenges associated with conducting experiments via videoconferencing.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100232"},"PeriodicalIF":0.0,"publicationDate":"2025-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144549575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Leading a scoping review on L2 pronunciation: Some key elements of methodology 领导二语发音的范围审查：方法论的一些关键要素

Research Methods in Applied Linguistics

Pub Date : 2025-07-02 DOI: 10.1016/j.rmal.2025.100216

Linda Terrier , Marie Garnier , Saandia Ali

This article describes the methodology of a scoping review covering 25 years of research on L2 English pronunciation. We focus on two key methodological steps required in any scoping review: identifying the information source and selecting the studies. We present a rationale for employing a manual search across prominent journals in the fields of phonetics and phonology, second language acquisition, and second language learning and teaching. We describe how we delineated the scope of the review by identifying 35 prominent journals and how we organized teamwork to select relevant studies. We show that seemingly straightforward inclusion criteria (L2 English, empirical research, and pronunciation) raise questions about the objects of study in the field. The final corpus includes 463 articles published in the 35 identified journals between 1996 and 2020. We demonstrate that Arksey and O’Malley’s framework for scoping reviews can be applied and adapted to the specificities of L2 English pronunciation research, but we also highlight the challenge of iterativity in study selection. As we present the distribution of articles over time and across journals, we make recommendations for future scoping reviews regarding the time span of the review and the identification of the initial information source. In particular, the Journal of Second Language Pronunciation, which stands out as a central venue for L2 English pronunciation research, would have been missed had we used a more typical keyword search across academic databases.

本文描述了一项涵盖25年第二语言英语发音研究的范围审查方法。我们集中在两个关键的方法学步骤需要在任何范围审查：确定信息来源和选择研究。我们提出了在语音学、第二语言习得和第二语言学习与教学领域采用人工搜索的基本原理。我们描述了我们如何通过识别35种著名期刊来划定综述的范围，以及我们如何组织团队来选择相关的研究。我们表明，看似简单的纳入标准（第二语言英语、实证研究和发音）引发了对该领域研究对象的质疑。最终的语料库包括1996年至2020年间在35个确定的期刊上发表的463篇文章。我们证明了Arksey和O 'Malley的范围审查框架可以应用并适应第二语言英语发音研究的特殊性，但我们也强调了研究选择中的迭代性挑战。当我们呈现文章随时间和期刊的分布时，我们就审查的时间跨度和初始信息源的确定为未来的范围审查提出建议。特别是《第二语言发音杂志》，它作为第二语言英语发音研究的中心场所，如果我们在学术数据库中使用更典型的关键词搜索，就会错过它。

{"title":"Leading a scoping review on L2 pronunciation: Some key elements of methodology","authors":"Linda Terrier , Marie Garnier , Saandia Ali","doi":"10.1016/j.rmal.2025.100216","DOIUrl":"10.1016/j.rmal.2025.100216","url":null,"abstract":"<div><div>This article describes the methodology of a scoping review covering 25 years of research on L2 English pronunciation. We focus on two key methodological steps required in any scoping review: identifying the information source and selecting the studies. We present a rationale for employing a manual search across prominent journals in the fields of phonetics and phonology, second language acquisition, and second language learning and teaching. We describe how we delineated the scope of the review by identifying 35 prominent journals and how we organized teamwork to select relevant studies. We show that seemingly straightforward inclusion criteria (L2 English, empirical research, and pronunciation) raise questions about the objects of study in the field. The final corpus includes 463 articles published in the 35 identified journals between 1996 and 2020. We demonstrate that Arksey and O’Malley’s framework for scoping reviews can be applied and adapted to the specificities of L2 English pronunciation research, but we also highlight the challenge of iterativity in study selection. As we present the distribution of articles over time and across journals, we make recommendations for future scoping reviews regarding the time span of the review and the identification of the initial information source. In particular, the <em>Journal of Second Language Pronunciation</em>, which stands out as a central venue for L2 English pronunciation research, would have been missed had we used a more typical keyword search across academic databases.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100216"},"PeriodicalIF":0.0,"publicationDate":"2025-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144524220","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Predicting the CEFR level of English listening texts with machine learning methods 用机器学习方法预测英语听力文本的CEFR水平

Research Methods in Applied Linguistics

Pub Date : 2025-07-01 DOI: 10.1016/j.rmal.2025.100234

Christopher Robert Cooper

Comprehension in listening texts is often judged by lexical coverage. However, this might not be easily interpretable for language teachers. The CEFR is becoming increasingly influential due to its standardized descriptors across languages. Learners are often placed into classes based on proficiency level, therefore a CEFR level is likely more interpretable than lexical coverage when judging listening text difficulty. Machine learning methods have been used to predict the CEFR level of English reading texts and learner writing, but no such studies exist for listening. The current study hopes to bridge this gap by investigating the potential to predict the CEFR level of listening texts. A corpus of CEFR-labelled listening texts (728 texts, 345,104 words) was compiled for text classification. Three types of variables were created from the corpus data to evaluate comparative predictive accuracy. The first method used linguistic and acoustic features. The others used text embeddings, which represent semantic meaning. The data was split into four classes: A1, A2, B1, and B2+. The accuracy of each method was evaluated by comparing the predicted label in the test data with the label from the original text. The most accurate method used OpenAI embeddings and Support Vector Machines. The overall accuracy was 0.81, with macro averages of precision = 0.75, recall = 0.78, and f-score = 0.76, indicating balanced classification performance across CEFR levels. This method has the potential to predict the CEFR level of listening texts, which could help practitioners and researchers match learners and participants to appropriate listening texts.

听力文本的理解通常是通过词汇覆盖来判断的。然而，对于语言教师来说，这可能不容易解释。CEFR由于其跨语言的标准化描述符而变得越来越有影响力。学习者通常根据熟练程度分组，因此在判断听力文本难度时，CEFR水平可能比词汇覆盖范围更容易解释。机器学习方法已经被用于预测英语阅读文本和学习者写作的CEFR水平，但还没有关于听力的研究。目前的研究希望通过调查预测听力文本的CEFR水平的潜力来弥合这一差距。编制了cefr标记听力文本语料库（728个文本，345,104个单词）进行文本分类。从语料库数据中创建了三种类型的变量来评估比较预测的准确性。第一种方法利用语言和声学特征。其他的则使用文本嵌入，表示语义。数据分为A1、A2、B1和B2+四类。通过比较测试数据中的预测标签与原始文本中的标签来评估每种方法的准确性。最准确的方法是使用OpenAI嵌入和支持向量机。总体准确率为0.81，宏观平均精度为0.75，召回率为0.78，f-score为0.76，表明CEFR各水平的分类性能平衡。该方法具有预测听力文本CEFR水平的潜力，可以帮助从业者和研究人员将学习者和参与者匹配到合适的听力文本。

{"title":"Predicting the CEFR level of English listening texts with machine learning methods","authors":"Christopher Robert Cooper","doi":"10.1016/j.rmal.2025.100234","DOIUrl":"10.1016/j.rmal.2025.100234","url":null,"abstract":"<div><div>Comprehension in listening texts is often judged by lexical coverage. However, this might not be easily interpretable for language teachers. The CEFR is becoming increasingly influential due to its standardized descriptors across languages. Learners are often placed into classes based on proficiency level, therefore a CEFR level is likely more interpretable than lexical coverage when judging listening text difficulty. Machine learning methods have been used to predict the CEFR level of English reading texts and learner writing, but no such studies exist for listening. The current study hopes to bridge this gap by investigating the potential to predict the CEFR level of listening texts. A corpus of CEFR-labelled listening texts (728 texts, 345,104 words) was compiled for text classification. Three types of variables were created from the corpus data to evaluate comparative predictive accuracy. The first method used linguistic and acoustic features. The others used text embeddings, which represent semantic meaning. The data was split into four classes: A1, A2, B1, and B2+. The accuracy of each method was evaluated by comparing the predicted label in the test data with the label from the original text. The most accurate method used OpenAI embeddings and Support Vector Machines. The overall accuracy was 0.81, with macro averages of precision = 0.75, recall = 0.78, and f-score = 0.76, indicating balanced classification performance across CEFR levels. This method has the potential to predict the CEFR level of listening texts, which could help practitioners and researchers match learners and participants to appropriate listening texts.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100234"},"PeriodicalIF":0.0,"publicationDate":"2025-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144518520","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating the linguistic complexity of machine translation and LLMs for EFL/ESL applications: An entropy weight method 评估机器翻译和llm在EFL/ESL应用中的语言复杂性：一种熵权法

Research Methods in Applied Linguistics

Pub Date : 2025-06-30 DOI: 10.1016/j.rmal.2025.100229

Yingqi Huang, Dechao Li, Andrew K.F. Cheung

English as a Foreign and Second Language (EFL/ESL) learners are increasingly using machine translation (MT) tools such as neural machine translations (NMTs) and large language models (LLMs) to enhance their language learning and translation processes due to their accuracy and efficiency in both cost and time compared with human translation. Given the distinct linguistic features exhibited by NMTs and LLMs, it is crucial to assess the linguistic complexity of texts produced by these tools to optimize their use in EFL/ESL teaching and learning. This study examines two forms of absolute linguistic complexity, namely lexical complexity and syntactic complexity, that influence EFL/ESL activities. Lexical complexity affects vocabulary recognition and semantic processing, while syntactic complexity influences sentence parsing and the internalization of grammatical rules. As both dimensions are multi-faceted and involve numerous indices that may vary in different directions (e.g., high values in certain measures and lower in others), an entropy weight method (EWM) is employed to assign data-driven weights and derive a balanced holistic complexity score. This approach enables a systematic comparison of translation outputs from NMTs (Google Translate, DeepL) and LLMs (ChatGPT-4o, OpenAI-o1). The findings reveal that LLMs generally exhibit higher holistic linguistic complexity, whereas NMTs tend to produce simpler translations. Pedagogically, LLM-translated texts may serve as more effective input for advanced language learners in EFL/ESL contexts, while NMT outputs may be more suitable for those with less linguistic proficiency.

英语作为外语和第二语言（EFL/ESL）学习者越来越多地使用机器翻译（MT）工具，如神经机器翻译（nmt）和大型语言模型（llm），以提高他们的语言学习和翻译过程，因为与人工翻译相比，它们在成本和时间上都更加准确和高效。鉴于nmt和llm所表现出的不同语言特征，评估这些工具产生的文本的语言复杂性以优化它们在EFL/ESL教学和学习中的使用是至关重要的。本研究考察了绝对语言复杂性的两种形式，即词汇复杂性和句法复杂性，它们对EFL/ESL活动的影响。词汇复杂性影响词汇识别和语义加工，句法复杂性影响句子解析和语法规则内化。由于这两个维度都是多方面的，并且涉及许多指标，这些指标可能在不同的方向上变化（例如，某些措施的值较高，而其他措施的值较低），因此采用熵权法（EWM）来分配数据驱动的权重，并得出平衡的整体复杂性评分。这种方法可以对nmt（谷歌Translate， DeepL）和llm （chatgpt - 40, openai - 01）的翻译输出进行系统比较。研究结果表明，法学硕士通常表现出更高的整体语言复杂性，而nmt倾向于产生更简单的翻译。在教学上，法学硕士翻译的文本可以作为高级语言学习者在EFL/ESL环境中更有效的输入，而NMT输出可能更适合那些语言水平较低的人。

{"title":"Evaluating the linguistic complexity of machine translation and LLMs for EFL/ESL applications: An entropy weight method","authors":"Yingqi Huang, Dechao Li, Andrew K.F. Cheung","doi":"10.1016/j.rmal.2025.100229","DOIUrl":"10.1016/j.rmal.2025.100229","url":null,"abstract":"<div><div>English as a Foreign and Second Language (EFL/ESL) learners are increasingly using machine translation (MT) tools such as neural machine translations (NMTs) and large language models (LLMs) to enhance their language learning and translation processes due to their accuracy and efficiency in both cost and time compared with human translation. Given the distinct linguistic features exhibited by NMTs and LLMs, it is crucial to assess the linguistic complexity of texts produced by these tools to optimize their use in EFL/ESL teaching and learning. This study examines two forms of absolute linguistic complexity, namely lexical complexity and syntactic complexity, that influence EFL/ESL activities. Lexical complexity affects vocabulary recognition and semantic processing, while syntactic complexity influences sentence parsing and the internalization of grammatical rules. As both dimensions are multi-faceted and involve numerous indices that may vary in different directions (e.g., high values in certain measures and lower in others), an entropy weight method (EWM) is employed to assign data-driven weights and derive a balanced holistic complexity score. This approach enables a systematic comparison of translation outputs from NMTs (Google Translate, DeepL) and LLMs (ChatGPT-4o, OpenAI-o1). The findings reveal that LLMs generally exhibit higher holistic linguistic complexity, whereas NMTs tend to produce simpler translations. Pedagogically, LLM-translated texts may serve as more effective input for advanced language learners in EFL/ESL contexts, while NMT outputs may be more suitable for those with less linguistic proficiency.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100229"},"PeriodicalIF":0.0,"publicationDate":"2025-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144513924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Bayesian approach to small samples: Mixed-effects modeling in L2 interventional research 小样本贝叶斯方法：L2介入性研究中的混合效应建模

Research Methods in Applied Linguistics

Pub Date : 2025-06-27 DOI: 10.1016/j.rmal.2025.100231

Man Ho Ivy Wong

Small sample sizes are a common challenge in second language (L2) research, particularly in classroom-based studies or exploratory intervention work. Traditional frequentist approaches often lack the flexibility needed to analyse such data meaningfully. This paper presents a two-study Bayesian tutorial designed to address the small-N problem using logistic mixed-effects models. In Study 1, we analyse pilot data from 27 final-year or postgraduate students across three instructional conditions, using Bayesian mixed-effects modelling with non-informative (uniform) priors to explore effects of instruction, time, conditional type, and proficiency on participants’ binary responses in two language assessment tasks (a processing test and a production test). In Study 2, we build on the pilot by modelling follow-up data from a refined version of the study, focusing on the one treatment group only. Here, we incorporate highly informed priors derived from the posterior estimates of Study 1, demonstrating how prior information can improve estimation and interpretability, even with small datasets. This paper offers practical guidance on specifying priors, modelling binary outcomes, and applying Bayesian reasoning across iterative L2 research designs.

小样本量是第二语言研究中常见的挑战，特别是在课堂研究或探索性干预工作中。传统的频率论方法往往缺乏对此类数据进行有意义分析所需的灵活性。本文提出了一个双研究贝叶斯教程，旨在解决使用逻辑混合效应模型的小n问题。在研究1中，我们分析了27名高年级或研究生在三种教学条件下的试点数据，使用非信息（统一）先验的贝叶斯混合效应模型来探索教学、时间、条件类型和熟练程度对参与者在两个语言评估任务（加工测试和生产测试）中的二元反应的影响。在研究2中，我们在试点的基础上，通过对研究的改进版本的随访数据进行建模，只关注一个治疗组。在这里，我们结合了从研究1的后验估计中得到的高度知情的先验，证明了先验信息如何提高估计和可解释性，即使是小数据集。本文为指定先验、二元结果建模以及在迭代L2研究设计中应用贝叶斯推理提供了实用指导。

{"title":"A Bayesian approach to small samples: Mixed-effects modeling in L2 interventional research","authors":"Man Ho Ivy Wong","doi":"10.1016/j.rmal.2025.100231","DOIUrl":"10.1016/j.rmal.2025.100231","url":null,"abstract":"<div><div>Small sample sizes are a common challenge in second language (L2) research, particularly in classroom-based studies or exploratory intervention work. Traditional frequentist approaches often lack the flexibility needed to analyse such data meaningfully. This paper presents a two-study Bayesian tutorial designed to address the small-N problem using logistic mixed-effects models. In Study 1<strong>,</strong> we analyse pilot data from 27 final-year or postgraduate students across three instructional conditions, using Bayesian mixed-effects modelling with non-informative (uniform) priors to explore effects of instruction, time, conditional type, and proficiency on participants’ binary responses in two language assessment tasks (a processing test and a production test). In Study 2, we build on the pilot by modelling follow-up data from a refined version of the study, focusing on the one treatment group only. Here, we incorporate highly informed priors derived from the posterior estimates of Study 1, demonstrating how prior information can improve estimation and interpretability, even with small datasets. This paper offers practical guidance on specifying priors, modelling binary outcomes, and applying Bayesian reasoning across iterative L2 research designs.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100231"},"PeriodicalIF":0.0,"publicationDate":"2025-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144490695","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Language without borders: A step-by-step guide to analyzing webcam eye-tracking data for L2 research 语言无国界：一步一步的指导分析网络摄像头眼动追踪数据的第二语言研究

Research Methods in Applied Linguistics

Pub Date : 2025-06-24 DOI: 10.1016/j.rmal.2025.100226

Jason Geller , Yanina Prystauka , Sarah E. Colby , Julia R. Drouin

Eye-tracking has become a valuable tool for studying cognitive processes in second language acquisition and bilingualism (Godfroid et al., 2024). While research-grade infrared eye-trackers are commonly used, several factors limit their widespread adoption. Recently, consumer-based webcam eye-tracking has emerged as an attractive alternative, requiring only a personal webcam and internet access. However, webcam-based eye-tracking introduces unique design and preprocessing challenges that must be addressed to ensure valid results. To help researchers navigate these challenges, we developed a comprehensive tutorial focused on visual world webcam eye-tracking for second language research. This guide covers key preprocessing steps—from reading in raw data to visualization and analysis—highlighting the open-source R package webgazeR (Geller, 2025), freely available at: https://github.com/jgeller112/webgazer. To demonstrate these steps, we analyze data collected via the Gorilla platform (Anwyl-Irvine et al., 2020) using a single-word Spanish visual world paradigm (VWP), showcasing evidence of competition both within and between Spanish and English. This tutorial aims to empower researchers by providing a step-by-step guide to successfully conduct webcam-based visual world eye-tracking studies. To follow along, please download the complete manuscript, code, and data from: https://github.com/jgeller112/L2_VWP_Webcam.

眼动追踪已成为研究第二语言习得和双语认知过程的重要工具（Godfroid et al., 2024）。虽然研究级红外眼动仪被广泛使用，但有几个因素限制了它们的广泛采用。最近，基于消费者的网络摄像头眼球追踪已经成为一种有吸引力的选择，只需要一个个人摄像头和互联网接入。然而，基于网络摄像头的眼动追踪引入了独特的设计和预处理挑战，必须解决这些挑战以确保有效的结果。为了帮助研究人员应对这些挑战，我们开发了一个全面的教程，专注于第二语言研究的视觉世界网络摄像头眼动追踪。本指南涵盖了关键的预处理步骤-从读取原始数据到可视化和分析-重点介绍了开源R包webgazeR (Geller, 2025)，免费提供：https://github.com/jgeller112/webgazer。为了演示这些步骤，我们使用单字西班牙语视觉世界范式（VWP）分析了通过大猩猩平台（Anwyl-Irvine等人，2020）收集的数据，展示了西班牙语和英语内部和之间竞争的证据。本教程旨在通过提供一步一步的指导，使研究人员能够成功地进行基于网络摄像头的视觉世界眼动追踪研究。请从https://github.com/jgeller112/L2_VWP_Webcam下载完整的手稿、代码和数据。

{"title":"Language without borders: A step-by-step guide to analyzing webcam eye-tracking data for L2 research","authors":"Jason Geller , Yanina Prystauka , Sarah E. Colby , Julia R. Drouin","doi":"10.1016/j.rmal.2025.100226","DOIUrl":"10.1016/j.rmal.2025.100226","url":null,"abstract":"<div><div>Eye-tracking has become a valuable tool for studying cognitive processes in second language acquisition and bilingualism (Godfroid et al., 2024). While research-grade infrared eye-trackers are commonly used, several factors limit their widespread adoption. Recently, consumer-based webcam eye-tracking has emerged as an attractive alternative, requiring only a personal webcam and internet access. However, webcam-based eye-tracking introduces unique design and preprocessing challenges that must be addressed to ensure valid results. To help researchers navigate these challenges, we developed a comprehensive tutorial focused on visual world webcam eye-tracking for second language research. This guide covers key preprocessing steps—from reading in raw data to visualization and analysis—highlighting the open-source R package webgazeR (Geller, 2025), freely available at: <span><span>https://github.com/jgeller112/webgazer</span><svg><path></path></svg></span>. To demonstrate these steps, we analyze data collected via the Gorilla platform (Anwyl-Irvine et al., 2020) using a single-word Spanish visual world paradigm (VWP), showcasing evidence of competition both within and between Spanish and English. This tutorial aims to empower researchers by providing a step-by-step guide to successfully conduct webcam-based visual world eye-tracking studies. To follow along, please download the complete manuscript, code, and data from: <span><span>https://github.com/jgeller112/L2_VWP_Webcam</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100226"},"PeriodicalIF":0.0,"publicationDate":"2025-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144471641","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0