Research Methods in Applied Linguistics最新文献

This study examines the coding procedures, practices, and measures of interrater agreement (IRA) for conducting rhetorical move analysis. We first systematically reviewed 65 English for Academic Purposes (EAP) genre studies, focusing primarily on their IRA rationales and practices. Based on the review outcome, we explored three IRA measures: percent agreement, Kappa coefficients (κ), and Krippendorff's multi-valued alpha (mv-α) in terms of their suitability for measuring IRA in rhetorical move analysis under different scenarios such as different sample sizes, number of coders, and multivalued coding. Specifically, the three researchers annotated 35 texts from the Corpus of Social Sciences Research Article Introductions (COSSRAI) (Lu et al., 2020) and subsequently compared the potential of the three IRA measures. Drawing on findings of the systematic review and the corpus study, we recommended strategies for ensuring methodological rigors and transparency in annotation and IRA practices for EAP genre analysis.

Vocabulary researchers are increasingly making use of the eye-tracking method to investigate topics that were traditionally examined using offline tests. Eye-tracking provides a direct measure of processing effort and attention allocation to lexical items. Previous research has provided useful insights about how different types of lexical items are processed in context and of the factors that affect processing patterns. More recent research has also contributed to our understanding of how unknown lexical items are processed in different learning conditions and its relation to learning gains. While eye-tracking has clearly supported researchers in gaining a clearer and more comprehensive understanding of vocabulary processing and learning, and has become a crucial tool in vocabulary research, currently there are no methodological discussions of eye-tracking with a particular focus on vocabulary. This special issue aims at addressing this gap and provides a detailed discussion of the uses and applications of eye-tracking in vocabulary research. This introduction to the special issue provides a brief review of current topics in eye-tracking based vocabulary research, identifying the main gaps that the special issue sets to answer, and introduces the different contributions of the special issue. The collection of papers that constitute this special issue aims at supporting researchers in conducting eye-tracking studies on lexical processing and learning and thus will help to move the field forward.

Applied linguists often aim to conduct research with implications for practitioners (widely construed), but only recently has a research agenda emerged which empirically investigates whether this aim is being regularly met (see Sato, 2023). While multiple survey studies (e.g., Marsden & Kasprowicz, 2017; Sato et al., 2021) have made important contributions to this literature, no study has systematically examined how researchers offer implications to intended practitioner audiences. To explore this topic, we conducted a systematic analysis of 118 empirical studies published in TESOL Quarterly. We developed highly reliable scales (avg. IRR = 0.88) to assess the quantity, salience, and quality of pedagogical implications (PIs). Qualitative genre analysis further allowed us to synthesize the features of highly informative PIs. Based on these findings, we offer concrete recommendations for maximizing PI quality for researchers, editors, and reviewers.

The last ten years have seen an increased use of the eye-tracking methodology in the investigation of vocabulary processing in both L1 and L2. In particular, some of the advantages afforded by this method – such as, the ability to analyse multiple areas of interest (AOIs) and to distinguish between early and late stages of processing – have contributed to our deeper understanding of the mechanisms behind the online processing of sequences above the word level, known as multi-word expressions (MWEs). One key finding widely reported in the eye-tracking literature is that MWEs are processed faster than novel sequences. Different properties of MWEs – frequency, figurativeness, literality, familiarity, transparency, predictability, collocation strength, contextual predictability, adjacency, flexibility, modifiability, and congruency – have been shown to contribute to this processing advantage and to affect different types of MWEs differently. In this review, we first focus on the specific methodological aspects that need to be considered in the application of eye-tracking in the investigation of MWEs. We then discuss how these aspects have been treated in the literature to examine the different properties of MWEs.

This methods tutorial introduces the Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC), a linguistic analysis tool developed by Kyle (2016). In the first part, the tutorial offers an accessible introduction to the key concepts and considerations for using the syntactic complexity and sophistication measures provided by TAASSC. It also overviews the incorporation of new lexico-grammatical features in more recent versions of TAASSC and compares the new versions with earlier versions. The second part of the tutorial guides users through the tool's application by demonstrating a case study of applying TAASSC to a linguistic analysis. It details the process of utilising TAASSC to obtain initial index profiles and conducting mixed-effects modelling using R. The focus is on analysing one aspect of the phrasal complexity development in British school children's writing across academic years, employing mixed-effects models to account for complex data structures and observation interdependence. This tutorial equips researchers with the knowledge to effectively employ TAASSC in linguistic analysis and showcases its utility in exploring syntactic complexity development.

A first and critical step towards successful engagement with replication is to provide justifications to replicate the initial study. Despite existing guidelines, many researchers are still left uninformed of how to justify their replication. In order to explore what justifications were made in the past and thereby to provide additional guidelines, the present study conducted a systematic review of calls for replication in the journal of Language Teaching. The study analyzed 24 calls for replication published from 2010 to 2020 that encouraged replication of 50 initial studies published from 1976 to 2019. These calls for replication were coded for suggested modifications to initial studies and aspects of validity. Three hundred suggested modifications were identified, many of which were associated with external validity (38%) or internal validity (34%). Construct validity (15%) and statistical conclusion validity (7%) were rarely mentioned. Further analyses showed that suggestions on exploratory replication studies tended to address construct validity (e.g., pragmatic competence) and focus less on external validity, but the opposite was true for suggestions on confirmatory replication studies (e.g., corrective feedback). Suggestions for future replication research and calls for replication are discussed.

Analysis of existing datasets of eye movements in reading is a valuable tool for vocabulary research because it allows researchers to examine word recognition in an authentic context. We argue that such secondary analysis is an important addition to new experimental studies and existing mega-studies because it examines word recognition in real text rather than in crammed conditions or in isolation. Corpora in which participants read long texts are particularly interesting because they provide rich material that can be better controlled for confounding variables, but a collection of small data sets can also be interesting because it contains more variation than is typically possible in a single study. We discuss the considerations to take into account when dealing with eye movement data in reading and urge colleagues to make their eye movement data available in the spirit of open science so that a larger database can be built more quickly.

The present study aimed to validate a self-report questionnaire created and used in Yabukoshi (2023) to assess learners’ perceived use of self-regulated learning (SRL) strategies for practicing listening skills to improve the TOEIC® listening scores outside the classroom in an English as a foreign language (EFL) context. The instrument comprised 14 items on metacognitive SRL strategies used for out-of-class listening practice, which were divided into three subscales: strategies used in the forethought, performance, and self-reflection phases, drawing on Zimmerman's SRL model. The questionnaire was administered to 135 EFL undergraduates in Japan and subjected to a psychometric evaluation to determine its structural aspect of validity through reliability tests and confirmatory factor analysis (CFA), as well as to assess its convergent and discriminant validity and its external aspect of validity (i.e., predictive validity). The instrument had sufficient internal consistency based on reliability tests (Cronbach's alpha). The CFA results indicated a relatively good and reasonable model fit, and the convergent validity was found to be adequate. The discriminant validity was not satisfactory, which was discussed based on SRL theory and previous empirical evidence. Multiple regression analysis revealed that the three strategy factors as a whole were a strong predictor of learners’ listening proficiency. The empirical evidence provides preliminary support for the application of SRL theory to the field of EFL listening. The results of this study contribute to the limited research on the assessment of SRL strategy use for listening practice outside EFL classrooms.

This study investigates ChatGPT's potential for measuring linguistic accuracy in second language writing for research purposes. We processed 100 L2 essays across five proficiency levels with ChatGPT-4 and manually coded for precision and recall with regard to ChatGPT's identification of errors. Our findings indicate a strong correlation (ρ = 0.97 using one method and .94 using another method) between ChatGPT's error detection and human coding, although this correlation diminishes with lower proficiency levels. While ChatGPT infrequently misidentifies errors, it often underestimates the total error count. The study also highlights ChatGPT's limitations, such as the issue of consistency, and provides guidelines for future research applications.

Learners need to know a considerable number of words to function in a second or foreign language. To help increase their word knowledge, learners are encouraged to engage in activities that provide a rich source of vocabulary like listening to music and audio books, and watching films, television, and video. In many of these types of activities, learners can listen to and read ‘matched’ content (i.e., text is both written and aural). For example, viewing television programs and films is often accompanied by subtitles that closely adhere to the auditory input. While reading and listening to matched content may be a fairly common experience, we have little understanding of how comprehenders process the two sources of information, nor how the addition of audio changes word reading or might impact word learning. Eye-tracking provides a means of measuring the effort associated with processing words, yet very few studies have explicitly investigated written-word processing while listening and even fewer have examined this in the context of word learning. The technology allows researchers to synchronize eye-movements in reading to an auditory text, but requires technical know-how. The goal of this research methods paper is to provide methodological and technical guidance on the use of eye-tracking in reading-while-listening with an emphasis on investigating vocabulary learning and processing.