Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.03
Reza Khany, Peyman Valizadeh
This paper presents a comprehensive and rigorous systematic review of the trajectories and challenges in Research Articles (RAs) of Second Language Teacher Education (SLTE) from 1980 to 2020. The RAs were carefully retrieved from two sources: (1) a pool of 100 top-tier journals based on SCImago Journal Ranking (SJR) in Linguistics and Language category and (2) seven well-known electronic databases. This study used thematic analysis. After conducting full-text screening and data extraction, each study was systematically coded for publication contents, themes, theories, and variable levels. Each study was further coded for methodological features, including research design, analyses, and sampling. The findings indicate a rapid and consistent growth in SLTE research with a notable emphasis on Teacher Professional Development Programs (TPDP), research in teacher education, psychology in teacher education, culture, knowledge-based teacher education, and CALL. In addition, analysis of the data revealed several trajectories, including fewer publications in teacher education journals compared to other journals, lack of adherence to a clear theoretical framework in 27% of the studies, limited use of software in qualitative studies, and underdeveloped themes that received little attention. This review sheds light on the most significant and emerging areas of research in the field, and the article concludes with implications for SLTE scholars, researchers, and practitioners.
{"title":"The Scope of Second Language Teacher Education Research from 1980 to 2020: A Systematic Review","authors":"Reza Khany, Peyman Valizadeh","doi":"10.32038/ltrq.2023.38.03","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.03","url":null,"abstract":"This paper presents a comprehensive and rigorous systematic review of the trajectories and challenges in Research Articles (RAs) of Second Language Teacher Education (SLTE) from 1980 to 2020. The RAs were carefully retrieved from two sources: (1) a pool of 100 top-tier journals based on SCImago Journal Ranking (SJR) in Linguistics and Language category and (2) seven well-known electronic databases. This study used thematic analysis. After conducting full-text screening and data extraction, each study was systematically coded for publication contents, themes, theories, and variable levels. Each study was further coded for methodological features, including research design, analyses, and sampling. The findings indicate a rapid and consistent growth in SLTE research with a notable emphasis on Teacher Professional Development Programs (TPDP), research in teacher education, psychology in teacher education, culture, knowledge-based teacher education, and CALL. In addition, analysis of the data revealed several trajectories, including fewer publications in teacher education journals compared to other journals, lack of adherence to a clear theoretical framework in 27% of the studies, limited use of software in qualitative studies, and underdeveloped themes that received little attention. This review sheds light on the most significant and emerging areas of research in the field, and the article concludes with implications for SLTE scholars, researchers, and practitioners.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"369 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139020289","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.02
Esmaeel Ali Salimi, Seyed Mohammadreza Mortazavi
Pragmatic competence entails awareness-raising of impoliteness. This paper delves into the pragmatics of impoliteness in online communication, focusing on X (Twitter) interactions, with a particular emphasis on 126 replies to Elon Musk's controversial tweet. Utilizing Culpeper's (2011) model of impoliteness formulae and implicational impoliteness, the study analyzes responses to this tweet, exploring the multifaceted dimensions of impoliteness. The research also investigates the moral order expectations expressed by the repliers, shedding light on the factors influencing their reactions and responses. By employing qualitative content analysis and thematic analysis, the study reveals the nuanced forms of impoliteness and the underlying moral order themes inherent in these online exchanges. Responsible wealth behavior, appropriate and non-offensive language, respect and decency, refraining from political narratives, engaging in social responsibility, and showingtransparency, accountability, and ethics were the identified moral order expectations. The findings underscore the relevance of examining impoliteness in the digital age, where public figures’ tweets can have far-reaching consequences and provoke diverse reactions from the online community. This research contributes to a deeper understanding of impoliteness in online discourse and its implications for social media interactions by language learners, which are currently lacking in language classrooms.
{"title":"“I Keep Forgetting You’re Still Alive”: Unmasking Impoliteness in the Xsphere","authors":"Esmaeel Ali Salimi, Seyed Mohammadreza Mortazavi","doi":"10.32038/ltrq.2023.38.02","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.02","url":null,"abstract":"Pragmatic competence entails awareness-raising of impoliteness. This paper delves into the pragmatics of impoliteness in online communication, focusing on X (Twitter) interactions, with a particular emphasis on 126 replies to Elon Musk's controversial tweet. Utilizing Culpeper's (2011) model of impoliteness formulae and implicational impoliteness, the study analyzes responses to this tweet, exploring the multifaceted dimensions of impoliteness. The research also investigates the moral order expectations expressed by the repliers, shedding light on the factors influencing their reactions and responses. By employing qualitative content analysis and thematic analysis, the study reveals the nuanced forms of impoliteness and the underlying moral order themes inherent in these online exchanges. Responsible wealth behavior, appropriate and non-offensive language, respect and decency, refraining from political narratives, engaging in social responsibility, and showingtransparency, accountability, and ethics were the identified moral order expectations. The findings underscore the relevance of examining impoliteness in the digital age, where public figures’ tweets can have far-reaching consequences and provoke diverse reactions from the online community. This research contributes to a deeper understanding of impoliteness in online discourse and its implications for social media interactions by language learners, which are currently lacking in language classrooms.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"225 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139012900","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.04
Maryam Khaksar, G. Kiany, Parvaneh ShayesteFar
As teachers play a significant role in language learners’ academic achievement, their training programs should be subjected to ongoing evaluation and analysis in order to ascertain teachers are equipped with adequate knowledge and skills. This study aimed to evaluate a private sector pre-service teacher-training program using the Context, Input, Process and Product (CIPP) model. For such a purpose, the program policies, planned course of actions and real practices, as well as the outcomes were closely studied through a sequential mixed-method design. The required data were obtained from the program’s stakeholder layers (N=296: 100 supervisors, 58 trainers, 100 teachers, and 38 trainees) participating in data collection phases using multiple methods and instruments including interviews, observations and surveys. All instruments were subjected to detailed validation and reliability investigations. The results revealed degrees of positive perspectives towards the program’s functioning and outcomes in training reflective, creative and energetic teachers. Although the participants reported optimistic views about the program quality, required modifications and essential improvements were strongly stressed by the trainers in terms of the length of the course, focus on classroom management, and provision of practice opportunities. The findings have implications for evaluation of teacher training programs implemented in similar contexts elsewhere.
{"title":"Using a CIPP-Based Model for Evaluation of Teacher Training Programs in a Private-sector EFL Institutes","authors":"Maryam Khaksar, G. Kiany, Parvaneh ShayesteFar","doi":"10.32038/ltrq.2023.38.04","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.04","url":null,"abstract":"As teachers play a significant role in language learners’ academic achievement, their training programs should be subjected to ongoing evaluation and analysis in order to ascertain teachers are equipped with adequate knowledge and skills. This study aimed to evaluate a private sector pre-service teacher-training program using the Context, Input, Process and Product (CIPP) model. For such a purpose, the program policies, planned course of actions and real practices, as well as the outcomes were closely studied through a sequential mixed-method design. The required data were obtained from the program’s stakeholder layers (N=296: 100 supervisors, 58 trainers, 100 teachers, and 38 trainees) participating in data collection phases using multiple methods and instruments including interviews, observations and surveys. All instruments were subjected to detailed validation and reliability investigations. The results revealed degrees of positive perspectives towards the program’s functioning and outcomes in training reflective, creative and energetic teachers. Although the participants reported optimistic views about the program quality, required modifications and essential improvements were strongly stressed by the trainers in terms of the length of the course, focus on classroom management, and provision of practice opportunities. The findings have implications for evaluation of teacher training programs implemented in similar contexts elsewhere.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"7 6","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139023782","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.06
Mansoor S. Almalki
This study contributes to the Global Southern epistemological debates on the professional identity negotiations of Global-North-educated English as a Foreign Language (EFL) instructors. Using semi-structured interviews, the study analyses how two Saudi Arabian EFL instructors, during their PhD studies in the United States, and upon their return home, coped with the phenomenon of native-speakerism and navigated their way through the binary of Native English-Speaking Teachers (NEST) and Non-Native English-Speaking Teachers (NNEST) in their teaching. They appreciated their Western education not due to the traditional privileged-Global-North-and-underprivileged-Global-South binary but because of the many ways in which it helped them negotiate their translingual identity. They saw themselves as better placed to give constructive feedback to learners but critiqued the NESTs’ inability to base themselves in the local culture and positioned themselves above NESTs due to their knowledge of indigenous Saudi culture. Thus, they dismantled the traditional privilege associated with native-speakerism but did not hail one category over the other. Instead, they picked from both categories the materials, ways, means and attitudes that best served their purpose. They strove for hybridity. Through their negotiation of their foreign education and local challenges, they developed a unique translingual identity.
{"title":"The NEST/NNEST Binary and Translingual Identity of U.S.-Educated EFL Instructors in Saudi Arabia: A Study on Linguistic and Cultural Navigation in the Classroom","authors":"Mansoor S. Almalki","doi":"10.32038/ltrq.2023.38.06","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.06","url":null,"abstract":"This study contributes to the Global Southern epistemological debates on the professional identity negotiations of Global-North-educated English as a Foreign Language (EFL) instructors. Using semi-structured interviews, the study analyses how two Saudi Arabian EFL instructors, during their PhD studies in the United States, and upon their return home, coped with the phenomenon of native-speakerism and navigated their way through the binary of Native English-Speaking Teachers (NEST) and Non-Native English-Speaking Teachers (NNEST) in their teaching. They appreciated their Western education not due to the traditional privileged-Global-North-and-underprivileged-Global-South binary but because of the many ways in which it helped them negotiate their translingual identity. They saw themselves as better placed to give constructive feedback to learners but critiqued the NESTs’ inability to base themselves in the local culture and positioned themselves above NESTs due to their knowledge of indigenous Saudi culture. Thus, they dismantled the traditional privilege associated with native-speakerism but did not hail one category over the other. Instead, they picked from both categories the materials, ways, means and attitudes that best served their purpose. They strove for hybridity. Through their negotiation of their foreign education and local challenges, they developed a unique translingual identity.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"285 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139021749","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.01
Fernando Peralta-Castro
The scope of this study is to observe features of environmental autonomy support and to identify aspects of autonomy. To this end, a case study was designed and implemented in a state school in the western region of Mexico, with the participation of twelve secondary school pupils, ages 12 to 14, who attended a non-required 40-hour course using the autonomous learning approach. A research journal was kept tracking the whole process, gaining knowledge of autonomous learning and examining the phenomenon in a real-life context. The journals were analyzed with the help of Atlas.ti software, and the researcher made inferences from the resulting data, including environmental autonomy support as well as the reactions of the pupils to the autonomous learning process. The analysis of data reveals the language-learning context and its tendency to foster an autonomy-supportive atmosphere. Data seems to cast light on two major issues: the importance of fostering an environment that encourages autonomous learning-related behavior, and the necessity of providing appropriate scaffolding strategies to assist students in their development of autonomous learning. The study provides valuable insights into autonomy support and autonomous learning in education, but its limitations include a small sample size and potential challenges in generalizing findings to broader contexts.
{"title":"Fostering Environmental Autonomy Support in an English Course for Middle School Learners in Western México","authors":"Fernando Peralta-Castro","doi":"10.32038/ltrq.2023.38.01","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.01","url":null,"abstract":"The scope of this study is to observe features of environmental autonomy support and to identify aspects of autonomy. To this end, a case study was designed and implemented in a state school in the western region of Mexico, with the participation of twelve secondary school pupils, ages 12 to 14, who attended a non-required 40-hour course using the autonomous learning approach. A research journal was kept tracking the whole process, gaining knowledge of autonomous learning and examining the phenomenon in a real-life context. The journals were analyzed with the help of Atlas.ti software, and the researcher made inferences from the resulting data, including environmental autonomy support as well as the reactions of the pupils to the autonomous learning process. The analysis of data reveals the language-learning context and its tendency to foster an autonomy-supportive atmosphere. Data seems to cast light on two major issues: the importance of fostering an environment that encourages autonomous learning-related behavior, and the necessity of providing appropriate scaffolding strategies to assist students in their development of autonomous learning. The study provides valuable insights into autonomy support and autonomous learning in education, but its limitations include a small sample size and potential challenges in generalizing findings to broader contexts.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"198 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139014074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-01DOI: 10.32038/ltrq.2023.38.05
Mahmuda Sharmin
Over the years, there has been a rising interest in combining debriefing teaching approaches and multimodality in classrooms to promote superior critical thinking, reflective thinking, and comprehension (Reyes-Chua, 2018). Such practices have also been instrumental in facilitating L2 learning and identity development (Johnson & Kendrick, 2017). Although most research has focused on the potential of multimodality in enhancing language learning and identity construction, only a handful of studies have investigated how debriefing teaching strategies and multimodal narrative tactics can empower adult minoritized L2 learners to confront racism and develop agency. Using ethnographic and action-based research data, this study examines the role of debriefing teaching strategies and multimodal narrative practices in developing agency, creating meaning, and addressing linguistic racism. The study took place in an intermediate ESL class in the Mid-South region of the USA. Over a period of ten weeks, students wrote ten multimodal narratives in shared Google Docs about their experiences with the English language. During debriefing sessions, they reflected on the narratives and addressed issues of racism by answering 'what,' 'so what,' and 'now what' questions. This study finds that learners can acquire language, develop agency, and confront linguicism through multimodal narrative practices and debriefing teaching strategies.
{"title":"Debriefing Teaching Strategies and Multimodal Narratives in ESL: Pedagogical Tools for Developing Agency, Making Meaning, and Confronting Racism","authors":"Mahmuda Sharmin","doi":"10.32038/ltrq.2023.38.05","DOIUrl":"https://doi.org/10.32038/ltrq.2023.38.05","url":null,"abstract":"Over the years, there has been a rising interest in combining debriefing teaching approaches and multimodality in classrooms to promote superior critical thinking, reflective thinking, and comprehension (Reyes-Chua, 2018). Such practices have also been instrumental in facilitating L2 learning and identity development (Johnson & Kendrick, 2017). Although most research has focused on the potential of multimodality in enhancing language learning and identity construction, only a handful of studies have investigated how debriefing teaching strategies and multimodal narrative tactics can empower adult minoritized L2 learners to confront racism and develop agency. Using ethnographic and action-based research data, this study examines the role of debriefing teaching strategies and multimodal narrative practices in developing agency, creating meaning, and addressing linguistic racism. The study took place in an intermediate ESL class in the Mid-South region of the USA. Over a period of ten weeks, students wrote ten multimodal narratives in shared Google Docs about their experiences with the English language. During debriefing sessions, they reflected on the narratives and addressed issues of racism by answering 'what,' 'so what,' and 'now what' questions. This study finds that learners can acquire language, develop agency, and confront linguicism through multimodal narrative practices and debriefing teaching strategies.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"235 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139019890","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-01DOI: 10.32038/ltrq.2023.37.05
Marita Härmälä
The study explores Finnish compulsory school language teachers’ formative and summative assessment practices. It aims at finding out what formative and summative assessment modes and tools the teachers use and what kind of evidence they collect for students’ final grades. The data were gathered through questionnaires administered in national evaluations of learning outcomes. The results indicate that teachers use multiple tools, i.e., scores, information on a task’s objective, oral feedback, and model performances. They listen to students paired discussions and talk with students about their progress. Students also assess their own skills. Teachers’ feedback on exams was information on a task’s objective, oral feedback, and model performances. When deciding on final grades, teachers use as evidence written tests and students’ working in lessons. However, there were some differences between languages and syllabi. Teachers gave the final grades based on students’ language proficiency at a given moment stressing the curricula’s content areas related to language proficiency. At the same time, teachers indicated to be basing their grades on averages in all the different content areas. The study has implications for teachers’ assessment literary training and for exploring more in detail the factors impacting students’ final grades.
{"title":"Finnish Compulsory School English and Swedish Teachers’ Classroom Assessment Practices","authors":"Marita Härmälä","doi":"10.32038/ltrq.2023.37.05","DOIUrl":"https://doi.org/10.32038/ltrq.2023.37.05","url":null,"abstract":"The study explores Finnish compulsory school language teachers’ formative and summative assessment practices. It aims at finding out what formative and summative assessment modes and tools the teachers use and what kind of evidence they collect for students’ final grades. The data were gathered through questionnaires administered in national evaluations of learning outcomes. The results indicate that teachers use multiple tools, i.e., scores, information on a task’s objective, oral feedback, and model performances. They listen to students paired discussions and talk with students about their progress. Students also assess their own skills. Teachers’ feedback on exams was information on a task’s objective, oral feedback, and model performances. When deciding on final grades, teachers use as evidence written tests and students’ working in lessons. However, there were some differences between languages and syllabi. Teachers gave the final grades based on students’ language proficiency at a given moment stressing the curricula’s content areas related to language proficiency. At the same time, teachers indicated to be basing their grades on averages in all the different content areas. The study has implications for teachers’ assessment literary training and for exploring more in detail the factors impacting students’ final grades.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135516056","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-01DOI: 10.32038/ltrq.2023.37.01
Christine Coombe, Tony Clark, Hassan Mohebbi
It is not every day, year, or even decade that a journal dedicates an entire issue to honour the contributions of an individual scholar in the field. The purpose of this Special Issue of Language Teaching Research Quarterly is to celebrate the work of James Dean Brown, a scholar who has made an enormous impact on the field of language testing and assessment. Following his retirement from the field, this Special Issue was created in JD’s honour and is dedicated to him and his many contributions to the field.
{"title":"Volume Introduction: Special Issue in Honour of James Dean Brown’s Five-Decade Contribution to Language Testing and Assessment","authors":"Christine Coombe, Tony Clark, Hassan Mohebbi","doi":"10.32038/ltrq.2023.37.01","DOIUrl":"https://doi.org/10.32038/ltrq.2023.37.01","url":null,"abstract":"It is not every day, year, or even decade that a journal dedicates an entire issue to honour the contributions of an individual scholar in the field. The purpose of this Special Issue of Language Teaching Research Quarterly is to celebrate the work of James Dean Brown, a scholar who has made an enormous impact on the field of language testing and assessment. Following his retirement from the field, this Special Issue was created in JD’s honour and is dedicated to him and his many contributions to the field.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135516060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-01DOI: 10.32038/ltrq.2023.37.12
Kevin Hirschi, Okim Kang
This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on G studies–a method of breaking down measurement variance–and D studies–a predictive study of the impact on reliability when modifying the number of raters, items, or other facets that assist the field in adopting measurement designs that include comprehensibility, accentedness, and intelligibility. When data from a single audio sample per learner were subjected to D-studies, we find that both semantic differential and rubric scales for comprehensibility were reliable at the .90 level with about 15 trained raters or 50 untrained crowdsourced raters. In order to offer generalizable and dependable evaluations, empirically informed recommendations are given, including considerations for the number of speech samples rated, or the granularity of the scales for various assessment and research purposes.
{"title":"How Many Raters Can Be Enough: G Theory Applied to Assessment and Measurement of L2 Speech Perception","authors":"Kevin Hirschi, Okim Kang","doi":"10.32038/ltrq.2023.37.12","DOIUrl":"https://doi.org/10.32038/ltrq.2023.37.12","url":null,"abstract":"This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on G studies–a method of breaking down measurement variance–and D studies–a predictive study of the impact on reliability when modifying the number of raters, items, or other facets that assist the field in adopting measurement designs that include comprehensibility, accentedness, and intelligibility. When data from a single audio sample per learner were subjected to D-studies, we find that both semantic differential and rubric scales for comprehensibility were reliable at the .90 level with about 15 trained raters or 50 untrained crowdsourced raters. In order to offer generalizable and dependable evaluations, empirically informed recommendations are given, including considerations for the number of speech samples rated, or the granularity of the scales for various assessment and research purposes.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"127 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135566319","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-01DOI: 10.32038/ltrq.2023.37.08
Jonathan Trace
The role of context in cloze tests has long been seen as both a benefit as well as a complication in their usefulness as a measure of second language comprehension (Brown, 2013). Passage cohesion, in particular, would seem to have a relevant and important effect on the degree to which cloze items function and the interpretability of performances (Brown, 1983; Dastjerdi & Talebinezhad, 2006; Oller & Jonz, 1994). With recent evidence showing that cloze items can require examinees to access information at both the sentence and passage level (Trace, 2020), it’s worthwhile to now look back and examine the relationship between aspects of passage cohesion—referential cohesion, semantic overlap, and incidence of conjunctives—and item difficulty by classification. The current study draws upon a large pool of cloze test passages and items (k = 377) originally used by Brown (1993) along with automated text analysis of cohesion (Coh-Metrix, McNamara et al., 2014) to examine the impact of passage cohesion on item function. Correlations, factor analysis, and linear regression point to clear though minimal differences for both sentential and intersentential items as they relate to aspects of passage cohesion, the results of which may inform future test design and interpretation of cloze performance.
{"title":"The Influence of Passage Cohesion on Cloze Test Item Difficulty","authors":"Jonathan Trace","doi":"10.32038/ltrq.2023.37.08","DOIUrl":"https://doi.org/10.32038/ltrq.2023.37.08","url":null,"abstract":"The role of context in cloze tests has long been seen as both a benefit as well as a complication in their usefulness as a measure of second language comprehension (Brown, 2013). Passage cohesion, in particular, would seem to have a relevant and important effect on the degree to which cloze items function and the interpretability of performances (Brown, 1983; Dastjerdi & Talebinezhad, 2006; Oller & Jonz, 1994). With recent evidence showing that cloze items can require examinees to access information at both the sentence and passage level (Trace, 2020), it’s worthwhile to now look back and examine the relationship between aspects of passage cohesion—referential cohesion, semantic overlap, and incidence of conjunctives—and item difficulty by classification. The current study draws upon a large pool of cloze test passages and items (k = 377) originally used by Brown (1993) along with automated text analysis of cohesion (Coh-Metrix, McNamara et al., 2014) to examine the impact of passage cohesion on item function. Correlations, factor analysis, and linear regression point to clear though minimal differences for both sentential and intersentential items as they relate to aspects of passage cohesion, the results of which may inform future test design and interpretation of cloze performance.","PeriodicalId":350461,"journal":{"name":"Language Teaching Research Quarterly","volume":"41 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135514844","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：