The importance of viewing tests in their policy contexts is now widely recognized in our field. The Language Testing Research Centre (LTRC), as a self-funding Centre, responds by necessity to policy shifts and the language testing initiatives it has undertaken over the years offer insights into broader societal and more local institutional imperatives. The Centre’s language testing activities also have the potential to influence policy, whether directly or indirectly. This paper offers a historical overview of the Centre’s various projects in the languages (other than English) arena over a 30-year period, describing the diverse orientations adopted (i.e. to inform, enact or evaluate policy) with particular reference to three cases illustrative of different policy trends. I speculate about the policy impact of each case, highlighting the complexities encountered and some of the factors favouring and inhibiting policy uptake. Insights from these particular cases are then linked to the research literature on policy impact to propose recommendations for ‘policy responsible‘ language testing.
{"title":"Towards ‘policy responsible’ language testing: Framing the Language Testing Research Centre’s contribution in languages other than English","authors":"C. Elder","doi":"10.58379/hqym2624","DOIUrl":"https://doi.org/10.58379/hqym2624","url":null,"abstract":"The importance of viewing tests in their policy contexts is now widely recognized in our field. The Language Testing Research Centre (LTRC), as a self-funding Centre, responds by necessity to policy shifts and the language testing initiatives it has undertaken over the years offer insights into broader societal and more local institutional imperatives. The Centre’s language testing activities also have the potential to influence policy, whether directly or indirectly. This paper offers a historical overview of the Centre’s various projects in the languages (other than English) arena over a 30-year period, describing the diverse orientations adopted (i.e. to inform, enact or evaluate policy) with particular reference to three cases illustrative of different policy trends. I speculate about the policy impact of each case, highlighting the complexities encountered and some of the factors favouring and inhibiting policy uptake. Insights from these particular cases are then linked to the research literature on policy impact to propose recommendations for ‘policy responsible‘ language testing.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87926752","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Generations: Research and Application at the LTRC","authors":"J. L. Bianco","doi":"10.58379/ciez8454","DOIUrl":"https://doi.org/10.58379/ciez8454","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"2 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84883347","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
All communication entails moments of breakdown and repair. Research has shown that test-takers in oral proficiency interviews use both verbal and nonverbal behaviors to navigate these breakdowns, which may occur more often in online speaking tests than face-to-face settings. While individuals are often idiosyncratic in their baseline nonverbal behavior, it is unknown to what extent speakers resemble or differ from each other during moments of repair. To date, a research agenda including a broader range of embodied behavior in the context of language tests is lacking. In this study, four Zoom-based video recordings of second language tests were used to investigate seven categories of behavior and their alignment with speech during moments of other-initiated repair., This contribution uses multimodal conversation analysis to analyze differences in behavior amongst the participants and how those patterns align with the turn-by-turn nature of repair. The analysis shows that participants varied substantially in both the frequency and duration of each type of behavior during repair, and furthermore differed in how they combined these during their interaction with the examiner. The study highlights the need for more research in how individuals demonstrate their interactional competence both verbally and nonverbally for greater construct representation in second language speaking tests.
{"title":"The face of communication breakdown: Multimodal repair in L2 oral proficiency interviews","authors":"J. Burton","doi":"10.58379/jlnv8752","DOIUrl":"https://doi.org/10.58379/jlnv8752","url":null,"abstract":"All communication entails moments of breakdown and repair. Research has shown that test-takers in oral proficiency interviews use both verbal and nonverbal behaviors to navigate these breakdowns, which may occur more often in online speaking tests than face-to-face settings. While individuals are often idiosyncratic in their baseline nonverbal behavior, it is unknown to what extent speakers resemble or differ from each other during moments of repair. To date, a research agenda including a broader range of embodied behavior in the context of language tests is lacking. In this study, four Zoom-based video recordings of second language tests were used to investigate seven categories of behavior and their alignment with speech during moments of other-initiated repair., This contribution uses multimodal conversation analysis to analyze differences in behavior amongst the participants and how those patterns align with the turn-by-turn nature of repair. The analysis shows that participants varied substantially in both the frequency and duration of each type of behavior during repair, and furthermore differed in how they combined these during their interaction with the examiner. The study highlights the need for more research in how individuals demonstrate their interactional competence both verbally and nonverbally for greater construct representation in second language speaking tests.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"30 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73452743","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Salaberry, M.R. & Burch, A.R. (Eds.) Assessing Speaking in Context: Expanding the Construct and its Applications","authors":"Ying Chen, Qi Lu","doi":"10.58379/ppnw1264","DOIUrl":"https://doi.org/10.58379/ppnw1264","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"17 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90677456","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Elicited imitation (EI) assessments have been shown to discriminate well between speakers across proficiency levels, but little has been reported on the effect L2 accent has on test-takers’ ability to understand and process the test items they hear. Furthermore, no study has investigated the effect of accent on test-taker perceptions of EI tests. This study examined the relationships among accent, accent familiarity, EI test item difficulty and test scores. To investigate, self-reports of students’ exposure to different varieties of English were obtained from a pre-assessment survey. An EI test (63 items) was then administered in which English language learners (n = 213) in the United States listened to test items in three varieties of English: American English, Australian English, and British English. A Rasch analysis found that the test had high reliability (person separation = .94), with intended item level and accent both having a significant effect on test item difficulty. Survey results indicated a moderate relationship between an examinee’s familiarity with a particular accent and their person ability estimate measures. These findings suggest that prompt accent should be considered in EI test development.
{"title":"The effect of prompt accent on elicited imitation assessments in English as a second language","authors":"J. Barrows, Troy L. Cox","doi":"10.58379/deqm7013","DOIUrl":"https://doi.org/10.58379/deqm7013","url":null,"abstract":"Elicited imitation (EI) assessments have been shown to discriminate well between speakers across proficiency levels, but little has been reported on the effect L2 accent has on test-takers’ ability to understand and process the test items they hear. Furthermore, no study has investigated the effect of accent on test-taker perceptions of EI tests. This study examined the relationships among accent, accent familiarity, EI test item difficulty and test scores. To investigate, self-reports of students’ exposure to different varieties of English were obtained from a pre-assessment survey. An EI test (63 items) was then administered in which English language learners (n = 213) in the United States listened to test items in three varieties of English: American English, Australian English, and British English. A Rasch analysis found that the test had high reliability (person separation = .94), with intended item level and accent both having a significant effect on test item difficulty. Survey results indicated a moderate relationship between an examinee’s familiarity with a particular accent and their person ability estimate measures. These findings suggest that prompt accent should be considered in EI test development.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"08 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86023252","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Any form of talk-in-interaction is organized in relation to progressivity (Schegloff, 2007; Hosoda & Aline, 2013). Progressivity of interaction may involve resolving interactional troubles and producing subsequent talk, and it is endemic to the organisation of conversation at the level of turn construction (Schegloff, 1979). In testing speaking skills, as progressivity of talk between peers is central for teachers to be able to assess students’ performances, troubles that halt progressivity and the resolution of such troubles by the students deserve close analyses. Against this background, this paper focuses on how paired L2 (i.e., English) speaking assessment interactions unfold in a Turkish higher education context. Using multimodal conversation analysis, we investigated the interactional resources that are deployed to maintain progressivity when there is a halt in the unfolding interaction, an under-researched phenomenon in L2 speaking assessment contexts. We used transcriptions of 100 video-recorded paired test interactions, each of which were approximately four minutes long. We identified and described the ways interactional trouble was flagged, paying close attention to how embodied resources, such as hand and other gestures, gaze direction, posture and body orientation, and facial expressions (Nevile, 2015), are used by the interactants. Based on 100 paired assessment interactions our findings reveal that in moments of interactional trouble participants make transitions to a sub-topic, formulate understandings, and engage in collaborative completions to maintain progressivity, using a variety of interactional resources as part of their interactional repertoires. The ways such troubles unfold in interaction and how they are resolved by L2 users to maintain progressivity of test-talk have potential to inform research on the assessment of Interactional Competence.
{"title":"Resolving interactional troubles and maintaining progressivity in paired speaking assessment in an EFL context","authors":"Merve Çoban, Olcay Sert","doi":"10.58379/twos3837","DOIUrl":"https://doi.org/10.58379/twos3837","url":null,"abstract":"Any form of talk-in-interaction is organized in relation to progressivity (Schegloff, 2007; Hosoda & Aline, 2013). Progressivity of interaction may involve resolving interactional troubles and producing subsequent talk, and it is endemic to the organisation of conversation at the level of turn construction (Schegloff, 1979). In testing speaking skills, as progressivity of talk between peers is central for teachers to be able to assess students’ performances, troubles that halt progressivity and the resolution of such troubles by the students deserve close analyses. Against this background, this paper focuses on how paired L2 (i.e., English) speaking assessment interactions unfold in a Turkish higher education context. Using multimodal conversation analysis, we investigated the interactional resources that are deployed to maintain progressivity when there is a halt in the unfolding interaction, an under-researched phenomenon in L2 speaking assessment contexts. We used transcriptions of 100 video-recorded paired test interactions, each of which were approximately four minutes long. We identified and described the ways interactional trouble was flagged, paying close attention to how embodied resources, such as hand and other gestures, gaze direction, posture and body orientation, and facial expressions (Nevile, 2015), are used by the interactants. Based on 100 paired assessment interactions our findings reveal that in moments of interactional trouble participants make transitions to a sub-topic, formulate understandings, and engage in collaborative completions to maintain progressivity, using a variety of interactional resources as part of their interactional repertoires. The ways such troubles unfold in interaction and how they are resolved by L2 users to maintain progressivity of test-talk have potential to inform research on the assessment of Interactional Competence.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"17 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"82792000","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Since the turn of the century, the field of language assessment has increasingly turned its eye towards the assessment of Interactional Competence (IC) (Galaczi & Taylor, 2018; Plough et al., 2018; Roever & Kasper, 2018). This study is premised on the argument that a key function of IC is to achieve and maintain intersubjectivity, as made publicly viewable through the practices that participants employ to display that they understand each other, and how they understand each other. The paper thus suggests that IC assessment can and should consider intersubjectivity as a ratable construct. Using Multimodal Conversation Analysis (Goodwin, 2018; Mondada, 2011), the paper examines two paired-speaking assessment tasks conducted by learners of German at the end of their fourth semester of study, focusing on how and when the learners display understanding of prior talk, examining how they receipt turns and display epistemic and affective stances in a publicly viewable way. The study suggests that the same practices the participants employ to display their understandings to each other can be used in a heuristic fashion by language testers to assess IC and concludes by considering the practical implications these have for classroom assessment.
{"title":"Assessing Interactional Competence: The role of intersubjectivity in a paired-speaking assessment task","authors":"A. R. Burch, Katharina Kley","doi":"10.58379/vkyg7791","DOIUrl":"https://doi.org/10.58379/vkyg7791","url":null,"abstract":"Since the turn of the century, the field of language assessment has increasingly turned its eye towards the assessment of Interactional Competence (IC) (Galaczi & Taylor, 2018; Plough et al., 2018; Roever & Kasper, 2018). This study is premised on the argument that a key function of IC is to achieve and maintain intersubjectivity, as made publicly viewable through the practices that participants employ to display that they understand each other, and how they understand each other. The paper thus suggests that IC assessment can and should consider intersubjectivity as a ratable construct. Using Multimodal Conversation Analysis (Goodwin, 2018; Mondada, 2011), the paper examines two paired-speaking assessment tasks conducted by learners of German at the end of their fourth semester of study, focusing on how and when the learners display understanding of prior talk, examining how they receipt turns and display epistemic and affective stances in a publicly viewable way. The study suggests that the same practices the participants employ to display their understandings to each other can be used in a heuristic fashion by language testers to assess IC and concludes by considering the practical implications these have for classroom assessment.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"47 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88092284","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"L.F. Bachman & B. Damböck. Language Assessment for Classroom Teachers","authors":"Lan Luo","doi":"10.58379/jlyv5956","DOIUrl":"https://doi.org/10.58379/jlyv5956","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"12 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88486463","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
While paired student discussion tests in EFL contexts are often graded using rubrics with broad descriptors, an alternative approach constructs the rubric via extensive written descriptions of video-recorded exemplary cases at each performance level. With its long history of deeply descriptive observation of interaction, Conversation Analysis (CA) is one apt tool for constructing such exemplar-based rubrics; but to what extent are non-CA specialist teacher-raters able to interpret a CA analysis in order to assess the test? This study explores this issue by comparing two paired EFL discussion tests that use exemplar-based rubrics, one written by a CA specialist and the other by EFL test constructors not specialized in CA. The complete dataset consists of test recordings (university-level Japanese learners of English, and secondary-level Swedish learners of English) and recordings of teacher-raters’ interaction. Our analysis focuses on ways experienced language educators perceive engagement while discussing their ratings of the video-recorded test talk in relation to the exemplars and descriptive rubrics. The study highlights differences in the way teacher-raters display their understanding of the notion of engagement within the tests, and demonstrates how CA rubrics can facilitate a more emically grounded assessment.
{"title":"How do raters understand rubrics for assessing L2 interactional engagement? A comparative study of CA- and non-CA-formulated performance descriptors","authors":"Erica Sandlund, T. Greer","doi":"10.58379/jciw3943","DOIUrl":"https://doi.org/10.58379/jciw3943","url":null,"abstract":"While paired student discussion tests in EFL contexts are often graded using rubrics with broad descriptors, an alternative approach constructs the rubric via extensive written descriptions of video-recorded exemplary cases at each performance level. With its long history of deeply descriptive observation of interaction, Conversation Analysis (CA) is one apt tool for constructing such exemplar-based rubrics; but to what extent are non-CA specialist teacher-raters able to interpret a CA analysis in order to assess the test? This study explores this issue by comparing two paired EFL discussion tests that use exemplar-based rubrics, one written by a CA specialist and the other by EFL test constructors not specialized in CA. The complete dataset consists of test recordings (university-level Japanese learners of English, and secondary-level Swedish learners of English) and recordings of teacher-raters’ interaction. Our analysis focuses on ways experienced language educators perceive engagement while discussing their ratings of the video-recorded test talk in relation to the exemplars and descriptive rubrics. The study highlights differences in the way teacher-raters display their understanding of the notion of engagement within the tests, and demonstrates how CA rubrics can facilitate a more emically grounded assessment.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"39 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84091941","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessment task design influences the elicitation of targeted linguistic and communicative demands by creating the conditions under which the task is performed. Through the task design, plausible language use contexts are also created. In this study, I examined the role of pragmatic variables in role-play design to ensure the context validity of assessment of interactional competence. Using conversation analysis (CA), I qualitatively analyzed high-level L2 English learners’ performances elicited from five role-plays that measure pragmatic abilities in interaction. I examined how the learners oriented to the pragmatic variables embedded in the role-plays. The elicited role-play performances shared action sequences (e.g., opening) and generic interactional features. However, the learners utilized sequential organizations and grammatical resources differently to accomplish context-specific actions. For example, the length of turns tended to be longer and sequential organizations were more extended when the learners were engaged in the role-plays of formal pragmatic functions with an interlocutor with more social distance. The learners utilized diverse grammatical formats in a non-uniform manner specific to sequential positions of pragmatic actions. I discuss the importance of assessment task design for context validity and eliciting evidence of interactional competence. Further, I argue how CA’s analytical attention to real-time details of interaction captures the evidence of interactional competence.
{"title":"Pragmatic variables in role-play design for the context validity of assessing interactional competence","authors":"S. Youn","doi":"10.58379/czpw6475","DOIUrl":"https://doi.org/10.58379/czpw6475","url":null,"abstract":"Assessment task design influences the elicitation of targeted linguistic and communicative demands by creating the conditions under which the task is performed. Through the task design, plausible language use contexts are also created. In this study, I examined the role of pragmatic variables in role-play design to ensure the context validity of assessment of interactional competence. Using conversation analysis (CA), I qualitatively analyzed high-level L2 English learners’ performances elicited from five role-plays that measure pragmatic abilities in interaction. I examined how the learners oriented to the pragmatic variables embedded in the role-plays. The elicited role-play performances shared action sequences (e.g., opening) and generic interactional features. However, the learners utilized sequential organizations and grammatical resources differently to accomplish context-specific actions. For example, the length of turns tended to be longer and sequential organizations were more extended when the learners were engaged in the role-plays of formal pragmatic functions with an interlocutor with more social distance. The learners utilized diverse grammatical formats in a non-uniform manner specific to sequential positions of pragmatic actions. I discuss the importance of assessment task design for context validity and eliciting evidence of interactional competence. Further, I argue how CA’s analytical attention to real-time details of interaction captures the evidence of interactional competence.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"61 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"78357261","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：