This study investigates assessment within oral academic assessment tasks, specifically focusing on the criteria used by discipline specialists and comparing them with those used to assess performance on TOEFL iBTTM speaking tasks. Three pairs of tutors from three faculties took part in verbal report sessions where they watched, rated and discussed the performances of ten native and ten non-native students completing first-year university oral assessment tasks in their discipline. The verbal report sessions were audio recorded, transcribed and segmented into meaning-based units prior to thematic analysis. The features which emerged from the analysis were compared with those described within the TOEFL speaking rubrics. The analysis found that while there were some broad similarities in the focus there were also marked differences. Two of the three TOEFL strands (delivery and content) were well-represented in the academic tasks assessments rubrics and tutor discussion. However, the quality of the non-native students’ language was only of concern when it was perceived as interfering with the student’s ability to communicate. An additional focus in the assessment of university tasks was the use of academic skills, prompts and aids, non-verbal communication and engagement with the audience.
{"title":"Contextualised judgements: A comparison of the rating criteria used to judge oral presentations in higher education and speaking performances in the TOEFL iBT™","authors":"A. Ducasse, Annie Brown","doi":"10.58379/jwid1797","DOIUrl":"https://doi.org/10.58379/jwid1797","url":null,"abstract":"This study investigates assessment within oral academic assessment tasks, specifically focusing on the criteria used by discipline specialists and comparing them with those used to assess performance on TOEFL iBTTM speaking tasks. Three pairs of tutors from three faculties took part in verbal report sessions where they watched, rated and discussed the performances of ten native and ten non-native students completing first-year university oral assessment tasks in their discipline. The verbal report sessions were audio recorded, transcribed and segmented into meaning-based units prior to thematic analysis. The features which emerged from the analysis were compared with those described within the TOEFL speaking rubrics. The analysis found that while there were some broad similarities in the focus there were also marked differences. Two of the three TOEFL strands (delivery and content) were well-represented in the academic tasks assessments rubrics and tutor discussion. However, the quality of the non-native students’ language was only of concern when it was perceived as interfering with the student’s ability to communicate. An additional focus in the assessment of university tasks was the use of academic skills, prompts and aids, non-verbal communication and engagement with the audience.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"36 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81410553","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Scholarship in language testing is increasingly embracing sociologically defined notions of language. This includes the notion of interactional competence (IC) which encompasses the various sequential, temporal, and embodied resources language learners utilize when interacting in the L2. This paper makes a contribution to scholarship that seeks to connect terminological and conceptual aspects of interaction research and research in language testing. To that end, this paper focuses on what is regular and structural about human talk-in-interaction, what it is that differentiates human interaction from prescriptively normative notions language professionals often apply to sentence-level lexis and grammar, and why including notions of interactional competencies is desirable in language testing. If language testing is to systematically target actual interactional abilities of L2 learners, then it is useful to examine what it is that makes interaction highly structured yet also dynamic to establish a conceptual basis for testing goals and practices.
{"title":"Testing interactional competence: Patterned yet dynamic aspects of L2 interaction","authors":"Thorsten Huth","doi":"10.58379/cotx7322","DOIUrl":"https://doi.org/10.58379/cotx7322","url":null,"abstract":"Scholarship in language testing is increasingly embracing sociologically defined notions of language. This includes the notion of interactional competence (IC) which encompasses the various sequential, temporal, and embodied resources language learners utilize when interacting in the L2. This paper makes a contribution to scholarship that seeks to connect terminological and conceptual aspects of interaction research and research in language testing. To that end, this paper focuses on what is regular and structural about human talk-in-interaction, what it is that differentiates human interaction from prescriptively normative notions language professionals often apply to sentence-level lexis and grammar, and why including notions of interactional competencies is desirable in language testing. If language testing is to systematically target actual interactional abilities of L2 learners, then it is useful to examine what it is that makes interaction highly structured yet also dynamic to establish a conceptual basis for testing goals and practices.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"67 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84008434","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"P. Seedhouse & F. Nakatsuhara. The Discourse of the IELTS Speaking Test: Interactional design and practice","authors":"Shih-Chang Chen","doi":"10.58379/liyg6723","DOIUrl":"https://doi.org/10.58379/liyg6723","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"93 4 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87660772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"N. Jones & N. Saville. Learning Oriented Assessment: A Systemic Approach (Vol. 45)","authors":"Ruijin Yang","doi":"10.58379/qcir4834","DOIUrl":"https://doi.org/10.58379/qcir4834","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"269 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"79855354","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Where Conversation Analysis meets Language Assessment: Toward expanding epistemologies and validity evidence","authors":"S. Youn, A. R. Burch","doi":"10.58379/dyki2461","DOIUrl":"https://doi.org/10.58379/dyki2461","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"83 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89110531","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Spell checkers are popular among writers but are seldom used or studied in second language (L2) writing assessment contexts. Little is known about how L2 test takers use spell checkers or how their responses are impacted by spell checker use. Making a first step towards addressing this knowledge gap was the goal of this study. We aimed to gain a preliminary understanding of L2 test takers’ spell checker use and its impact with a total of 61 adult English learners who responded to two computer-delivered English writing tasks. Half were randomly selected to be provided with a built-in spell checker. The resulting responses and keystroke logs from the test takers were analyzed to examine their spell checker use and its impact on the product and process of their writing. The test takers who chose to use the spell checker were highly comparable to those who did not use it in terms of their general English proficiency. A series of regression analyses indicated that the spell checker users, on average, wrote fewer words and made fewer spelling errors than those who did not use it. The spell checker users also tended to write at a slower pace for one of the two tasks.
{"title":"A preliminary look at the impact of spell checker use during an L2 English writing assessment","authors":"Ikkyu Choi, Yeonsuk Cho","doi":"10.58379/seqz3214","DOIUrl":"https://doi.org/10.58379/seqz3214","url":null,"abstract":"Spell checkers are popular among writers but are seldom used or studied in second language (L2) writing assessment contexts. Little is known about how L2 test takers use spell checkers or how their responses are impacted by spell checker use. Making a first step towards addressing this knowledge gap was the goal of this study. We aimed to gain a preliminary understanding of L2 test takers’ spell checker use and its impact with a total of 61 adult English learners who responded to two computer-delivered English writing tasks. Half were randomly selected to be provided with a built-in spell checker. The resulting responses and keystroke logs from the test takers were analyzed to examine their spell checker use and its impact on the product and process of their writing. The test takers who chose to use the spell checker were highly comparable to those who did not use it in terms of their general English proficiency. A series of regression analyses indicated that the spell checker users, on average, wrote fewer words and made fewer spelling errors than those who did not use it. The spell checker users also tended to write at a slower pace for one of the two tasks.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"60 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87929517","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The current paper presents a case study where an attempt was made to re-purpose an existing language testing system. More specifically, the paper investigated the suitability of an online diagnostic assessment system (DIALANG) for tracking the English language proficiency of three groups of university students in four different skills/aspects over a period of three years. The study presents the results, issues and challenges that arise when an assessment system is used for a different purpose than it was originally designed for and discusses the lessons learnt.
{"title":"Using DIALANG to track English language learners' progress over time","authors":"Nantia Kektsidou, Dina Tsagari","doi":"10.58379/qfsp2110","DOIUrl":"https://doi.org/10.58379/qfsp2110","url":null,"abstract":"The current paper presents a case study where an attempt was made to re-purpose an existing language testing system. More specifically, the paper investigated the suitability of an online diagnostic assessment system (DIALANG) for tracking the English language proficiency of three groups of university students in four different skills/aspects over a period of three years. The study presents the results, issues and challenges that arise when an assessment system is used for a different purpose than it was originally designed for and discusses the lessons learnt.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"52 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"78043742","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A.E. Tyler, L. Ortega, M. Uno & H.I. Park (eds.) Usage inspired L2 Instruction: Researched Pedagogy","authors":"Ivy Chen","doi":"10.58379/qntb4949","DOIUrl":"https://doi.org/10.58379/qntb4949","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"150 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"77419098","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Test fairness has been recognised as a fundamental requirement of test validation. Two quantitative approaches to investigate test fairness, the Rasch-based differential item functioning (DIF) detection method and a measurement invariance technique called multiple indicators, multiple causes (MIMIC), were adopted and compared in a test fairness study of the Pearson Test of English (PTE) Academic Reading test (n = 783). The Rasch partial credit model (PCM) showed no statistically significant uniform DIF across gender and, similarly, the MIMIC analysis showed that measurement invariance was maintained in the test. However, six pairs of significant non-uniform DIF (p < 0.05) were found in the DIF analysis. A discussion of the results and post-hoc content analysis is presented and the theoretical and practical implications of the study for test developers and language assessment are discussed.
测试公平性已被认为是测试验证的基本要求。采用两种定量方法来调查考试公平性，即基于rasch的差异项目功能(DIF)检测方法和称为多指标，多原因(MIMIC)的测量不变性技术，并在Pearson test of English (PTE)学术阅读测试(n = 783)的考试公平性研究中进行了比较。Rasch部分信用模型(PCM)显示，在性别之间没有统计学意义上的统一DIF，类似地，MIMIC分析显示，在测试中保持测量不变性。然而，在DIF分析中发现了6对显著的非均匀DIF (p < 0.05)。对结果和事后内容分析进行了讨论，并讨论了该研究对测试开发者和语言评估的理论和实践意义。
{"title":"Examining test fairness across gender in a computerised reading test: A comparison between the Rasch-based DIF technique and MIMIC","authors":"Xuelian Zhu, Vahid Aryadoust","doi":"10.58379/nvft3338","DOIUrl":"https://doi.org/10.58379/nvft3338","url":null,"abstract":"Test fairness has been recognised as a fundamental requirement of test validation. Two quantitative approaches to investigate test fairness, the Rasch-based differential item functioning (DIF) detection method and a measurement invariance technique called multiple indicators, multiple causes (MIMIC), were adopted and compared in a test fairness study of the Pearson Test of English (PTE) Academic Reading test (n = 783). The Rasch partial credit model (PCM) showed no statistically significant uniform DIF across gender and, similarly, the MIMIC analysis showed that measurement invariance was maintained in the test. However, six pairs of significant non-uniform DIF (p < 0.05) were found in the DIF analysis. A discussion of the results and post-hoc content analysis is presented and the theoretical and practical implications of the study for test developers and language assessment are discussed.","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"145 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"72665485","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Application of Rasch measurement theory in language assessment: Using measurement to enhance language assessment research and practice","authors":"Jinsong Fan, U. Knoch, T. Bond","doi":"10.58379/cmil9239","DOIUrl":"https://doi.org/10.58379/cmil9239","url":null,"abstract":"<jats:p>n/a</jats:p>","PeriodicalId":29650,"journal":{"name":"Studies in Language Assessment","volume":"1990 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90401456","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：