Authenticity of academic lecture passages in high-stakes tests: A temporal fluency perspective

Abstract

Corpus-based studies have offered the domain definition inference for test developers. Yet, corpus-based studies on temporal fluency measures (e.g., speech rate) have been limited, especially in the context of academic lecture settings. This made it difficult for test developers to sample representative fluency features to create authentic listening passages. To address this issue, the Fluency Corpus of Academic English Lectures (FCAEL) was created to offer insight into the thresholds for temporal fluency features in academic lecture settings. The current study compared the corpus data to the academic lecture passages in the Test of English as a Foreign Language Internet-based test (TOEFL iBT) and International English Language Testing System (IELTS) to examine the domain definition inference of these tests. In total, 14 temporal fluency measures were examined. A bootstrapped one-way multivariate analysis of variance (MANOVA), followed by a series of bootstrapped analyses of variances (ANOVAs), independent t-test, and Tukey tests showed some support for the tests, although many limitations were also found. The study suggests the 25th–75th percentile of FCAEL as tentative thresholds for each temporal fluency feature. The proposal may be useful for test developers to create and revise test materials. Coding schemes, analysis codes, and raw corpus data are available on the project’s Open Science Framework page, exemplifying how Open Science can provide benefits beyond the academic community.