Why replicate? Systematic review of calls for replication in Language Teaching

Abstract

A first and critical step towards successful engagement with replication is to provide justifications to replicate the initial study. Despite existing guidelines, many researchers are still left uninformed of how to justify their replication. In order to explore what justifications were made in the past and thereby to provide additional guidelines, the present study conducted a systematic review of calls for replication in the journal of Language Teaching. The study analyzed 24 calls for replication published from 2010 to 2020 that encouraged replication of 50 initial studies published from 1976 to 2019. These calls for replication were coded for suggested modifications to initial studies and aspects of validity. Three hundred suggested modifications were identified, many of which were associated with external validity (38%) or internal validity (34%). Construct validity (15%) and statistical conclusion validity (7%) were rarely mentioned. Further analyses showed that suggestions on exploratory replication studies tended to address construct validity (e.g., pragmatic competence) and focus less on external validity, but the opposite was true for suggestions on confirmatory replication studies (e.g., corrective feedback). Suggestions for future replication research and calls for replication are discussed.