Evaluating methodological quality of meta-analyses: A case study of meta-analyses on associations between parental involvement and students’ learning outcomes

Abstract

The present study employs an umbrella review of methodological quality in 41 meta-analyses examining associations between parental involvement and students’ learning outcomes. It develops a comprehensive methodology framework for evaluating the methodological quality of meta-analyses. The framework includes different aspects from aims and conceptualization to reporting of results in meta-analyses, addressing advanced methodological topics in meta-analysis, and considering methodological reflections on limitations and suggestions for future research from meta-analysts themselves. Application of the framework to the 41 meta-analyses enables us to identify various methodological strengths and areas for improvement. The strengths comprised clear definitions of problem, comprehensive search for primary studies, reliable coding, attending to methodological quality of primary studies, reporting magnitude and variation of effect sizes, and investigating publication bias. The areas for improvement pertained to conceptual and operational definitions of key variables, transparency in searching and coding primary studies, clarifying model assumptions, handling dependent effect sizes and missing data, investigating statistical outliers, determining statistical power, use of meta-regression, providing summary tables for primary studies, and interpreting effect sizes. The study is significant in pointing the way forward for enhancing the methodological rigor of meta-analyses on parental involvement and for providing a comprehensive evaluation framework that can be used to similarly identify strengths and areas for improvement in meta-analyses in other fields.