Manipulating Common Method Variance via Experimental Conditions

Abstract

Research data collected from single respondents may raise concerns regarding common method variance (CMV), which is believed to threaten the validity of findings. The primary concern is that CMV can inflate substantive relationships, such that they appear statistically significant when they are not. Thus, understanding the nature of CMV is critical, especially when one considers the popularity—and sometimes necessity—of using self-report data. Research examining CMV has found conflicting evidence about the impact of CMV. Researchers who believe CMV influences findings have proposed solutions to combat any real or perceived potential bias, including changing survey instructions and using marker variables, but few studies have examined the efficacy of these approaches. The purpose of this study is to examine the impact of these techniques and the nature of CMV using an experimental design. To conduct the experiment, multiple versions of a survey, which vary in their use of the remedial approaches, are utilized to collect data, which resulted in 1,069 usable responses. The experimental design was based on the faking literature and included instructions intended to induce or reduce the levels of CMV. Further, two different marker variables are used to determine the degree to which they create a psychological separation in substantive variables. Correlation analysis and measurement invariance are used to analyze the data. This study posits that, if CMV is a substantial concern for self-report data and these approaches are effective, then findings will differ in surveys that incorporate such approaches from surveys that do not. Results indicate few differences in experimental conditions, meaning that regardless of instructions or marker variable, substantive item correlations remained statistically similar. The results indicate this is likely due to the minimal impact of CMV, given that the proposed methods of correction did not significantly influence research findings. These findings have implications for researchers in that they do not support that CMV, or at least its proposed remedies, significantly alter findings. However, support for the null conclusions, in spite of appropriate statistical power, warrant future research examining the nature and impact of CMV.