Comparison of Competing Approaches to Analyzing Cross-Classified Data: Random Effects Models, Ordinary Least Squares, or Fixed Effects with Cluster Robust Standard Errors

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in education. However, when the focus of a study is on the regression coefficients at level one rather than on the random effects, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) could be appropriate approaches. These alternative methods may be advantageous because they rely on weaker assumptions than what is required by CCREM. We conducted a Monte Carlo Simulation study to compare the performance of CCREM, OLS-CRVE, and FE-CRVE in models with crossed random effects, including conditions where homoscedasticity assumptions and exogeneity assumptions held and conditions where they were violated. We found that CCREM performed the best when its assumptions are all met. However, when homoscedasticity assumptions are violated, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. FE-CRVE showed the best performance when the exogeneity assumption is violated. Thus, we recommend two-way FE-CRVE as a good alternative to CCREM, particularly if the homoscedasticity or exogeneity assumptions of the CCREM might be in doubt.