Model Selection for Mixed-Effects Location-Scale Models with Confidence Interval for LOO or WAIC Difference.

Abstract

LOO (Leave-One-Out cross-validation) and WAIC (Widely Applicable Information Criterion) are widely used for model selection in Bayesian statistics. Most studies select the model with the smallest value based on point estimates, often without considering the differences in fit indices or the uncertainty of the estimates. To address this gap, we propose a sequential method for comparing models based on confidence intervals for $\Delta \text{LOO}$ or $\Delta \text{WAIC.}$ A simulation study was conducted to evaluate this method in selecting mixed-effects location-scale models (MELSMs). Our study revealed that the sequential methods, especially when using a 90% confidence interval, achieved a higher accuracy rate of model selection compared to the point method when the true model was simple, had a large magnitude of random intercept in the scale model, or had a large sample size. Models selected by the sequential methods demonstrated higher power, narrower credible interval width, smaller standard errors for the fixed effect in the location model, and lower bias for the random effect of the intercept in the location model. Differences between LOO and WAIC were significant only when the level-1 sample size was small, with LOO performing better when the true model had homogeneous or severe heterogeneity in residual variances.