Statistical analysis of very high‐dimensional data sets of hierarchically structured binary variables with missing data: An application to marine corps readiness evaluations

Abstract

Abstract : The present analysis deals with very high-dimensional data sets, each one containing close to nine hundred binary variables. Each data set corresponds to an evaluation of one complex system. These data sets are characterized by large portions of missing data where, moreover, the unobserved variables are not the same in different evaluations. Thus, the problems which confront the statistical analysis are those of multivariate binary data analysis, where the number of variables is much larger than the sample size and in which missing data varies with the sample elements. The variables, however, are hierarchically structured and the problem of clustering variables to groups does not exist in the present study. In order to motivate the statistical problem under consideration, the Marine Corps Combat Readiness Evaluation System (MCCRES) is described for infantry battalions and then used for exposition. The present paper provides a statistical model for data from MCCRES and develops estimation and prediction procedures which utilize the dependence structure. The E-M algorithm is applied to obtain maximum likelihood estimates of the parameters of interest. Numerical examples illustrate the proposed methods.