A setwise EWMA scheme for monitoring high-dimensional datastreams

Abstract

The monitoring of high-dimensional data streams has become increasingly important for real-time detection of abnormal activities in many statistical process control (SPC) applications. Although the multivariate SPC has been extensively studied in the literature, the challenges associated with designing a practical monitoring scheme for high-dimensional processes when between-streams correlation exists are yet to be addressed well. Classical [Formula: see text]-test-based schemes do not work well because the contamination bias in estimating the covariance matrix grows rapidly with the increase of dimension. We propose a test statistic which is based on the “divide-and-conquer” strategy, and integrate this statistic into the multivariate exponentially weighted moving average charting scheme for Phase II process monitoring. The key idea is to calculate the [Formula: see text] statistics on low-dimensional sub-vectors and to combine them together. The proposed procedure is essentially distribution-free and computation efficient. The control limit is obtained through the asymptotic distribution of the test statistic under some mild conditions on the dependence structure of stream observations. Our asymptotic results also shed light on quantifying the size of a reference sample required. Both theoretical analysis and numerical results show that the proposed method is able to control the false alarm rate and deliver robust change detection.