Self-Starting Monitoring and Dynamic Sampling of High-Dimensional Data Streams

Abstract

In today’s manufacturing industries, the development of sensor technology and Internet of Things has made real-time process monitoring of high-dimensional data increasingly vital. However, resource constraints, such as limited power, budget, and transmission capacity, often prevent access to full data streams in real time. This means that practitioners need to effectively monitor the process based on only partially observed data by dynamically deciding the sampling layout in real time. Another common challenge of process monitoring in practice is the lack of historical reference data, which can occur due to process/system upgrades or equipment replacements. To address these critical challenges, this paper proposes MASS (Monitoring with Adaptive Sampling under Self-starting scheme), a novel self-starting monitoring approach tailored to monitor high-dimensional data streams when only limited resources and historical reference data are available. Our monitoring framework is based on a quantile-based nonparametric CUSUM procedure with likelihood ratio-based statistics, and then the Thompson Sampling (TS) algorithm is adopted to handle partially observed data in the self-starting scenario. A key feature of our proposed method is its adaptive estimation of the out-of-control distribution and data quantiles, which ensures robust detection for various shifts in data streams with arbitrary and heterogeneous distributions, even in cases with limited reference data. The outperformance of the proposed method is demonstrated through simulation experiments and a real-world case study. Note to Practitioners—This paper is motivated by the critical challenges of online process monitoring when only limited resources (e.g., limited power availability, limited number of sensors, and limited transmission capacity) and limited historical in-control reference data are available. For example, consider a scenario where a newly established production system requires online monitoring across multiple data streams. In such cases, there is often a deficiency of reference data crucial for constructing a reliable control chart. Additionally, due to resource constraints, it is frequently infeasible to gather information from all streams associated with the production line at each epoch in real time. Unlike previous methods which require either a sufficient amount of reference data, or fully observable data streams, this paper proposes a novel monitoring and dynamic sampling scheme to effectively monitor partially observable data streams with only a small amount of reference data. To implement the methodology, it requires: (i) to initiate the process by estimating quantiles with a small amount of reference data, (ii) to determine which data streams to observe at each time epoch, (iii) to adaptively update the estimation of process parameters during online monitoring, and (iv) to construct a set of local and global statistics that can be used to quickly detect the system anomaly in real time. Numerical experiments and a real-world case study suggest that our proposed method efficiently leverages available data to reduce detection delays and enhance effectiveness against various shifts, in comparison to the benchmark methods.