Evaluating early predictive performance of machine learning approaches for engineering change schedule – A case study using predictive process monitoring techniques

Abstract

By applying machine learning algorithms, predictive business process monitoring (PBPM) techniques provide an opportunity to counteract undesired outcomes of processes. An especially complex variation of business processes is the engineering change (EC) process. Here, failing to adhere to planned implementation dates can have severe impacts on assembly lines, and it is paramount that potential negative cases are identified as early as possible. Current PBPM research, however, has seldomly investigated the predictive performance of machine learning approaches and their applicability at early process steps, let alone for the EC process. In our research, we show that given adequate feature encoding, shallow learners can accurately predict schedule adherence after process initialisation. Based on EC data from an automotive manufacturer, we provide a case sensitive performance overview on algorithm-encoding combinations. For that, three algorithms (XGBoost, Random Forest, LSTM) were combined with four encoding techniques. The encoding techniques used were the two common aggregation-based and index-based last state encoding, and two new combinations of these, which we term advanced aggregation-based and complex aggregation-based encoding. The study indicates that XGBoost-index-encoded approaches outclass regarding average predictive performance, whereas Random-Forest-aggregation-encoded approaches perform better regarding temporal stability due to reduced influence by dynamic features. Our research provides a case-based reasoning approach for deciding on which algorithm-encoding combination and evaluation metrics to apply. In doing so, we provide a blueprint for an early warning and monitoring method within the EC process and other similarly complex processes.