Assisting pre-delivery firmware quality assessments using ensemble learning

Abstract

ABSTRACTThis study uses retrospective data for firmware tests as the input data sets to train four machine learning models with embedded standalone classifiers. None of these models provide accurate predictions during validation, so model optimization trials adjust the training-validation data portfolio and hyper parameters for each model. Consequently, only the random forest classifier with the best parametric settings just achieves the 90% prediction accuracy required by the standard. Ensemble learning (EL) is then applied using several combinations over the standalone models, and the EL model using logistic regression as the meta classifier increases the accuracy by 6% (i.e. to 96%), which is sufficient for establishing a predictive system. Using the ‘X-minute’ method, it is further identified that the execution period (also the data sampling period) for the sequential read test workload can be reduced from 30 (in current practice) to 20 minutes and that the predictions are sufficiently accurate for system implementation using the EL model. Applying the similarity confirmation method for each pair of ‘score vectors’ (each of which contains a model’s prediction accuracies), several observations distinguishing the performance and the predictive behavioral patterns of the benchmarked models are further confirmed. The knowledge from this advanced research has implications which may benefit future practice in industry.CO EDITOR-IN-CHIEF: Sun, Hung-MinASSOCIATE EDITOR: Sun, Hung-MinKEYWORDS: Quality controlfirmware testingensemble machine learningprocess re-engineering and optimizationdecision-support systemAI in industry Nomenclature AI=artificial intelligenceAPS=automated predictive systemCD=continuous deliveryCI=continuous integrationCOVID-19=corona-virus disease 2019CSV=comma-separated valuesCWV=criteria weight vectorDDDM (D3M)=data-driven decision-makingDSS=decision support systemsEL=ensemble learningFN=false negativeFP=false positiveFW=firmwareI/O=input and outputk-NN=k nearest neighborsLR=logistic regressionMADM=multi-attribute decision-makingMCDM=multi-criteria decision-makingML=machine learningOWV=opinion weight vectorR&D=research and developmentRF=random forestROV=rand order vectorSCM=similarity confirmation methodSOP=standard operating procedureSSD=solid state driveSV=score vectorSVM=support vector machineTN=true negativeTP=true positiveTTM=time to marketVCS=version control systemDisclosure statementNo potential conflict of interest was reported by the authors.Additional informationFundingThis work was supported by the Ministry of Science and Technology, Taiwan (ROC), under grants [MOST-108-2511-H-009-009-MY3, MOST-109-2410-H-992 -015 and MOST-111-2410-H-992-011], each in part.