Towards a More Reliable Interpretation of Defect Models

Abstract

Software Quality Assurance (SQA) activities are exercised to ensure high-quality software systems. Defect models help developers identify the most risky modules to prioritise their limited SQA resources. The interpretation of defect models also helps managers understand what factors impact software quality to chart quality improvement plans. Unfortunately, the commonly-used interpretation techniques (e.g., ANOVA for logistic regression and variable importance for random forests) only explain defect models at the high level (e.g., what factors impact software quality). Researchers and practitioners also raise concerns about a lack of explainability of defect models that hinders the adoption in practice. This thesis hypothesises that: A lack of explainability poses a critical challenge when adopting defect models in practice. To validate the hypothesis, we formulate 3 research questions, i.e., (1) what is the best defect modelling workflow that produces the most accurate and reliable interpretation of defect models?, (2) what is the best technique for explaining the predictions of defect models?, and (3) how do practitioners perceive when adopting explainable defect models? Through case studies of publicly-available open-source and industrial software systems, the results show that correlated variables impact the interpretation of defect models and must be mitigated; our proposed feature selection technique, AutoSpearman, is the only studied feature selection technique that can automatically mitigate correlated variables with a little impact on model performance; and the instance-level interpretation of defect models is needed to derive actionable insights to guide operational and technical decisions in SQA efforts.