Examining the Relationships between Performance Requirements and “Not a Problem” Defect Reports

Abstract

Missing or imprecise requirements can lead stakeholders to make incorrect assumptions. A "not a problem" defect report (NaP) describes a software behavior that a stakeholder regards as a problem while the developer believes this behavior is acceptable and chooses not to take any action. As a result, a NaP wastes the time of the development team because resources are spent analyzing the problem but the quality of the software is not improved. Performance requirements specification and analysis are instance-based. System performance can change based upon the execution environment or usage patterns. To understand how the availability and precision of performance requirements can affect NaP occurrence rate, we conducted a case study on an embedded control module. We applied the performance refinement and evolution model to examine the relationship between each factor in the performance requirements and the corresponding NaP occurrence rate. Our findings show that precise specification of subjects or workloads lowers the occurrence rate of NaPs. Precise specification of measures or environments does not lower the occurrence rate of NaPs. Finally, the availability of performance requirements does not affect NaP occurrence rate in this case study.