Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors

Abstract

Robustness testing is crucial for verifying autonomous vehicles, especially for safety-critical deep learning components like light detection and ranging (LiDAR) object detectors. Metamorphic testing (MT) assesses the robustness by automatically generating test cases based on abstract system specifications known as metamorphic relations (MRs). However, a key challenge is ensuring a traceable safety argumentation for MRs that is in line with industry standards. To ensure this traceability, we derive seven traceable metamorphic transformations from defects identified through interviews with industry experts. Another challenge is prioritizing failures by safety criticality, as not all failing test cases, as evaluated by current intersection over union (IoU)-based metamorphic oracles, pose the same safety risk. We address this by introducing novel egocentric test oracles based on traffic participants’ bounding boxes shifted into or out of the ego vehicle’s expected lane. Testing five LiDAR object detection systems working on two datasets by executing half a million metamorphic test cases (MTCs) shows that the number of failures decreases from 48k using IoU metrics to 342 safety-critical failures with our novel test oracle “shift out of ego lane.” This reduction enables testers to stay within the test analysis budget and, hence, manually analyze each failed MTC by prioritizing safety-critical test failures.