Demo: Querying Labelled Data with Scenario Programs for Sim-to-Real Validation

Simulation-based testing is becoming a core element of assessing the safety of autonomous vehicles (AVs) by government and industry. For example, the National Highway Traffic Safety Administration stated that self-driving technology should be tested in simulation before deployment [1], and Waymo recently used simulation to sup-port the claim that self-driving cars are safer than human drivers [2]. A number of open-source simulation environments designed to sup-port automated AV testing are available [3]–[5], as well as simulators which focus on realistic rendering of specific types of sensors such as LiDAR and radar [6], [7]. There are also a variety of black-box and white-box techniques to search for failure scenarios causing an AV to violate its safety specifications (e.g. [8]–[13]).