Reduce the Handicap: Performance Estimation for AI Systems Safety Certification

Abstract

The safety validation of AI and ML-based systems is challenging, as (i) analytical validation needs to include the interaction with a complex and stochastic physical environment and (ii) empirical validation needs to observe very long time-horizons to get enough “statistical signal” for the typically very low safety-related incident rate. This paper proposes an approach that amplifies the empirical evidence by introducing a handicap that reduces the system performance—making safety-related failures empirically more visible in a controlled environment—and gradually removing the handicap so that the convergence to the final incident rate can be estimated. Two numerical case studies are used to support and exemplify the approach.