Designing Safety Critical Software Systems to Manage Inherent Uncertainty

Abstract

Deploying machine learning algorithms in safety critical systems raises new challenges for system designers. The opaque nature of some algorithms together with the potentially large input space makes reasoning or formally proving safety difficult. In this paper, we argue that the inherent uncertainty that comes from using certain classes of machine learning algorithms can be mitigated through the development of software architecture design patterns. New or adapted patterns will allow faster roll out time for new technologies and decrease the negative impact machine learning components can have on safety critical systems. We outline the important safety challenges that machine learning algorithms raise and define three important directions for the development of new architectural patterns.