Toward a Machine Learning Development Lifecycle for Product Certification and Approval in Aviation

Abstract

This article presents a new machine learning (ML) development lifecycle which will constitute the core of the new aeronautical standard on ML called AS6983, jointly being developed by working group WG-114/G34 of EUROCAE and SAE. The article also presents a survey of several existing standards and guidelines related to ML in aeronautics, automotive, and industrial domains by comparing and contrasting their scope, purpose, and results. Standards and guidelines reviewed include the European Union Aviation Safety Agency (EASA) Concept Paper, the DEEL (DEpendable and Explainable Learning) white paper “Machine Learning in Certified Systems”, Aerospace Vehicle System Institute (AVSI) Authorization for Expenditure (AFE) 87 report on Machine Learning, Guidance on the Assurance of Machine Learning for use in Autonomous Systems (AMLAS), Laboratoire National de Metrologie et d’Essais (LNE) Certification Standard of Processes for AI, the Underwriters Laboratories (UL) 4600 Safety Standard for Autonomous Vehicles, and the paper on Assuring the Machine Learning Lifecycle. These standards and guidelines are examined from the perspective of the learning assurance objectives they propose, and the means of evaluation and compliance for achieving these learning objectives. The reference used for comparison is the list of learning assurance objectives defined within the framework of AS6983 development. From this comparative analysis, and based on a coverage criterion defined in this article, only three (3) standards and guidelines exceed 50% coverage of the Machine Learning Development Lifecycle (MLDL) learning assurance objectives baseline. The next steps of this work are to update the AS6983 learning assurance objectives and improve the associated means of compliance to approach a coverage score of 100%, and offer a certification-based process to other domains that could benefit from the AS6983 standard.