Structured expert scenario methodology for autonomous system validation applied to a multi- UAS ground control station design

Abstract

For the past several years, our team has developed and experimented with a novel method to support the verification and validation (V&V) of autonomous systems. This method, entitled Expert-Guided Scenarios for Autonomy Validation (EGS-AV), combines an extensible, database-oriented information model for operational scenarios with a systematic process for eliciting validation information from subject matter experts (SMEs). EGS-AV fills a critical V&V need, providing a standard information model for operational scenarios and a comprehensive process for eliciting the highest-priority content of such scenarios. EGS-AV is designed to support V&V activities throughout the system development lifecycle and is especially valuable for finding flaws and weaknesses in the pre-prototype stage of complex, autonomous systems. In this paper, we describe the experimental application of EGS-AV to a ground control station (GCS) for multiple sUAS (small unmanned aircraft systems) in a future commercial services context. The GCS design, called Fleet HQ, was developed to the level of preliminary functions, architecture, and operational characteristics. The Fleet HQ design was then run through the EGS-AV method which identified and documented critical scenarios collaboratively with SMEs having a range of relevant expertise (piloting, design, operations management, testing, etc.). Outcomes of EGS-AV in this application included the targeted scenarios (most significant off-nominal factors and potential accident sequences) and recommended design considerations. Typically EGS-AV analysis products translate to design improvements or further V&V analysis, simulation, and/or testing. In this paper, we describe the Fleet HQ EGS-AV analysis activities and outcomes. We also draw conclusions based on this analysis for both the Fleet HQ design and the nascent EGS-AV method.