Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality

The functionality of Fault Detection, Isolation, and Recovery (FDIR) is a key factor in achieving the high reliability of space systems. The test suites for the FDIR functionality in JAXA’s space systems are manually designed by expert engineers with decades of experience to achieve as high combination coverage with a small test suite as possible. However, there are only a few engineers who can perform such ad-hoc test suite design. Therefore, FDIR functionality testing requires a supportive method to generate a test suite with the high combination coverage with the smallest size that can be executed in the development timescale. In this paper, we describe our experience in applying popular combinatorial testing techniques to generate the real-world earth-observation satellite’s FDIR functionality test suites and comparing them with conventional human-derived test suite. The purpose of this comparison is to check the capability of the existing combinatorial testing methods toward FDIR functionality testing. Here, the FDIR functionality testing were treated as combinatorial configuration testing. As a result, we found that the 2-way coverage rate by the human, PICT, ACTS and the HAYST method were 72.7%, 66.3%, 68.8% and 72.2% with 16, 10, 10 and 14 test cases, respectively.