SFRAT – An Extendable Software Reliability Assessment Tool

Abstract

Since its release in the mid-1990s, the Microsoft Windows-based software reliability modeling tool CASRE has been downloaded over 3000 times from the Open Channel Foundation's website. It was also included on the CDROM distributed with the Handbook of Software Reliability Engineering (M. Lyu, ed). In the years since it was first released, however, CASRE has become more difficult to use. This is mainly because there have been no updates since 2000. The last version of Windows on which CASRE would reliably execute was Windows XP, and since it was developed explicitly for Windows, it is not feasible to run it on other platforms. Software development and acquisition organizations continued to be interested in using tools of the same type as CASRE. In 2013, the U.S. Naval Air Systems Command (NAVAIR) contacted the authors at the Jet Propulsion Laboratory and the University of Massachusetts to determine whether a) CASRE could be modified to run in contemporary environments, or b) whether a new tool with the same functionality as CASRE could be developed with modern programming languages and techniques. After weighing the alternatives, a decision was made to develop a new tool rather than updating CASRE. There were several reasons for which this decision was made. First of all, CASRE had been developed with the programming languages available at the time, in this case C and Fortran. Developing a new tool would allow use of modern, expressive languages and development environments that were particularly well suited to the statistical modeling domain (e.g., R, RStudio). Developing a new tool would also allow us to implement it so that it would run in modern operating environments, specifically Windows, Mac OS, Unix, and Linux. This last design decision would make it possible for users who would not have been able to run CASRE to use the new tool. We wanted to address the difficulty of adding new models to CASRE, since it was not architected for ease of modification. We wanted to develop a tool for which it would be easy to add new models and model evaluation techniques (e.g., prequential likelihood ratio, Akaike Information Criterion). This would enable organizations already using software reliability modeling to manage their testing to place the models they were using into a common framework, making it easier to evaluate multiple sets of results to gain additional insight into their testing process. Researchers would also gain an advantage by having a common framework in which to work with multiple models, analyze their results, and identify relationships among them. Finally, this would make it practical to distribute as open-source software, to which contributors could add new models and evaluation techniques as they were developed by the research community and validated by practitioners. We have used these ideas to develop the Software Failure and Reliability Assessment Tool (SFRAT). It is implemented in R, uses the Shiny user interface package, and will run in any environment in which R and RStudio can run. We intend for it to be an open-source tool with a mechanism for contributors to add new capabilities. Our hope is that distributing SFRAT as open-source software will allow it to retain currency in the software reliability practice and research communities.