Identifying silent failures of SaaS services using finite state machine based invariant analysis

Field failure analysis is usually driven by a characterization of the different time related properties of failure. This characterization does not help the production support team in understanding the root cause. In order to pinpoint the root cause of failure, one of the most effective techniques used is checking for violations of the system invariants which are the consistent, time invariant correlations that exist in the system. Understanding when and where these violations happen helps in detecting the root cause of the failure. Silent failures, on the other hand are characterized by no evidence of failures either in the console or in the field failure logs. They are unearthed at moments of crisis, either with a customer complaint or other cascading failures. These failures often result in data loss or data corruption, creating many latent errors. Accumulation of these errors over time results in degraded system performance. This represents the problem of software aging and restoration of the system, i.e. its rejuvenation becomes a critical need. Subsequent to the restoration, a rigorous failure detection mechanism is needed to detect them early. What we describe in the paper is a novel method that could be used to detect silent failures using a combination of invariant violation checking and finite state machine based analysis of the system. We use the audit-trail logs of system to extract information about the state and transitions for FSM representation. Currently our research work was limited to proving its efficiency. We applied this approach to our SaaS platform and were able to detect 36 silent failures over a period of 9 months. As next steps, we will implement this as a part of automated failure detection in the operational SaaS platforms.