Crash Injection to Persistent Memory for Recovery Code Validation

Persistent Memory(PM) has non-volatilability and byte-addressability, and it can be used in many situations due to its high reliability and high performance. However, the persis-tent nature of PM has great impact on “rejuvenation”. Crash consistency bugs, which result in inconsistent data structures inside PM after system crashes, cannot be recovered by restarting the crashed program because the data structures in PM are not initialized with the restarts. Most of existing tools for detecting crash consistency bugs adopt static analysis that can explore a wider range of PM code regions and can detect bugs effectively, but it is hard for these tools to consider all the possible states because of the combinatorial explosion. In addition, PM programs usually have recovery code, which recovers PM data from inconsistent states, hence a crash consistency bug can be recovered to a correct state and it should not be reported as a bug. To simulate the execution of PM programs and detect crash consistency bugs dynamically, we propose PM Crash Injector, the first crash injection tool for PM programs to check the correctness of the recovery code. Like fault injection tools, PM Crash Injector injects system crashes into PM programs to cause crash consistency bugs intentionally. If the recovery code works correctly, inconsistent states in PM will be recovered, but if not, they will be left in PM regions and detected as unexpected behavior the program. PM Crash Injector has found 3 bugs in real-world PM systems and 6 manually inserted bugs in the sample programs of PMDK.