On the Impact of log Compaction on Incrementally Checkpointing Stateful Stream-Processing Operators

Incremental checkpointing (IC) is a fault-tolerance technique used in several stateful distributed stream processing systems. It relies on continuously logging state updates to a remote storage service and periodically compacting the update-log via a background process. We highlight a tradeoff between the intensity of compaction of the IC update-log (and the associated resource overhead) and its impact on recovery time in such systems. We also highlight the control parameters that can be used to adjust this tradeoff in the Apache Samza stream processing system, and demonstrate this tradeoff experimentally.