Aris Chronarakis, Antonis Papaioannou, K. Magoutis
{"title":"On the Impact of log Compaction on Incrementally Checkpointing Stateful Stream-Processing Operators","authors":"Aris Chronarakis, Antonis Papaioannou, K. Magoutis","doi":"10.1109/SRDSW49218.2019.00008","DOIUrl":null,"url":null,"abstract":"Incremental checkpointing (IC) is a fault-tolerance technique used in several stateful distributed stream processing systems. It relies on continuously logging state updates to a remote storage service and periodically compacting the update-log via a background process. We highlight a tradeoff between the intensity of compaction of the IC update-log (and the associated resource overhead) and its impact on recovery time in such systems. We also highlight the control parameters that can be used to adjust this tradeoff in the Apache Samza stream processing system, and demonstrate this tradeoff experimentally.","PeriodicalId":297328,"journal":{"name":"2019 38th International Symposium on Reliable Distributed Systems Workshops (SRDSW)","volume":"63 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2019 38th International Symposium on Reliable Distributed Systems Workshops (SRDSW)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/SRDSW49218.2019.00008","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Incremental checkpointing (IC) is a fault-tolerance technique used in several stateful distributed stream processing systems. It relies on continuously logging state updates to a remote storage service and periodically compacting the update-log via a background process. We highlight a tradeoff between the intensity of compaction of the IC update-log (and the associated resource overhead) and its impact on recovery time in such systems. We also highlight the control parameters that can be used to adjust this tradeoff in the Apache Samza stream processing system, and demonstrate this tradeoff experimentally.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
