Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00164
Mengya Li, Yang Qin, Bing Liu, X. Chu
Blockchain is essentially a distributed ledger shared by all nodes in the system. All nodes in blockchain are equal, and each node holds all transactions and blocks in the network. As the network continues to expand, the data rises linearly. Participates are about to face the problem of storage limitation. Blockchain is hard to scale.This paper introduces ICIStrategy, a multi-node collaborative storage strategy based on intra-cluster integrity. In ICIStrategy, we divide all participates into several clusters. Each cluster requires holding all data of the network, whereas a node within the cluster does not need to maintain data integrity. It aims to solve the storage pressure by reducing the amount data that each participate need to store and reduce communication overhead by collaboratively storing and verifying blocks through in-cluster nodes. Moreover, the ICIStrategy could greatly save the overhead of bootstrapping. We show the mode of operation in our strategy. We further analysis the performance of ICIStrategy and conduct simulation experiments. The results of several comparative experiments show that our strategy just needs 25% of storage space needed by Rapidchain, which indeed solve the problem of storage limitation and improve the blockchain performance.
{"title":"A Multi-node Collaborative Storage Strategy via Clustering in Blockchain Network","authors":"Mengya Li, Yang Qin, Bing Liu, X. Chu","doi":"10.1109/ICDCS47774.2020.00164","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00164","url":null,"abstract":"Blockchain is essentially a distributed ledger shared by all nodes in the system. All nodes in blockchain are equal, and each node holds all transactions and blocks in the network. As the network continues to expand, the data rises linearly. Participates are about to face the problem of storage limitation. Blockchain is hard to scale.This paper introduces ICIStrategy, a multi-node collaborative storage strategy based on intra-cluster integrity. In ICIStrategy, we divide all participates into several clusters. Each cluster requires holding all data of the network, whereas a node within the cluster does not need to maintain data integrity. It aims to solve the storage pressure by reducing the amount data that each participate need to store and reduce communication overhead by collaboratively storing and verifying blocks through in-cluster nodes. Moreover, the ICIStrategy could greatly save the overhead of bootstrapping. We show the mode of operation in our strategy. We further analysis the performance of ICIStrategy and conduct simulation experiments. The results of several comparative experiments show that our strategy just needs 25% of storage space needed by Rapidchain, which indeed solve the problem of storage limitation and improve the blockchain performance.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127488539","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00147
Youfu Li, Matteo Interlandi, Fotis Psallidas, Wei Wang, C. Zaniolo
In Data-Intensive Scalable Computing (DISC) Systems, data transformations are concealed by exposed APIs, and intermediate execution moments are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection) are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present SEIZE, a unified framework that enables dataflow inspection— wiretapping the data-path with listening logic —in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow puncutation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying SEIZE in both Apache Spark and Apache Flink. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5% in Spark and 10% in Flink).
{"title":"SEIZE User Desired Moments: Runtime Inspection for Parallel Dataflow Systems","authors":"Youfu Li, Matteo Interlandi, Fotis Psallidas, Wei Wang, C. Zaniolo","doi":"10.1109/ICDCS47774.2020.00147","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00147","url":null,"abstract":"In Data-Intensive Scalable Computing (DISC) Systems, data transformations are concealed by exposed APIs, and intermediate execution moments are masked under dataflow transitions. Consequently, many crucial features and optimizations (e.g., debugging, data provenance, runtime skew detection) are not well-supported. Inspired by our experience in implementing features and optimizations over DISC systems, we present SEIZE, a unified framework that enables dataflow inspection— wiretapping the data-path with listening logic —in MapReduce-style programming model. We generalize our lessons learned by providing a set of primitives defining dataflow inspection, orchestration options for different inspection granularities, and operator decomposition and dataflow puncutation strategy for dataflow intervention. We demonstrate the generality and flexibility of the approach by deploying SEIZE in both Apache Spark and Apache Flink. Our experiments show that, the overhead introduced by the inspection logic is most of the time negligible (less than 5% in Spark and 10% in Flink).","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122816225","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00075
Lijie Xu, Xingtong Ye, Kai Kang, Tian Guo, Wensheng Dou, Wei Wang, Jun Wei
Stream clustering is an important data mining technique to capture the evolving patterns in real-time data streams. Today’s data streams, e.g., IoT events and Web clicks, are usually high-speed and contain dynamically-changing patterns. Existing stream clustering algorithms usually follow an online-offline paradigm with a one-record-at-a-time update model, which was designed for running in a single machine. These stream clustering algorithms, with this sequential update model, cannot be efficiently parallelized and fail to deliver the required high throughput for stream clustering.In this paper, we present DistStream, a distributed framework that can effectively scale out online-offline stream clustering algorithms. To parallelize these algorithms for high throughput, we develop a mini-batch update model with efficient parallelization approaches. To maintain high clustering quality, DistStream’s mini-batch update model preserves the update order in all the computation steps during parallel execution, which can reflect the recent changes for dynamically-changing streaming data. We implement DistStream atop Spark Streaming, as well as four representative stream clustering algorithms based on DistStream. Our evaluation on three real-world datasets shows that DistStream-based stream clustering algorithms can achieve sublinear throughput gain and comparable (99%) clustering quality with their single-machine counterparts.
{"title":"DistStream: An Order-Aware Distributed Framework for Online-Offline Stream Clustering Algorithms","authors":"Lijie Xu, Xingtong Ye, Kai Kang, Tian Guo, Wensheng Dou, Wei Wang, Jun Wei","doi":"10.1109/ICDCS47774.2020.00075","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00075","url":null,"abstract":"Stream clustering is an important data mining technique to capture the evolving patterns in real-time data streams. Today’s data streams, e.g., IoT events and Web clicks, are usually high-speed and contain dynamically-changing patterns. Existing stream clustering algorithms usually follow an online-offline paradigm with a one-record-at-a-time update model, which was designed for running in a single machine. These stream clustering algorithms, with this sequential update model, cannot be efficiently parallelized and fail to deliver the required high throughput for stream clustering.In this paper, we present DistStream, a distributed framework that can effectively scale out online-offline stream clustering algorithms. To parallelize these algorithms for high throughput, we develop a mini-batch update model with efficient parallelization approaches. To maintain high clustering quality, DistStream’s mini-batch update model preserves the update order in all the computation steps during parallel execution, which can reflect the recent changes for dynamically-changing streaming data. We implement DistStream atop Spark Streaming, as well as four representative stream clustering algorithms based on DistStream. Our evaluation on three real-world datasets shows that DistStream-based stream clustering algorithms can achieve sublinear throughput gain and comparable (99%) clustering quality with their single-machine counterparts.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"69 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129960935","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00034
Zhimeng Yin, Wenchao Jiang, Ruofeng Liu, S. Kim, T. He
Overcrowded wireless devices in unlicensed bands compete for spectrum access, generating excessive cross-technology interference (CTI), which has become a major source of performance degradation especially for low-power IoT (e.g., ZigBee) networks. This paper presents a new forward error correction (FEC) mechanism to alleviate CTI, named SafetyNet. Designed for ZigBee, SafetyNet is inspired by the observation that ZigBee is overly robust for environment noises, but insufficiently protected from high-power CTI. By effectively embedding correction code bits into the PHY layer SafetyNet significantly enhances CTI robustness without compromising noise resilience. SafetyNet additionally offers a set of unique features including (i) transparency, making it compatible with millions of readily-deployed ZigBee devices and (ii) zero additional cost on energy and spectrum, as it does not increase the frame length. Such features not only differentiate SafetyNet from known FEC techniques (e.g., Hamming and Reed-Solomon), but also uniquely position it to be critically beneficial for today’s crowded wireless environment. Our extensive evaluation on physical testbeds shows that SafetyNet significantly improves ZigBee’s CTI robustness under a wide range of networking settings, where it corrects 55% of the corrupted packets.
{"title":"SafetyNet: Interference Protection via Transparent PHY Layer Coding","authors":"Zhimeng Yin, Wenchao Jiang, Ruofeng Liu, S. Kim, T. He","doi":"10.1109/ICDCS47774.2020.00034","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00034","url":null,"abstract":"Overcrowded wireless devices in unlicensed bands compete for spectrum access, generating excessive cross-technology interference (CTI), which has become a major source of performance degradation especially for low-power IoT (e.g., ZigBee) networks. This paper presents a new forward error correction (FEC) mechanism to alleviate CTI, named SafetyNet. Designed for ZigBee, SafetyNet is inspired by the observation that ZigBee is overly robust for environment noises, but insufficiently protected from high-power CTI. By effectively embedding correction code bits into the PHY layer SafetyNet significantly enhances CTI robustness without compromising noise resilience. SafetyNet additionally offers a set of unique features including (i) transparency, making it compatible with millions of readily-deployed ZigBee devices and (ii) zero additional cost on energy and spectrum, as it does not increase the frame length. Such features not only differentiate SafetyNet from known FEC techniques (e.g., Hamming and Reed-Solomon), but also uniquely position it to be critically beneficial for today’s crowded wireless environment. Our extensive evaluation on physical testbeds shows that SafetyNet significantly improves ZigBee’s CTI robustness under a wide range of networking settings, where it corrects 55% of the corrupted packets.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129837173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00194
J. Nelson, R. Palmieri
One of the consequences of ultra-fast networks like InfiniBand is that known implications of Non-uniform Memory Access (NUMA) locality now constitute a higher percentage of execution time for distributed systems employing Remote Direct Memory Access (RDMA). Our findings quantify the role NUMA plays in RDMA operation performance and uncovers unexpected behavior.
{"title":"On the Performance Impact of NUMA on One-sided RDMA Interactions","authors":"J. Nelson, R. Palmieri","doi":"10.1109/ICDCS47774.2020.00194","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00194","url":null,"abstract":"One of the consequences of ultra-fast networks like InfiniBand is that known implications of Non-uniform Memory Access (NUMA) locality now constitute a higher percentage of execution time for distributed systems employing Remote Direct Memory Access (RDMA). Our findings quantify the role NUMA plays in RDMA operation performance and uncovers unexpected behavior.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"11 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123689278","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00160
P. Hillmann, Marcus Knüpfer, Erik Heiland, A. Karcher
The constantly growing size of blockchains becomes a challenge with the increasing usage. Especially the storage of unwanted data in a blockchain is an issue, because it cannot be removed naturally. In order to counteract this problem, we present the first concept for the selective deletion of single entries in a blockchain. For this purpose, the general consensus algorithm is extended by the functionality of regularly creating summary blocks. Previous data of the chain are summarized and stored again in a new block, leaving out unwanted information. With a shifting marker of the Genesis Block, data can be deleted from the beginning of a blockchain. In this way, the technology of the blockchain becomes fully transactional. The concept is independent of a specific block structure, network structure, or consensus algorithm. Moreover, this functionality can be adapted to current blockchains to solve multiple problems related to scalability. This approach enables the transfer of blockchain technology to further fields of application, among others in the area of Industry 4.0 and Product Life-cycle Management.
{"title":"Selective Deletion in a Blockchain","authors":"P. Hillmann, Marcus Knüpfer, Erik Heiland, A. Karcher","doi":"10.1109/ICDCS47774.2020.00160","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00160","url":null,"abstract":"The constantly growing size of blockchains becomes a challenge with the increasing usage. Especially the storage of unwanted data in a blockchain is an issue, because it cannot be removed naturally. In order to counteract this problem, we present the first concept for the selective deletion of single entries in a blockchain. For this purpose, the general consensus algorithm is extended by the functionality of regularly creating summary blocks. Previous data of the chain are summarized and stored again in a new block, leaving out unwanted information. With a shifting marker of the Genesis Block, data can be deleted from the beginning of a blockchain. In this way, the technology of the blockchain becomes fully transactional. The concept is independent of a specific block structure, network structure, or consensus algorithm. Moreover, this functionality can be adapted to current blockchains to solve multiple problems related to scalability. This approach enables the transfer of blockchain technology to further fields of application, among others in the area of Industry 4.0 and Product Life-cycle Management.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"101 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116359705","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00096
Xiaohai Dai, Jiang Xiao, Wenhui Yang, Chaofan Wang, Jian Chang, Rui Han, Hai Jin
In the Bitcoin system, transaction history of an address can be useful in many scenarios, such as the balance calculation and behavior analysis. However, it is non-trivial for a common user who runs a light node to fetch historical transactions, since it only stores headers without any transaction details. It usually has to request a full node who stores the complete data. Validation of these query results is critical and mainly involves two aspects: correctness and completeness. The former can be implemented via Merkle branch easily, while the latter is quite difficult in the Bitcoin protocol. To enable the completeness validation, a strawman design is proposed, which simply includes the BF (Bloom filter) in the headers. However, since the size of BF is about KB, light nodes in the strawman will suffer from the incremental storage burden. What’s worse, an integrated block must be transmitted when BF cannot work, resulting in large network overhead. In this paper, we propose LVQ, the first lightweight verifiable query approach that reduces the storage requirement and network overhead at the same time. To be specific, by only storing the hash of BF in headers, LVQ keeps data stored by light nodes being little. Besides, LVQ introduces a novel BMT (BF integrated Merkle Tree) structure for lightweight query, which can eliminate the communication costs of query results by merging the multiple successive BFs. Furthermore, when BF cannot work, a lightweight proof by SMT (Sorted Merkle Tree) is exploited to further reduce the network overhead. The security analysis confirms LVQ’s ability to enable both correctness and completeness validation. In addition, the experimental results demonstrate its lightweight.
{"title":"LVQ: A Lightweight Verifiable Query Approach for Transaction History in Bitcoin","authors":"Xiaohai Dai, Jiang Xiao, Wenhui Yang, Chaofan Wang, Jian Chang, Rui Han, Hai Jin","doi":"10.1109/ICDCS47774.2020.00096","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00096","url":null,"abstract":"In the Bitcoin system, transaction history of an address can be useful in many scenarios, such as the balance calculation and behavior analysis. However, it is non-trivial for a common user who runs a light node to fetch historical transactions, since it only stores headers without any transaction details. It usually has to request a full node who stores the complete data. Validation of these query results is critical and mainly involves two aspects: correctness and completeness. The former can be implemented via Merkle branch easily, while the latter is quite difficult in the Bitcoin protocol. To enable the completeness validation, a strawman design is proposed, which simply includes the BF (Bloom filter) in the headers. However, since the size of BF is about KB, light nodes in the strawman will suffer from the incremental storage burden. What’s worse, an integrated block must be transmitted when BF cannot work, resulting in large network overhead. In this paper, we propose LVQ, the first lightweight verifiable query approach that reduces the storage requirement and network overhead at the same time. To be specific, by only storing the hash of BF in headers, LVQ keeps data stored by light nodes being little. Besides, LVQ introduces a novel BMT (BF integrated Merkle Tree) structure for lightweight query, which can eliminate the communication costs of query results by merging the multiple successive BFs. Furthermore, when BF cannot work, a lightweight proof by SMT (Sorted Merkle Tree) is exploited to further reduce the network overhead. The security analysis confirms LVQ’s ability to enable both correctness and completeness validation. In addition, the experimental results demonstrate its lightweight.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"257 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124227035","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00116
Yue Li, Han Liu, Zhiqiang Yang, Qian Ren, Lei Wang, Bangdao Chen
Smart contracts on the Ethereum blockchain are notoriously known as vulnerable to external attacks. Many of their issues led to a considerably large financial loss as they resulted from broken payments by digital assets, e.g., cryptocurrency. Existing research focused on specific patterns to find such problems, e.g., reentrancy bug, nondeterministic recipient etc., yet may lead to false alarms or miss important issues. To mitigate these limitations, we designed the SafePay analysis framework to find unfair payments in Ethereum smart contracts. Compared to existing analyzers, SafePay can detect potential blockchain transactions with feasible exploits thus effectively avoid false reports. Specifically, the detection is driven by a systematic search for violations on fair value exchange (FVE), i.e., a new security invariant introduced in SafePay to indicate that each party “fairly” pays to others. The preliminary evaluation validated the efficacy of SafePay by reporting previously unknown issues and decreasing the number of false alarms.
{"title":"SafePay on Ethereum: A Framework For Detecting Unfair Payments in Smart Contracts","authors":"Yue Li, Han Liu, Zhiqiang Yang, Qian Ren, Lei Wang, Bangdao Chen","doi":"10.1109/ICDCS47774.2020.00116","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00116","url":null,"abstract":"Smart contracts on the Ethereum blockchain are notoriously known as vulnerable to external attacks. Many of their issues led to a considerably large financial loss as they resulted from broken payments by digital assets, e.g., cryptocurrency. Existing research focused on specific patterns to find such problems, e.g., reentrancy bug, nondeterministic recipient etc., yet may lead to false alarms or miss important issues. To mitigate these limitations, we designed the SafePay analysis framework to find unfair payments in Ethereum smart contracts. Compared to existing analyzers, SafePay can detect potential blockchain transactions with feasible exploits thus effectively avoid false reports. Specifically, the detection is driven by a systematic search for violations on fair value exchange (FVE), i.e., a new security invariant introduced in SafePay to indicate that each party “fairly” pays to others. The preliminary evaluation validated the efficacy of SafePay by reporting previously unknown issues and decreasing the number of false alarms.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127903783","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00113
Yichen Jia, Feng Chen
Log-Structured Merge (LSM) tree based key-value stores are widely deployed in data centers. Due to its complex internal structures, appropriately configuring a modern key-value data store system, which can have more than 50 parameters with various hardware and system settings, is a highly challenging task. Currently, the industry still heavily relies on a traditional, experience-based, hand-tuning approach for performance tuning. Many simply adopt the default setting out of the box with no changes. Auto-tuning, as a self-adaptive solution, is thus highly appealing for achieving optimal or near-optimal performance in real-world deployment.In this paper, we quantitatively study and compare five optimization methods for auto-tuning the performance of LSM-tree based key-value stores. In order to evaluate the auto-tuning processes, we have conducted an exhaustive set of experiments over RocksDB, a representative LSM-tree data store. We have collected over 12,000 experimental records in 6 months, with about 2,000 software configurations of 6 parameters on different hardware setups. We have compared five representative algorithms, in terms of throughput, the 99th percentile tail latency, convergence time, real-time system throughput, and the iteration process, etc. We find that multi-objective optimization (MOO) methods can achieve a good balance among multiple targets, which satisfies the unique needs of key-value services. The more specific Quality of Service (QoS) requirements users can provide, the better performance these algorithms can achieve. We also find that the number of concurrent threads and the write buffer size are the two most impactful parameters determining the throughput and the 99th percentile tail latency across different hardware and workloads. Finally, we provide system-level explanations for the auto-tuning results and also discuss the associated implications for system designers and practitioners. We hope this work will pave the way towards a practical, high-speed auto-tuning solution for key-value data store systems.
{"title":"Kill Two Birds with One Stone: Auto-tuning RocksDB for High Bandwidth and Low Latency","authors":"Yichen Jia, Feng Chen","doi":"10.1109/ICDCS47774.2020.00113","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00113","url":null,"abstract":"Log-Structured Merge (LSM) tree based key-value stores are widely deployed in data centers. Due to its complex internal structures, appropriately configuring a modern key-value data store system, which can have more than 50 parameters with various hardware and system settings, is a highly challenging task. Currently, the industry still heavily relies on a traditional, experience-based, hand-tuning approach for performance tuning. Many simply adopt the default setting out of the box with no changes. Auto-tuning, as a self-adaptive solution, is thus highly appealing for achieving optimal or near-optimal performance in real-world deployment.In this paper, we quantitatively study and compare five optimization methods for auto-tuning the performance of LSM-tree based key-value stores. In order to evaluate the auto-tuning processes, we have conducted an exhaustive set of experiments over RocksDB, a representative LSM-tree data store. We have collected over 12,000 experimental records in 6 months, with about 2,000 software configurations of 6 parameters on different hardware setups. We have compared five representative algorithms, in terms of throughput, the 99th percentile tail latency, convergence time, real-time system throughput, and the iteration process, etc. We find that multi-objective optimization (MOO) methods can achieve a good balance among multiple targets, which satisfies the unique needs of key-value services. The more specific Quality of Service (QoS) requirements users can provide, the better performance these algorithms can achieve. We also find that the number of concurrent threads and the write buffer size are the two most impactful parameters determining the throughput and the 99th percentile tail latency across different hardware and workloads. Finally, we provide system-level explanations for the auto-tuning results and also discuss the associated implications for system designers and practitioners. We hope this work will pave the way towards a practical, high-speed auto-tuning solution for key-value data store systems.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"24 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"113941596","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-01DOI: 10.1109/ICDCS47774.2020.00169
R. Shinkuma, Yoshinobu Yamada, Takehiro Sato, E. Oki
Real-time prediction of communications (or road) traffic by using cloud computing and sensor data collected by Internet-of-Things (IoT) devices would be very useful application of big-data analytics. However, upstream data flow from IoT devices to the cloud server could be problematic, even in fifth generation (5G) networks, because networks have mainly been designed for downstream data flows like for video delivery. This paper proposes a framework in which a software defined network (SDN), edge server, and cloud server cooperate with each other to control the upstream flow to maintain the accuracy of the real-time predictions under the condition of a limited network bandwidth. The framework consists of a system model, methods of prediction and determining the importance of data using machine learning, and a mathematical optimization. Our key idea is that the SDN controller optimizes data flows in the SDN on the basis of feature importance scores, which indicate the importance of the data in terms of the prediction accuracy. The feature importance scores are extracted from the prediction model by a machine-learning feature selection method that has traditionally been used to suppress effects of noise or irrelevant input variables. Our framework is examined in a simulation study using a real dataset consisting of mobile traffic logs. The results validate the framework; it maintains prediction accuracy under the constraint of limited available network bandwidth. Potential applications are also discussed.
{"title":"Flow control in SDN-Edge-Cloud cooperation system with machine learning","authors":"R. Shinkuma, Yoshinobu Yamada, Takehiro Sato, E. Oki","doi":"10.1109/ICDCS47774.2020.00169","DOIUrl":"https://doi.org/10.1109/ICDCS47774.2020.00169","url":null,"abstract":"Real-time prediction of communications (or road) traffic by using cloud computing and sensor data collected by Internet-of-Things (IoT) devices would be very useful application of big-data analytics. However, upstream data flow from IoT devices to the cloud server could be problematic, even in fifth generation (5G) networks, because networks have mainly been designed for downstream data flows like for video delivery. This paper proposes a framework in which a software defined network (SDN), edge server, and cloud server cooperate with each other to control the upstream flow to maintain the accuracy of the real-time predictions under the condition of a limited network bandwidth. The framework consists of a system model, methods of prediction and determining the importance of data using machine learning, and a mathematical optimization. Our key idea is that the SDN controller optimizes data flows in the SDN on the basis of feature importance scores, which indicate the importance of the data in terms of the prediction accuracy. The feature importance scores are extracted from the prediction model by a machine-learning feature selection method that has traditionally been used to suppress effects of noise or irrelevant input variables. Our framework is examined in a simulation study using a real dataset consisting of mobile traffic logs. The results validate the framework; it maintains prediction accuracy under the constraint of limited available network bandwidth. Potential applications are also discussed.","PeriodicalId":158630,"journal":{"name":"2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS)","volume":"82 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132076851","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：