Multicore architectures are increasingly becoming prone to transient faults. In this paper we briefly present Shield, a middleware to provide transactional applications with resiliency to those faults that can happen anytime during the execution of a processor but do not cause any hardware interruption. Shield is inspired by the state machine replication approach, where computational resources are partitioned, the shared state is fully replicated, and requests are executed by all partitions in the same order. Shield embeds a set of algorithmic and system innovations to limit the overhead with respect to non-fault-tolerant solutions. They include a fast total order layer that lets application threads and computational nodes co-operate in order to fast deliver.
{"title":"On Preserving Data Integrity of Transactional Applications on Multicore Architectures","authors":"Mohamed Mohamedin, R. Palmieri, B. Ravindran","doi":"10.1109/ICDCS.2015.98","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.98","url":null,"abstract":"Multicore architectures are increasingly becoming prone to transient faults. In this paper we briefly present Shield, a middleware to provide transactional applications with resiliency to those faults that can happen anytime during the execution of a processor but do not cause any hardware interruption. Shield is inspired by the state machine replication approach, where computational resources are partitioned, the shared state is fully replicated, and requests are executed by all partitions in the same order. Shield embeds a set of algorithmic and system innovations to limit the overhead with respect to non-fault-tolerant solutions. They include a fast total order layer that lets application threads and computational nodes co-operate in order to fast deliver.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128389190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Localization is one of the key operations in mobile networks. Due to the limitations of GPS, many researchers have devised a variety of different range-free and range-based localization schemes. Range-free schemes utilize the connectivity information to localize mobile nodes. However, the use of the connectivity information allows a high degree of freedom in terms of pinpointing the location of mobile nodes, which leads to low localization precision. Range-based schemes can achieve high localization precision because they require the fine-granularity distance information. Nevertheless, they normally result in high computation complexity and do not work well when part of the distance measurements are missing. In this paper, we propose a distributed range-based localization scheme, DISCO, that uses a series of minimization problems that only involve convex optimization to arrive at high localization precision and low computation complexity. In addition, when some distance measurements are not available, DISCO utilizes the partial distance information to achieve satisfactory localization results. Furthermore, DISCO is a distributed algorithm, which means that it scales well. The performance of DISCO is analyzed through simulation experiments. An in-depth analysis of the time complexity of DISCO is also included in this paper.
{"title":"DISCO: A Distributed Localization Scheme for Mobile Networks","authors":"Jie Cheng, Qiang Ye, Hongwei Du, Chuang Liu","doi":"10.1109/ICDCS.2015.60","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.60","url":null,"abstract":"Localization is one of the key operations in mobile networks. Due to the limitations of GPS, many researchers have devised a variety of different range-free and range-based localization schemes. Range-free schemes utilize the connectivity information to localize mobile nodes. However, the use of the connectivity information allows a high degree of freedom in terms of pinpointing the location of mobile nodes, which leads to low localization precision. Range-based schemes can achieve high localization precision because they require the fine-granularity distance information. Nevertheless, they normally result in high computation complexity and do not work well when part of the distance measurements are missing. In this paper, we propose a distributed range-based localization scheme, DISCO, that uses a series of minimization problems that only involve convex optimization to arrive at high localization precision and low computation complexity. In addition, when some distance measurements are not available, DISCO utilizes the partial distance information to achieve satisfactory localization results. Furthermore, DISCO is a distributed algorithm, which means that it scales well. The performance of DISCO is analyzed through simulation experiments. An in-depth analysis of the time complexity of DISCO is also included in this paper.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"85 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131850990","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tan Zhang, Xian Zhang, Fangming Liu, H. Leng, Qian Yu, Guanfeng Liang
With the rapid proliferation of smartphones, hundreds of millions of mobile users are attracted to Instant Messaging (IM) apps. While such apps have brought convenience to our life, it comes with the price of great energy consumption, as these apps keep sending heartbeat messages to the server periodically in order to maintain an always-online connection. These frequent and fragmented transmissions result in a considerable amount of energy waste. In this paper, we investigate the "cost and potential of heartbeats". We quantify power consumption of heartbeats of real-world IM apps through extensive measurements. The measurement results confirm that huge power consumption is induced by heartbeats. The goal of this paper is to save energy by turning the energy wastage of heartbeats into transmitting useful data. Thus, we develop eTrain, a transmission management system running on Android phones, which takes advantage of IM heartbeats (as trains) to piggyback aggregated delay-tolerant apps' data such as e-mail and Weibo (as cargoes) via an online transmission strategy, so as to minimize the cumulative tail energy without sacrificing user-specified deadlines. Compared to other existing works, eTrain can reduce more energy consumption under the same settings. Experiments conducted on smartphones show that eTrain can achieve 12%-33% energy saving in various application scenarios.
{"title":"eTrain: Making Wasted Energy Useful by Utilizing Heartbeats for Mobile Data Transmissions","authors":"Tan Zhang, Xian Zhang, Fangming Liu, H. Leng, Qian Yu, Guanfeng Liang","doi":"10.1109/ICDCS.2015.20","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.20","url":null,"abstract":"With the rapid proliferation of smartphones, hundreds of millions of mobile users are attracted to Instant Messaging (IM) apps. While such apps have brought convenience to our life, it comes with the price of great energy consumption, as these apps keep sending heartbeat messages to the server periodically in order to maintain an always-online connection. These frequent and fragmented transmissions result in a considerable amount of energy waste. In this paper, we investigate the \"cost and potential of heartbeats\". We quantify power consumption of heartbeats of real-world IM apps through extensive measurements. The measurement results confirm that huge power consumption is induced by heartbeats. The goal of this paper is to save energy by turning the energy wastage of heartbeats into transmitting useful data. Thus, we develop eTrain, a transmission management system running on Android phones, which takes advantage of IM heartbeats (as trains) to piggyback aggregated delay-tolerant apps' data such as e-mail and Weibo (as cargoes) via an online transmission strategy, so as to minimize the cumulative tail energy without sacrificing user-specified deadlines. Compared to other existing works, eTrain can reduce more energy consumption under the same settings. Experiments conducted on smartphones show that eTrain can achieve 12%-33% energy saving in various application scenarios.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131683071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
To improve the quality of video streaming subject to video bitrate or communication channel capacity, a high-quality video is encoded into multiple layers of unequal importance. Layers that provide higher quality rely on the previous layers for successful reconstruction of transmitted video packets. Hence, if a video packet in a reference layer is corrupted or lost during transmission, the dependent layers cannot be reconstructed successfully, and the resources consumed to transmit them are wasted. To address this problem, unequal error protection (UEP) techniques have been proposed to provide appropriate level of protection to each layer according to their importance. Nonetheless, the importance of a piece of video content is determined by not only the layering structure, but also coding dependency imposed by encoding decisions. In this paper, based on a deep inspection of coding and prediction in SVC (a layered video coding standard) and an analysis of seven real SVC videos, we conclude that macro block-level coding dependency will provide a more accurate importance measure when applying UEP to protection video packets in noisy channels.
{"title":"Inspecting Coding Dependency in Layered Video Coding for Efficient Unequal Error Protection","authors":"M. R. Zakerinasab, Mea Wang","doi":"10.1109/ICDCS.2015.96","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.96","url":null,"abstract":"To improve the quality of video streaming subject to video bitrate or communication channel capacity, a high-quality video is encoded into multiple layers of unequal importance. Layers that provide higher quality rely on the previous layers for successful reconstruction of transmitted video packets. Hence, if a video packet in a reference layer is corrupted or lost during transmission, the dependent layers cannot be reconstructed successfully, and the resources consumed to transmit them are wasted. To address this problem, unequal error protection (UEP) techniques have been proposed to provide appropriate level of protection to each layer according to their importance. Nonetheless, the importance of a piece of video content is determined by not only the layering structure, but also coding dependency imposed by encoding decisions. In this paper, based on a deep inspection of coding and prediction in SVC (a layered video coding standard) and an analysis of seven real SVC videos, we conclude that macro block-level coding dependency will provide a more accurate importance measure when applying UEP to protection video packets in noisy channels.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129847940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In an intelligence transportation system (ITS), to increase traffic efficiency, a number of dynamic route guidance schemes have been designed to assist drivers in determining the optimal route for their travels. In order to determine optimal routes, it is critical to effectively predict the traffic condition of roads along the guided routes based on real-time traffic information to mitigate traffic congestion and improve traffic efficiency. In this paper, we propose a Dynamic En-route Decision real-time Route guidance (DEDR) scheme to effectively mitigate road congestion caused by the sudden increase of vehicles and reduce travel time. Particularly, DEDR considers real-time traffic information generation and transmission. Based on the shared traffic information, DEDR introduces Trust Probability to predict traffic conditions and dynamically en-route determine alternative optimal routes. In addition, DEDR considers multiple metrics to comprehensively assess traffic conditions and drivers can determine optimal route with individual preference of these metrics during travel. DEDR also considers effects of external factors (e.g., Bad weather, incidents, etc.) on traffic conditions. Through a combination of extensive theoretical analysis and simulation experiments, our data shows that DEDR can greatly increase the efficiency of an ITS in terms of great time efficiency and balancing efficiency in comparison with existing schemes.
{"title":"A Novel Dynamic En-Route Decision Real-Time Route Guidance Scheme in Intelligent Transportation Systems","authors":"Jie Lin, Wei Yu, Xinyu Yang, Qingyu Yang, Xinwen Fu, Wei Zhao","doi":"10.1109/ICDCS.2015.15","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.15","url":null,"abstract":"In an intelligence transportation system (ITS), to increase traffic efficiency, a number of dynamic route guidance schemes have been designed to assist drivers in determining the optimal route for their travels. In order to determine optimal routes, it is critical to effectively predict the traffic condition of roads along the guided routes based on real-time traffic information to mitigate traffic congestion and improve traffic efficiency. In this paper, we propose a Dynamic En-route Decision real-time Route guidance (DEDR) scheme to effectively mitigate road congestion caused by the sudden increase of vehicles and reduce travel time. Particularly, DEDR considers real-time traffic information generation and transmission. Based on the shared traffic information, DEDR introduces Trust Probability to predict traffic conditions and dynamically en-route determine alternative optimal routes. In addition, DEDR considers multiple metrics to comprehensively assess traffic conditions and drivers can determine optimal route with individual preference of these metrics during travel. DEDR also considers effects of external factors (e.g., Bad weather, incidents, etc.) on traffic conditions. Through a combination of extensive theoretical analysis and simulation experiments, our data shows that DEDR can greatly increase the efficiency of an ITS in terms of great time efficiency and balancing efficiency in comparison with existing schemes.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121871744","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Shankar Karuppayah, Stefanie Roos, C. Rossow, M. Mühlhäuser, Mathias Fischer
The emerging trend of highly-resilient P2P botnets poses a huge security threat to our modern society. Carefully designed countermeasures as applied in sophisticated P2P botnets such as P2P Zeus impede botnet monitoring and successive takedown. These countermeasures reduce the accuracy of the monitored data, such that an exact reconstruction of the botnet's topology is hard to obtain efficiently. However, an accurate topology snapshot, revealing particularly the identities of all bots, is crucial to execute effective botnet takedown operations. With the goal of obtaining the required snapshot in an efficient manner, we provide a detailed description and analysis of the P2P Zeus neighbor list restriction mechanism. As our main contribution, we propose ZeusMilker, a mechanism for circumventing the existing anti-monitoring countermeasures of P2P Zeus. In contrast to existing approaches, our mechanism deterministically reveals the complete neighbor lists of bots and hence can efficiently provide a reliable topology snapshot of P2P Zeus. We evaluated ZeusMilker on a real-world dataset and found that it outperforms state-of-the-art techniques for botnet monitoring with regard to the number of queries needed to retrieve a bot's complete neighbor list. Furthermore, ZeusMilker is provably optimal in retrieving the complete neighbor list, requiring at most 2n queries for an n-elemental list. Moreover, we also evaluated how the performance of ZeusMilker is impacted by various protocol changes designed to undermine its provable performance bounds.
{"title":"Zeus Milker: Circumventing the P2P Zeus Neighbor List Restriction Mechanism","authors":"Shankar Karuppayah, Stefanie Roos, C. Rossow, M. Mühlhäuser, Mathias Fischer","doi":"10.1109/ICDCS.2015.69","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.69","url":null,"abstract":"The emerging trend of highly-resilient P2P botnets poses a huge security threat to our modern society. Carefully designed countermeasures as applied in sophisticated P2P botnets such as P2P Zeus impede botnet monitoring and successive takedown. These countermeasures reduce the accuracy of the monitored data, such that an exact reconstruction of the botnet's topology is hard to obtain efficiently. However, an accurate topology snapshot, revealing particularly the identities of all bots, is crucial to execute effective botnet takedown operations. With the goal of obtaining the required snapshot in an efficient manner, we provide a detailed description and analysis of the P2P Zeus neighbor list restriction mechanism. As our main contribution, we propose ZeusMilker, a mechanism for circumventing the existing anti-monitoring countermeasures of P2P Zeus. In contrast to existing approaches, our mechanism deterministically reveals the complete neighbor lists of bots and hence can efficiently provide a reliable topology snapshot of P2P Zeus. We evaluated ZeusMilker on a real-world dataset and found that it outperforms state-of-the-art techniques for botnet monitoring with regard to the number of queries needed to retrieve a bot's complete neighbor list. Furthermore, ZeusMilker is provably optimal in retrieving the complete neighbor list, requiring at most 2n queries for an n-elemental list. Moreover, we also evaluated how the performance of ZeusMilker is impacted by various protocol changes designed to undermine its provable performance bounds.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"191 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123566642","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we introduce WhiFind, which is an indoor white space identification system. It leverages matrix completion techniques to provide accurate indoor white space availability information with only a small number of spectrum sensors deployed.
{"title":"WhiFind: A Matrix Completion Approach for Indoor White Space Identification","authors":"Xiao Hu, Fan Wu, Xiaofeng Gao, Guihai Chen","doi":"10.1109/ICDCS.2015.92","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.92","url":null,"abstract":"In this paper, we introduce WhiFind, which is an indoor white space identification system. It leverages matrix completion techniques to provide accurate indoor white space availability information with only a small number of spectrum sensors deployed.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131748092","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this work, we propose a dynamic coalescing algorithm for IEEE 802.3az standard, which dynamically adapts coalescing operations based on the current load and on the delay experienced in the link. Our results show that our algorithm almost doubles the energy efficiency of EEE with static coalescing while keeping packet delay bounded.
{"title":"Improving the Energy Benefit for 802.3az Using Dynamic Coalescing Techniques","authors":"Angelos Chatzipapas, V. Mancuso","doi":"10.1109/ICDCS.2015.80","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.80","url":null,"abstract":"In this work, we propose a dynamic coalescing algorithm for IEEE 802.3az standard, which dynamically adapts coalescing operations based on the current load and on the delay experienced in the link. Our results show that our algorithm almost doubles the energy efficiency of EEE with static coalescing while keeping packet delay bounded.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114464006","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
N. Pandey, Kaiwen Zhang, Stéphane Weiss, H. Jacobsen, R. Vitenberg
Modern applications for distributed publish/subscribe systems often require stream aggregation capabilities along with rich data filtering. When compared to other distributed systems, aggregation in pub/sub differentiates itself as a complex problem which involves dynamic dissemination paths that are difficult to predict and optimize for a priori, temporal fluctuations in publication rates, and the mixed presence of aggregated and non-aggregated workloads. In this paper, we propose a formalization for the problem of minimizing communication traffic in the context of aggregation in pub/sub. We present a solution to this minimization problem by using a reduction to the well-known problem of minimum vertex cover in a bipartite graph. This solution is optimal under the strong assumption of complete knowledge of future publications. We call the resulting algorithm "Aggregation Decision, Optimal with Complete Knowledge" (ADOCK). We also show that under a dynamic setting without full knowledge, ADOCK can still be applied to produce a low, yet not necessarily optimal, communication cost. We also devise a computationally cheaper dynamic approach called "Aggregation Decision with Weighted Publication" (WAD). We compare our solutions experimentally using two real datasets and explore the trade-offs with respect to communication and computation costs.
{"title":"Minimizing the Communication Cost of Aggregation in Publish/Subscribe Systems","authors":"N. Pandey, Kaiwen Zhang, Stéphane Weiss, H. Jacobsen, R. Vitenberg","doi":"10.1109/ICDCS.2015.54","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.54","url":null,"abstract":"Modern applications for distributed publish/subscribe systems often require stream aggregation capabilities along with rich data filtering. When compared to other distributed systems, aggregation in pub/sub differentiates itself as a complex problem which involves dynamic dissemination paths that are difficult to predict and optimize for a priori, temporal fluctuations in publication rates, and the mixed presence of aggregated and non-aggregated workloads. In this paper, we propose a formalization for the problem of minimizing communication traffic in the context of aggregation in pub/sub. We present a solution to this minimization problem by using a reduction to the well-known problem of minimum vertex cover in a bipartite graph. This solution is optimal under the strong assumption of complete knowledge of future publications. We call the resulting algorithm \"Aggregation Decision, Optimal with Complete Knowledge\" (ADOCK). We also show that under a dynamic setting without full knowledge, ADOCK can still be applied to produce a low, yet not necessarily optimal, communication cost. We also devise a computationally cheaper dynamic approach called \"Aggregation Decision with Weighted Publication\" (WAD). We compare our solutions experimentally using two real datasets and explore the trade-offs with respect to communication and computation costs.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123887934","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hao Wang, Li Chen, Kai Chen, Ziyang Li, Yiming Zhang, Haibing Guan, Zhengwei Qi, Dongsheng Li, Yanhui Geng
Data-parallel computing frameworks (DCF) such as MapReduce, Spark, and Dryad etc. Have tremendous applications in big data and cloud computing, and throw tons of flows into data center networks. In this paper, we design and implement FLOWPROPHET, a general framework to predict traffic flows for DCFs. To this end, we analyze and summarize the common features of popular DCFs, and gain a key insight: since application logic in DCFs is naturally expressed by directed acyclic graphs (DAG), DAG contains necessary time and data dependencies for accurate flow prediction. Based on the insight, FLOWPROPHET extracts DAGs from user applications, and uses the time and data dependencies to calculate flow information 4-tuple, (source, destination, flow_size, establish_time), ahead-of-time for all flows. We also provide generic programming interface to FLOWPROPHET, so that current and future DCFs can deploy FLOWPROPHET readily. We implement FLOWPROPHET on both Spark and Hadoop, and perform extensive evaluations on a testbed with 37 physical servers. Our implementation and experiments demonstrate that, with time in advance and minimal cost, FLOWPROPHET can achieve almost 100% accuracy in source, destination, and flow size predictions. With accurate prediction from FLOWPROPHET, the job completion time of a Hadoop TeraSort benchmark is reduced by 12.52% on our cluster with a simple network scheduler.
{"title":"FLOWPROPHET: Generic and Accurate Traffic Prediction for Data-Parallel Cluster Computing","authors":"Hao Wang, Li Chen, Kai Chen, Ziyang Li, Yiming Zhang, Haibing Guan, Zhengwei Qi, Dongsheng Li, Yanhui Geng","doi":"10.1109/ICDCS.2015.43","DOIUrl":"https://doi.org/10.1109/ICDCS.2015.43","url":null,"abstract":"Data-parallel computing frameworks (DCF) such as MapReduce, Spark, and Dryad etc. Have tremendous applications in big data and cloud computing, and throw tons of flows into data center networks. In this paper, we design and implement FLOWPROPHET, a general framework to predict traffic flows for DCFs. To this end, we analyze and summarize the common features of popular DCFs, and gain a key insight: since application logic in DCFs is naturally expressed by directed acyclic graphs (DAG), DAG contains necessary time and data dependencies for accurate flow prediction. Based on the insight, FLOWPROPHET extracts DAGs from user applications, and uses the time and data dependencies to calculate flow information 4-tuple, (source, destination, flow_size, establish_time), ahead-of-time for all flows. We also provide generic programming interface to FLOWPROPHET, so that current and future DCFs can deploy FLOWPROPHET readily. We implement FLOWPROPHET on both Spark and Hadoop, and perform extensive evaluations on a testbed with 37 physical servers. Our implementation and experiments demonstrate that, with time in advance and minimal cost, FLOWPROPHET can achieve almost 100% accuracy in source, destination, and flow size predictions. With accurate prediction from FLOWPROPHET, the job completion time of a Hadoop TeraSort benchmark is reduced by 12.52% on our cluster with a simple network scheduler.","PeriodicalId":129182,"journal":{"name":"2015 IEEE 35th International Conference on Distributed Computing Systems","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114316267","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：