Missing values appearing in multivariate time series often prevent further and in-depth analysis in real-world applications. To handle those missing values, advanced multivariate time series imputation methods are expected to (1) consider bi-directional temporal correlations, (2) model cross-variable correlations, and (3) approximate original data's distribution. However, most of existing approaches are not able to meet all the three above-mentioned requirements. Drawing on advances in machine learning, we propose BRNN-GAN, a generative adversarial network with bi-directional RNN cells. The BRNN cell is designed to model bi-directional temporal and cross-variable correlations, and the GAN architecture is employed to learn original data's distribution. By conducting comprehensive experiments on two public datasets, the experimental results show that our proposed BRNN-GAN outperforms all the baselines in terms of achieving the lowest Mean Absolute Error (MAE).
{"title":"BRNN-GAN: Generative Adversarial Networks with Bi-directional Recurrent Neural Networks for Multivariate Time Series Imputation","authors":"Zejun Wu, Chao Ma, Xiaochuan Shi, Libing Wu, Dian Zhang, Yutian Tang, M. Stojmenovic","doi":"10.1109/ICPADS53394.2021.00033","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00033","url":null,"abstract":"Missing values appearing in multivariate time series often prevent further and in-depth analysis in real-world applications. To handle those missing values, advanced multivariate time series imputation methods are expected to (1) consider bi-directional temporal correlations, (2) model cross-variable correlations, and (3) approximate original data's distribution. However, most of existing approaches are not able to meet all the three above-mentioned requirements. Drawing on advances in machine learning, we propose BRNN-GAN, a generative adversarial network with bi-directional RNN cells. The BRNN cell is designed to model bi-directional temporal and cross-variable correlations, and the GAN architecture is employed to learn original data's distribution. By conducting comprehensive experiments on two public datasets, the experimental results show that our proposed BRNN-GAN outperforms all the baselines in terms of achieving the lowest Mean Absolute Error (MAE).","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129647240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00051
Ziyu Wang, Cong Sun, Qingsong Yao, Duo Ding, Jianfeng Ma
The recent advance of the Internet of Things and autonomous systems brings massive security threats to the network of low-end embedded devices. Remote attestation is a hardware-assisted technique to verify the integrity and trustworthiness of software on remote devices. The recently proposed collective remote attestations have focused on attesting to the highly dynamic and disruptive device networks. However, they are generally inefficient due to the homogeneous node setting for the robustness of attestation reports aggregation. In this work, we propose Delica, an efficient and robust collective attestation framework for dynamic and disruptive networks. We differentiate the role of provers and aggregators to limit the redundant communications and attestation evidence aggregations for efficiency. Delica is capable of mitigating DoS attacks and detecting physical and black-hole attacks. The experimental results and analysis show that Delica can greatly reduce the per-node computational cost and reduce the network attestation cost by over 75% compared with the state-of-the-art approaches on disruptive networks.
{"title":"Delica: Decentralized Lightweight Collective Attestation for Disruptive IoT Networks","authors":"Ziyu Wang, Cong Sun, Qingsong Yao, Duo Ding, Jianfeng Ma","doi":"10.1109/ICPADS53394.2021.00051","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00051","url":null,"abstract":"The recent advance of the Internet of Things and autonomous systems brings massive security threats to the network of low-end embedded devices. Remote attestation is a hardware-assisted technique to verify the integrity and trustworthiness of software on remote devices. The recently proposed collective remote attestations have focused on attesting to the highly dynamic and disruptive device networks. However, they are generally inefficient due to the homogeneous node setting for the robustness of attestation reports aggregation. In this work, we propose Delica, an efficient and robust collective attestation framework for dynamic and disruptive networks. We differentiate the role of provers and aggregators to limit the redundant communications and attestation evidence aggregations for efficiency. Delica is capable of mitigating DoS attacks and detecting physical and black-hole attacks. The experimental results and analysis show that Delica can greatly reduce the per-node computational cost and reduce the network attestation cost by over 75% compared with the state-of-the-art approaches on disruptive networks.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"462 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129567338","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00107
Christian Siebert
Checksums are used to detect errors that might occur while storing or communicating data. Checking the integrity of data is well-established, but only for smaller data sets. Contrary, supercomputers have to deal with huge amounts of data, which introduces failures that may remain undetected. Therefore, additional protection becomes a necessity at large scale. However, checking the integrity of larger data sets, especially in case of distributed data, clearly requires parallel approaches. We show how popular checksums, such as CRC-32 or Adler-32, can be parallelized efficiently. This also disproves a widespread belief that parallelizing aforementioned checksums, especially in a scalable way, is not possible. The mathematical properties behind these checksums enable a method to combine partial checksums such that its result corresponds to the checksum of the concatenated partial data. Our parallel checksum algorithm utilizes this combination idea in a scalable hierarchical reduction scheme to combine the partial checksums from an arbitrary number of processing elements. Although this reduction scheme can be implemented manually using most parallel programming interfaces, we use the Message Passing Interface, which supports such a functionality directly via non-commutative user-defined reduction operations. In conjunction with the efficient checksum capabilities of the zlib library, our algorithm can not only be implemented conveniently and in a portable way, but also very efficiently. Additional shared-memory parallelization within compute nodes completes our hybrid parallel checksum solutions, which show a high scalability of up to 524,288 threads. At this scale, computing the checksums of 240 TiB data took only 3.4 seconds for CRC-32 and 2.6 seconds for Adler-32. Finally, we discuss the APES application as a representative of dynamic supercomputer applications. Thanks to our scalable checksum algorithm, even such applications are now able to detect many errors within their distributed data sets.
{"title":"Highly Scalable Parallel Checksums","authors":"Christian Siebert","doi":"10.1109/ICPADS53394.2021.00107","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00107","url":null,"abstract":"Checksums are used to detect errors that might occur while storing or communicating data. Checking the integrity of data is well-established, but only for smaller data sets. Contrary, supercomputers have to deal with huge amounts of data, which introduces failures that may remain undetected. Therefore, additional protection becomes a necessity at large scale. However, checking the integrity of larger data sets, especially in case of distributed data, clearly requires parallel approaches. We show how popular checksums, such as CRC-32 or Adler-32, can be parallelized efficiently. This also disproves a widespread belief that parallelizing aforementioned checksums, especially in a scalable way, is not possible. The mathematical properties behind these checksums enable a method to combine partial checksums such that its result corresponds to the checksum of the concatenated partial data. Our parallel checksum algorithm utilizes this combination idea in a scalable hierarchical reduction scheme to combine the partial checksums from an arbitrary number of processing elements. Although this reduction scheme can be implemented manually using most parallel programming interfaces, we use the Message Passing Interface, which supports such a functionality directly via non-commutative user-defined reduction operations. In conjunction with the efficient checksum capabilities of the zlib library, our algorithm can not only be implemented conveniently and in a portable way, but also very efficiently. Additional shared-memory parallelization within compute nodes completes our hybrid parallel checksum solutions, which show a high scalability of up to 524,288 threads. At this scale, computing the checksums of 240 TiB data took only 3.4 seconds for CRC-32 and 2.6 seconds for Adler-32. Finally, we discuss the APES application as a representative of dynamic supercomputer applications. Thanks to our scalable checksum algorithm, even such applications are now able to detect many errors within their distributed data sets.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"75 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124986170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00042
Lifang Lin, Yuhui Deng, Yi Zhou
Data deduplication has been widely used to improve storage space utilization, however, it is baffled by data fragmen-tation: logically consecutive chunks physically scattered across various containers. Many rewriting schemes, rewriting fragment-ed duplicate chunks into new containers, attempt to alleviate the restore performance degradation caused by fragmentation. Unfortunately, these schemes rely on a fixed threshold and fail to choose the appropriate set of old containers for rewriting, which leads to substantial redundant chunks existing in the retrieved containers when restoring backups. To address this issue, we propose a flexible threshold rewriting scheme to improve restore performance while maintaining high backup performance. We define an effectiveness metric - valid container reference counts (VCRC) - that facilitates identifying the appropriate containers for rewriting. We design a greedy-algorithm-based algorithm called F-greedy that dynamically adjusts the threshold according to the distribution of containers' VCRC, aiming to rewrite low-VCRC containers. We quantitatively evaluate F-greedy on three real-world backup datasets in terms of restore performance, backup performance, and storage overhead. The empirical results show that compared with two state-of-the-art schemes (Capping and SMR), our scheme improves the restore speed of the exiting algorithms by 1.3x - 2.4x while achieving similar backup performance.
{"title":"Improving Restore Performance of Deduplication Systems via a Greedy Rewriting Scheme","authors":"Lifang Lin, Yuhui Deng, Yi Zhou","doi":"10.1109/ICPADS53394.2021.00042","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00042","url":null,"abstract":"Data deduplication has been widely used to improve storage space utilization, however, it is baffled by data fragmen-tation: logically consecutive chunks physically scattered across various containers. Many rewriting schemes, rewriting fragment-ed duplicate chunks into new containers, attempt to alleviate the restore performance degradation caused by fragmentation. Unfortunately, these schemes rely on a fixed threshold and fail to choose the appropriate set of old containers for rewriting, which leads to substantial redundant chunks existing in the retrieved containers when restoring backups. To address this issue, we propose a flexible threshold rewriting scheme to improve restore performance while maintaining high backup performance. We define an effectiveness metric - valid container reference counts (VCRC) - that facilitates identifying the appropriate containers for rewriting. We design a greedy-algorithm-based algorithm called F-greedy that dynamically adjusts the threshold according to the distribution of containers' VCRC, aiming to rewrite low-VCRC containers. We quantitatively evaluate F-greedy on three real-world backup datasets in terms of restore performance, backup performance, and storage overhead. The empirical results show that compared with two state-of-the-art schemes (Capping and SMR), our scheme improves the restore speed of the exiting algorithms by 1.3x - 2.4x while achieving similar backup performance.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"204 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122839659","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00036
Kunyu Li, Nan Wang, Xinyu Liu
In recommender systems, due to the lack of explicit feedback features, datasets with implicit feedback are always accustomed to train all samples without separating them during model training, without considering the non-consistency of samples. This leads to a significant decrease in sample utilization and creates challenges for model training. Also, little work has been done to explore the intrinsic laws implied in the implicit feedback dataset and how to effectively train the implicit feedback data. In this paper, we first summarize the variation pattern of loss with model training for different rating samples in the explicit feedback dataset, and find that model training is highly sensitive to the ratings. Second, we design an adaptive hierarchical training function with dynamic thresholds that can effectively distinguish different rating samples in the dataset, thus optimizing the implicit feedback dataset into an explicit feedback dataset to some extent. Finally, to better learn samples with different ratings, we also propose an adaptive hierarchical training strategy to obtain better training results in the implicit feedback dataset. Extensive experiments on three datasets show that our approach achieves excellent performance and greatly improves the performance of the model.
{"title":"Sensitivity loss training based implicit feedback","authors":"Kunyu Li, Nan Wang, Xinyu Liu","doi":"10.1109/ICPADS53394.2021.00036","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00036","url":null,"abstract":"In recommender systems, due to the lack of explicit feedback features, datasets with implicit feedback are always accustomed to train all samples without separating them during model training, without considering the non-consistency of samples. This leads to a significant decrease in sample utilization and creates challenges for model training. Also, little work has been done to explore the intrinsic laws implied in the implicit feedback dataset and how to effectively train the implicit feedback data. In this paper, we first summarize the variation pattern of loss with model training for different rating samples in the explicit feedback dataset, and find that model training is highly sensitive to the ratings. Second, we design an adaptive hierarchical training function with dynamic thresholds that can effectively distinguish different rating samples in the dataset, thus optimizing the implicit feedback dataset into an explicit feedback dataset to some extent. Finally, to better learn samples with different ratings, we also propose an adaptive hierarchical training strategy to obtain better training results in the implicit feedback dataset. Extensive experiments on three datasets show that our approach achieves excellent performance and greatly improves the performance of the model.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122014212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00097
Shaolin Yu, Jihong Zhu, Jiali Yang, Yulong Zhan
To improve the overall efficiency and reliability of Byzantine protocols in large sparse networks, we propose a new system assumption for developing multi-scale fault-tolerant systems, with which several kinds of multi-scale Byzantine protocols are developed in large sparse networks with high system assumption coverage. By extending the traditional Byzantine adversary to the multi-scale adversaries, it is shown that efficient deterministic Byzantine broadcast and Byzantine agreement can be built in logarithmic-degree networks. Meanwhile, it is shown that the multi-scale adversary can make a finer trade-off between the system assumption coverage and the overall efficiency of the Byzantine protocols, especially when a small portion of the low-layer small-scale protocols are allowed to fail arbitrarily. With this, efficient Byzantine protocols can be built in large sparse networks with high system reliability.
{"title":"Boosting Byzantine Protocols in Large Sparse Networks with High System Assumption Coverage","authors":"Shaolin Yu, Jihong Zhu, Jiali Yang, Yulong Zhan","doi":"10.1109/ICPADS53394.2021.00097","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00097","url":null,"abstract":"To improve the overall efficiency and reliability of Byzantine protocols in large sparse networks, we propose a new system assumption for developing multi-scale fault-tolerant systems, with which several kinds of multi-scale Byzantine protocols are developed in large sparse networks with high system assumption coverage. By extending the traditional Byzantine adversary to the multi-scale adversaries, it is shown that efficient deterministic Byzantine broadcast and Byzantine agreement can be built in logarithmic-degree networks. Meanwhile, it is shown that the multi-scale adversary can make a finer trade-off between the system assumption coverage and the overall efficiency of the Byzantine protocols, especially when a small portion of the low-layer small-scale protocols are allowed to fail arbitrarily. With this, efficient Byzantine protocols can be built in large sparse networks with high system reliability.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121059376","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Bitcoin, the first decentralized cryptocurrency system, uses a simple but effective difficulty adjustment algorithm to stabilize its average time of the block creation at 10 minutes. Over time, the volatility of the Bitcoin price has become higher and higher, and it causes the total hashrate (the hash power of the entire network) constantly fluctuating. Both facts and our experimental results prove that Bitcoin's difficulty adjustment algorithm cannot respond in time while the total hashrate is constantly fluctuating. Hence, we propose a consensus protocol with a real-time difficulty adjustment algorithm, RTPoW. RTPoW allows the blockchain to adjust the difficulty target of each block by predicting the real-time total hashrate, so the block time can remain stable even if the total hashrate is wildly fluctuating. To evaluate the effect of RTPoW, we implemented a simulator of an experimental environment and tested our algorithm. The results obtained have confirmed its effectiveness and stability.
{"title":"RTPoW: A Proof-of-Work Consensus Scheme with Real-Time Difficulty Adjustment Algorithm","authors":"Weijia Feng, Zhenfu Cao, Jiachen Shen, Xiaolei Dong","doi":"10.1109/ICPADS53394.2021.00035","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00035","url":null,"abstract":"Bitcoin, the first decentralized cryptocurrency system, uses a simple but effective difficulty adjustment algorithm to stabilize its average time of the block creation at 10 minutes. Over time, the volatility of the Bitcoin price has become higher and higher, and it causes the total hashrate (the hash power of the entire network) constantly fluctuating. Both facts and our experimental results prove that Bitcoin's difficulty adjustment algorithm cannot respond in time while the total hashrate is constantly fluctuating. Hence, we propose a consensus protocol with a real-time difficulty adjustment algorithm, RTPoW. RTPoW allows the blockchain to adjust the difficulty target of each block by predicting the real-time total hashrate, so the block time can remain stable even if the total hashrate is wildly fluctuating. To evaluate the effect of RTPoW, we implemented a simulator of an experimental environment and tested our algorithm. The results obtained have confirmed its effectiveness and stability.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"95 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124690969","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00077
Jing Yang, Zhenqiang Xu, Jiliang Wang
Low-Power Wide-Area Networks (LPWANs) have been shown as a promising technique to provide long-range low-power communication for large-scale IoT devices. In this paper, however, we show the poor performance of LoRa network due to its link diversity in macro- and micro- scope through one-month measurements in an area of $2.2 kmtimes 1.5 km$. We present FerryLink, which exploits such link diversity and leverages peer nodes to ferry data of weak links, to combat performance degradation. Traditional arts (e.g., building multi-hop networks) are inefficient or too heavyweight for the current star-topology-based LoRa network. FerryLink thus proposes a novel ferry mechanism combining RSSI sampling and Channel Activity detection(CAD) to suit multiple orthogonal transmission parameters of LoRa. To reduce energy overhead, FerryLink leverages convention windows for coarse-grained transmission synchronization between two coupled nodes. Finally, FerryLink utilizes the orthogonality of uplink and downlink signals to avoid data redundancy due to the ferry mechanism, maintaining comparable capacity with original LPWANs. We build FerryLink on top of LoRaWANwith commercial off-the-shelf hardware. The extensive evaluation results show that FerryLink effectively improves the packet delivery rate (PDR) of LoRa nodes (to over 95%), achieves 2x less energy overhead, and increases communication range by 50% compared with the original LoRaWAN.
{"title":"FerryLink: Combating Link Degradation for Practical LPWAN Deployments","authors":"Jing Yang, Zhenqiang Xu, Jiliang Wang","doi":"10.1109/ICPADS53394.2021.00077","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00077","url":null,"abstract":"Low-Power Wide-Area Networks (LPWANs) have been shown as a promising technique to provide long-range low-power communication for large-scale IoT devices. In this paper, however, we show the poor performance of LoRa network due to its link diversity in macro- and micro- scope through one-month measurements in an area of $2.2 kmtimes 1.5 km$. We present FerryLink, which exploits such link diversity and leverages peer nodes to ferry data of weak links, to combat performance degradation. Traditional arts (e.g., building multi-hop networks) are inefficient or too heavyweight for the current star-topology-based LoRa network. FerryLink thus proposes a novel ferry mechanism combining RSSI sampling and Channel Activity detection(CAD) to suit multiple orthogonal transmission parameters of LoRa. To reduce energy overhead, FerryLink leverages convention windows for coarse-grained transmission synchronization between two coupled nodes. Finally, FerryLink utilizes the orthogonality of uplink and downlink signals to avoid data redundancy due to the ferry mechanism, maintaining comparable capacity with original LPWANs. We build FerryLink on top of LoRaWANwith commercial off-the-shelf hardware. The extensive evaluation results show that FerryLink effectively improves the packet delivery rate (PDR) of LoRa nodes (to over 95%), achieves 2x less energy overhead, and increases communication range by 50% compared with the original LoRaWAN.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129795986","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00112
Haipeng Du, Danfu Yuan, Weizhan Zhang, Q. Zheng
HTTP adaptive streaming (HAS) has revolutionized video distribution over the Internet due to its prominent benefit of outstanding quality of experience (QoE). Due to the pull-based nature of HTTP/1.1, the client must make requests for each segment. This usually causes high request overhead and low bandwidth utilization and finally reduces QoE. Currently, research into the HAS adaptive bitrate algorithm typically focuses on the server-push feature introduced in the new HTTP standard, which enables the client to receive multiple segments with a single request. Every time a request is sent, the client must simultaneously make decisions on the number of segments the server should push and the bitrate of these future segments. As the decision space complexity increases, existing rule-based strategies inevitably fail to achieve optimal performance. In this paper, we present D-Push, an HAS framework that combines deep reinforcement learning (DRL) techniques. Instead of relying on inaccurate assumptions about the environment and network capacity variation models, D-Push trains a DRL model and makes decisions by exploiting the QoE of past decisions through the training process and adapts to a wide range of highly dynamic environments. The experimental results show that D-Push outperforms the existing state-of-the-art algorithm by 12%-24% in terms of the average QoE.
{"title":"Dynamic Push for HTTP Adaptive Streaming with Deep Reinforcement Learning","authors":"Haipeng Du, Danfu Yuan, Weizhan Zhang, Q. Zheng","doi":"10.1109/ICPADS53394.2021.00112","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00112","url":null,"abstract":"HTTP adaptive streaming (HAS) has revolutionized video distribution over the Internet due to its prominent benefit of outstanding quality of experience (QoE). Due to the pull-based nature of HTTP/1.1, the client must make requests for each segment. This usually causes high request overhead and low bandwidth utilization and finally reduces QoE. Currently, research into the HAS adaptive bitrate algorithm typically focuses on the server-push feature introduced in the new HTTP standard, which enables the client to receive multiple segments with a single request. Every time a request is sent, the client must simultaneously make decisions on the number of segments the server should push and the bitrate of these future segments. As the decision space complexity increases, existing rule-based strategies inevitably fail to achieve optimal performance. In this paper, we present D-Push, an HAS framework that combines deep reinforcement learning (DRL) techniques. Instead of relying on inaccurate assumptions about the environment and network capacity variation models, D-Push trains a DRL model and makes decisions by exploiting the QoE of past decisions through the training process and adapts to a wide range of highly dynamic environments. The experimental results show that D-Push outperforms the existing state-of-the-art algorithm by 12%-24% in terms of the average QoE.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130930378","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-12-01DOI: 10.1109/ICPADS53394.2021.00122
Jiannan Zheng, Haitao Zhang, Yilin Jin, Huadong Ma
With the rapid development of high-speed Internet access and popularization of high-performance smart devices, past decade has witnessed the great development of crowdsourced live streaming (CLS) service. Transcoding and video distribution are essential in CLS service to guarantee viewer engagement. Large CLS systems gradually migrate their services to multi-cloud platforms. However, highly dynamic viewers' requests influence transcoding and CDN distribution decisions, eventually lead to fluctuation in QoE and increase in operational cost. It is challenging for the CLS system to serve viewer's requests in multi-cloud platforms with fluctuation in cloud transcoding and distribution performance. In this paper, we propose a collaborative framework of cloud transcoding and distribution supporting CLS service. First, we define cost model and QoE model in multi-cloud platforms, comprehensively considering cloud transcoding and distribution. Second, we propose a collaborative cost-efficient approach based on multi-agent decision model. We use a G-Greedy exploration approach to learn what actions to take by exploration and exploitation based on the state of current environment. The trace-driven experiments demonstrate that our proposed approach is cost-efficient and QoE-maintained and can reduce operational cost compared with alternatives (5.37%-21.21%) while maintaining QoE of viewers.
{"title":"Collaborative Framework of Cloud Transcoding and Distribution Supporting Cost-Efficient Crowdsourced Live Streaming","authors":"Jiannan Zheng, Haitao Zhang, Yilin Jin, Huadong Ma","doi":"10.1109/ICPADS53394.2021.00122","DOIUrl":"https://doi.org/10.1109/ICPADS53394.2021.00122","url":null,"abstract":"With the rapid development of high-speed Internet access and popularization of high-performance smart devices, past decade has witnessed the great development of crowdsourced live streaming (CLS) service. Transcoding and video distribution are essential in CLS service to guarantee viewer engagement. Large CLS systems gradually migrate their services to multi-cloud platforms. However, highly dynamic viewers' requests influence transcoding and CDN distribution decisions, eventually lead to fluctuation in QoE and increase in operational cost. It is challenging for the CLS system to serve viewer's requests in multi-cloud platforms with fluctuation in cloud transcoding and distribution performance. In this paper, we propose a collaborative framework of cloud transcoding and distribution supporting CLS service. First, we define cost model and QoE model in multi-cloud platforms, comprehensively considering cloud transcoding and distribution. Second, we propose a collaborative cost-efficient approach based on multi-agent decision model. We use a G-Greedy exploration approach to learn what actions to take by exploration and exploitation based on the state of current environment. The trace-driven experiments demonstrate that our proposed approach is cost-efficient and QoE-maintained and can reduce operational cost compared with alternatives (5.37%-21.21%) while maintaining QoE of viewers.","PeriodicalId":309508,"journal":{"name":"2021 IEEE 27th International Conference on Parallel and Distributed Systems (ICPADS)","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131502174","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：