{"title":"apache flink中流应用的能量感知调度和两层协调负载平衡","authors":"Hongjian Li, Junlin Li, Xiaolin Duan, Jianglin Xia","doi":"10.1016/j.future.2024.107681","DOIUrl":null,"url":null,"abstract":"<div><div>Apache Flink has become one of the highly regarded streaming computing frameworks with its excellent advantages of high throughput, low latency, and high reliability. However, the default task scheduling policy follows the first-come-first-served principle, which fails to fully consider the differences in energy efficiency and resource loading of nodes in heterogeneous clusters and may lead to high energy consumption and uneven load distribution when executing jobs. To solve this problem, this paper proposes a two-tier coordinated load balancing and energy-saving scheduling optimization strategy. First, we construct an energy efficiency model based on Service Level Agreements (SLA) and design an Energy-Saving Scheduling Algorithm (ESSA) based on this model, aiming to reduce the energy consumption of Flink clusters when executing jobs. This ESSA algorithm integrally considers the effects of two SLA performance metrics including node response time and throughput on node energy consumption, as well as the differences in the energy efficiencies of different nodes in heterogeneous clusters. Second, in order to solve the load imbalance problem that may be caused by Flink’s default scheduling policy, an Energy-Aware Two-Tier Coordinated Load Balancing algorithm (TTCLB-EA) is proposed, which optimizes the cluster load at both the inter-node and intra-node levels through task based on energy efficiency priorities. Experimental results show that compared with the default scheduling strategy, round-robin scheduling strategy, and St-Stream, the proposed algorithm improves about 14.59%, 12.75%, and 7.32% in load balancing, while saving about 14.52%, 10.54%, and 7.58% in energy consumption, respectively. The proposed algorithms not only enhance the performance of the Flink cluster but also help to reduce energy consumption and achieve more efficient resource utilization.</div></div>","PeriodicalId":55132,"journal":{"name":"Future Generation Computer Systems-The International Journal of Escience","volume":"166 ","pages":"Article 107681"},"PeriodicalIF":6.2000,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Energy-aware scheduling and two-tier coordinated load balancing for streaming applications in apache flink\",\"authors\":\"Hongjian Li, Junlin Li, Xiaolin Duan, Jianglin Xia\",\"doi\":\"10.1016/j.future.2024.107681\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>Apache Flink has become one of the highly regarded streaming computing frameworks with its excellent advantages of high throughput, low latency, and high reliability. However, the default task scheduling policy follows the first-come-first-served principle, which fails to fully consider the differences in energy efficiency and resource loading of nodes in heterogeneous clusters and may lead to high energy consumption and uneven load distribution when executing jobs. To solve this problem, this paper proposes a two-tier coordinated load balancing and energy-saving scheduling optimization strategy. First, we construct an energy efficiency model based on Service Level Agreements (SLA) and design an Energy-Saving Scheduling Algorithm (ESSA) based on this model, aiming to reduce the energy consumption of Flink clusters when executing jobs. This ESSA algorithm integrally considers the effects of two SLA performance metrics including node response time and throughput on node energy consumption, as well as the differences in the energy efficiencies of different nodes in heterogeneous clusters. Second, in order to solve the load imbalance problem that may be caused by Flink’s default scheduling policy, an Energy-Aware Two-Tier Coordinated Load Balancing algorithm (TTCLB-EA) is proposed, which optimizes the cluster load at both the inter-node and intra-node levels through task based on energy efficiency priorities. Experimental results show that compared with the default scheduling strategy, round-robin scheduling strategy, and St-Stream, the proposed algorithm improves about 14.59%, 12.75%, and 7.32% in load balancing, while saving about 14.52%, 10.54%, and 7.58% in energy consumption, respectively. The proposed algorithms not only enhance the performance of the Flink cluster but also help to reduce energy consumption and achieve more efficient resource utilization.</div></div>\",\"PeriodicalId\":55132,\"journal\":{\"name\":\"Future Generation Computer Systems-The International Journal of Escience\",\"volume\":\"166 \",\"pages\":\"Article 107681\"},\"PeriodicalIF\":6.2000,\"publicationDate\":\"2025-01-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Future Generation Computer Systems-The International Journal of Escience\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0167739X24006459\",\"RegionNum\":2,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, THEORY & METHODS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Future Generation Computer Systems-The International Journal of Escience","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0167739X24006459","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, THEORY & METHODS","Score":null,"Total":0}
Energy-aware scheduling and two-tier coordinated load balancing for streaming applications in apache flink
Apache Flink has become one of the highly regarded streaming computing frameworks with its excellent advantages of high throughput, low latency, and high reliability. However, the default task scheduling policy follows the first-come-first-served principle, which fails to fully consider the differences in energy efficiency and resource loading of nodes in heterogeneous clusters and may lead to high energy consumption and uneven load distribution when executing jobs. To solve this problem, this paper proposes a two-tier coordinated load balancing and energy-saving scheduling optimization strategy. First, we construct an energy efficiency model based on Service Level Agreements (SLA) and design an Energy-Saving Scheduling Algorithm (ESSA) based on this model, aiming to reduce the energy consumption of Flink clusters when executing jobs. This ESSA algorithm integrally considers the effects of two SLA performance metrics including node response time and throughput on node energy consumption, as well as the differences in the energy efficiencies of different nodes in heterogeneous clusters. Second, in order to solve the load imbalance problem that may be caused by Flink’s default scheduling policy, an Energy-Aware Two-Tier Coordinated Load Balancing algorithm (TTCLB-EA) is proposed, which optimizes the cluster load at both the inter-node and intra-node levels through task based on energy efficiency priorities. Experimental results show that compared with the default scheduling strategy, round-robin scheduling strategy, and St-Stream, the proposed algorithm improves about 14.59%, 12.75%, and 7.32% in load balancing, while saving about 14.52%, 10.54%, and 7.58% in energy consumption, respectively. The proposed algorithms not only enhance the performance of the Flink cluster but also help to reduce energy consumption and achieve more efficient resource utilization.
期刊介绍:
Computing infrastructures and systems are constantly evolving, resulting in increasingly complex and collaborative scientific applications. To cope with these advancements, there is a growing need for collaborative tools that can effectively map, control, and execute these applications.
Furthermore, with the explosion of Big Data, there is a requirement for innovative methods and infrastructures to collect, analyze, and derive meaningful insights from the vast amount of data generated. This necessitates the integration of computational and storage capabilities, databases, sensors, and human collaboration.
Future Generation Computer Systems aims to pioneer advancements in distributed systems, collaborative environments, high-performance computing, and Big Data analytics. It strives to stay at the forefront of developments in grids, clouds, and the Internet of Things (IoT) to effectively address the challenges posed by these wide-area, fully distributed sensing and computing systems.