The most popular in-memory computing framework --- Spark --- has a number of performance-critical configuration parameters. Manually tuning these parameters for optimized performance is not practical because the parameter tuning space is huge. Searching algorithms such as genetic algorithm can be used to automatically search the optimal configurations. However, there are several such algorithms and it is unclear which one is better in the case of Spark configuration parameter tuning. To address this issue, we experimentally compare two searching algorithms --- the Genetic Algorithm (GA) and the Particle Swarm Optimization (PSO) --- in searching the optimal configurations for Spark applications. We made several interesting observations. For one, PSO converges 2x faster than GA but the performance tuned by the configuration parameters found by PSO is slightly poorer than that by GA. Second, PSO shows better scalability with respect to the number of configuration parameters than GA. Finally, we find PSO is more robust than GA across different searching processes. Based on these observations, we recommend one to use PSO in Spark performance tuning context.
{"title":"An Experimental Comparison Between Genetic Algorithm and Particle Swarm Optimization in Spark Performance Tuning","authors":"Yuzhao Wang, Qixiao Liu, Junqing Yu, Zhibin Yu","doi":"10.1145/3129457.3129494","DOIUrl":"https://doi.org/10.1145/3129457.3129494","url":null,"abstract":"The most popular in-memory computing framework --- Spark --- has a number of performance-critical configuration parameters. Manually tuning these parameters for optimized performance is not practical because the parameter tuning space is huge. Searching algorithms such as genetic algorithm can be used to automatically search the optimal configurations. However, there are several such algorithms and it is unclear which one is better in the case of Spark configuration parameter tuning. To address this issue, we experimentally compare two searching algorithms --- the Genetic Algorithm (GA) and the Particle Swarm Optimization (PSO) --- in searching the optimal configurations for Spark applications. We made several interesting observations. For one, PSO converges 2x faster than GA but the performance tuned by the configuration parameters found by PSO is slightly poorer than that by GA. Second, PSO shows better scalability with respect to the number of configuration parameters than GA. Finally, we find PSO is more robust than GA across different searching processes. Based on these observations, we recommend one to use PSO in Spark performance tuning context.","PeriodicalId":345943,"journal":{"name":"Proceedings of the first Workshop on Emerging Technologies for software-defined and reconfigurable hardware-accelerated Cloud Datacenters","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132249794","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the first Workshop on Emerging Technologies for software-defined and reconfigurable hardware-accelerated Cloud Datacenters","authors":"","doi":"10.1145/3129457","DOIUrl":"https://doi.org/10.1145/3129457","url":null,"abstract":"","PeriodicalId":345943,"journal":{"name":"Proceedings of the first Workshop on Emerging Technologies for software-defined and reconfigurable hardware-accelerated Cloud Datacenters","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131630729","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：