Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00050
Srikanth Bethu, B. Babu, S. G. Rao, R. Florence
Knowledge discovery and Data mining plays a major role in computational intensive tasks with high range of applications. With the increase of volume and dimension of data, the distributed features perform operations in a reasonable period. MapReduce programming is suitable for distributed large scale data processing that provides different ways of solutions to the same problem, that (one) has particular constraints and properties. In this paper, we give comparative analysis and its approaches for computing KNN on MapReduce[1] theoretically and experimental evaluation. Load balancing, accuracy and complexity are analyzed on each step of data preprocessing, data partitioning and computation. The experiment results in this are produced by using variety of datasets. Time and Space complexity are analyzed periodically on each dataset and gives new advantages and short comings that are discussed for each algorithm. Finally this paper can be used as a reference material to handle KNN [2] based problems in the idea of Mapreducing in Big Data.
{"title":"Map Reduce by K-Nearest Neighbor Joins","authors":"Srikanth Bethu, B. Babu, S. G. Rao, R. Florence","doi":"10.1109/CYBERC.2018.00050","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00050","url":null,"abstract":"Knowledge discovery and Data mining plays a major role in computational intensive tasks with high range of applications. With the increase of volume and dimension of data, the distributed features perform operations in a reasonable period. MapReduce programming is suitable for distributed large scale data processing that provides different ways of solutions to the same problem, that (one) has particular constraints and properties. In this paper, we give comparative analysis and its approaches for computing KNN on MapReduce[1] theoretically and experimental evaluation. Load balancing, accuracy and complexity are analyzed on each step of data preprocessing, data partitioning and computation. The experiment results in this are produced by using variety of datasets. Time and Space complexity are analyzed periodically on each dataset and gives new advantages and short comings that are discussed for each algorithm. Finally this paper can be used as a reference material to handle KNN [2] based problems in the idea of Mapreducing in Big Data.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133840334","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00073
Zhongke Li, H. Zhao, Xiaoping Su, C. Wan
A novel unidirectional authentication method is proposed to solve the problems of access security and to protect the privacy of the tag in RFID ecosystems on basis of asymmetric cryptography. It provides better security with shorter length of secret key and lower computational cost in contrast to other methods of asymmetric cryptographies. This method implements one-way authentication from the reader-writer to the tag, and allows the tag to update its private key to improve the security of RFID authentication. This method also allows the tag to update its own name, which can keep the attackers from tracking the tag so as to solve the privacy protection of the tags.
{"title":"Asymmetric Cryptography Based Unidirectional Authentication Method for RFID","authors":"Zhongke Li, H. Zhao, Xiaoping Su, C. Wan","doi":"10.1109/CYBERC.2018.00073","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00073","url":null,"abstract":"A novel unidirectional authentication method is proposed to solve the problems of access security and to protect the privacy of the tag in RFID ecosystems on basis of asymmetric cryptography. It provides better security with shorter length of secret key and lower computational cost in contrast to other methods of asymmetric cryptographies. This method implements one-way authentication from the reader-writer to the tag, and allows the tag to update its private key to improve the security of RFID authentication. This method also allows the tag to update its own name, which can keep the attackers from tracking the tag so as to solve the privacy protection of the tags.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"1 11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134016282","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00084
Zizhe Wang, L. Qi, Tie Yun, Yi Ding, Yang Bai
The rapid development of Unmanned Aerial Vehicle (UAV) technology, also known as drones, has made people benefit in many ways, but it also brings privacy and security issues, such as appeared at private place, airports, prisons, etc. Therefore, the detection of drones in a specific area is crucial. Video detection is an effective method with various advantages. In this paper, we have used the background subtraction method to detect the moving object in the video sequence which recorded by static cameras, then extracted the global Fourier descriptors and the local HOG features of the moving object images. Finally, the FD+HOG features have been sent to the SVM classifier for classification and recognition. Our algorithm is simple and efficient, an overall with 98% accuracy was obtained in our data set.
{"title":"Drone Detection Based on FD-HOG Descriptor","authors":"Zizhe Wang, L. Qi, Tie Yun, Yi Ding, Yang Bai","doi":"10.1109/CYBERC.2018.00084","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00084","url":null,"abstract":"The rapid development of Unmanned Aerial Vehicle (UAV) technology, also known as drones, has made people benefit in many ways, but it also brings privacy and security issues, such as appeared at private place, airports, prisons, etc. Therefore, the detection of drones in a specific area is crucial. Video detection is an effective method with various advantages. In this paper, we have used the background subtraction method to detect the moving object in the video sequence which recorded by static cameras, then extracted the global Fourier descriptors and the local HOG features of the moving object images. Finally, the FD+HOG features have been sent to the SVM classifier for classification and recognition. Our algorithm is simple and efficient, an overall with 98% accuracy was obtained in our data set.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134096238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00023
E. Basan, M. Medvedev, Stanislav Teterevyatnikov
this research is devoted to the analysis of the effect of denial of service attacks on a set of specific parameters of mobile robot network nodes. This analysis made it possible to understand what parameters are appropriate to evaluate for detecting attacks. Another goal of the study was to develop a methodology for instrumental verification of the security of a group of mobile robots. For the testing of mobile robots, a device was developed based on a single-board computer, the Raspberry Pi. This device is a universal portable test device for wireless networks, both from external and internal networks. As a result of the experimental study, the set of parameters was influenced by the attacks: CPU, RAM (MB), CPU temp (°C), Power consumption (mAh), Number of packages. Analyzing these metrics of the attacker and the victim can be found. Furthermore, the effectiveness of the attack can be assessed. As a result of the research, a set of tools were identified that are best suited for the implementation of attacks.
{"title":"Analysis of the Impact of Denial of Service Attacks on the Group of Robots","authors":"E. Basan, M. Medvedev, Stanislav Teterevyatnikov","doi":"10.1109/CYBERC.2018.00023","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00023","url":null,"abstract":"this research is devoted to the analysis of the effect of denial of service attacks on a set of specific parameters of mobile robot network nodes. This analysis made it possible to understand what parameters are appropriate to evaluate for detecting attacks. Another goal of the study was to develop a methodology for instrumental verification of the security of a group of mobile robots. For the testing of mobile robots, a device was developed based on a single-board computer, the Raspberry Pi. This device is a universal portable test device for wireless networks, both from external and internal networks. As a result of the experimental study, the set of parameters was influenced by the attacks: CPU, RAM (MB), CPU temp (°C), Power consumption (mAh), Number of packages. Analyzing these metrics of the attacker and the victim can be found. Furthermore, the effectiveness of the attack can be assessed. As a result of the research, a set of tools were identified that are best suited for the implementation of attacks.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130848573","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00038
Zhenzhou Wang, Wei Huo, Pingping Yu, Lin Qi, Ning Cao
Internet of Things(IOT) technology provides sufficient information technology and necessary decision-making basis for intelligent transportation. Internet of vehicles(IOV) is based on the architecture of the perception layer, network layer, and application layer. This article focuses on the tail light status in the perception layer of automotive active safety systems and proposes a method of vehicle taillamp detection and recognition based on active contour extraction and color feature analysis to solve the problem of information perception against the front vehicle. First, the front vehicles are detected and extracted by convolutional neural network and active contour models. Second, it analyzes the characteristics of taillamp pair and then segment the correct taillamp pair by color space conversion and location correlation principle. On this basis, the histograms distribution of taillamp are analyzed in different states and complete the recognition of taillamp. The experimental results show that the method can accurately identify the taillamp signal of the front vehicle in the daytime and effectively reduce the rate of false judgement caused by light interference.
物联网技术为智能交通提供了充分的信息技术和必要的决策依据。车联网(Internet of vehicle, IOV)是基于感知层、网络层和应用层的架构。本文以汽车主动安全系统感知层中的尾灯状态为研究对象，提出了一种基于主动轮廓提取和颜色特征分析的汽车尾灯检测与识别方法，解决了对前车的信息感知问题。首先，利用卷积神经网络和主动轮廓模型对前方车辆进行检测和提取;其次，分析尾灯对的特征，利用颜色空间转换和位置相关原理分割出正确的尾灯对;在此基础上，分析不同状态下尾灯的直方图分布，完成尾灯的识别。实验结果表明，该方法能准确识别白天前车尾灯信号，有效降低光干扰造成的误判断率。
{"title":"Research on Vehicle Taillight Detection and Semantic Recognition Based on Internet of Vehicle","authors":"Zhenzhou Wang, Wei Huo, Pingping Yu, Lin Qi, Ning Cao","doi":"10.1109/CYBERC.2018.00038","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00038","url":null,"abstract":"Internet of Things(IOT) technology provides sufficient information technology and necessary decision-making basis for intelligent transportation. Internet of vehicles(IOV) is based on the architecture of the perception layer, network layer, and application layer. This article focuses on the tail light status in the perception layer of automotive active safety systems and proposes a method of vehicle taillamp detection and recognition based on active contour extraction and color feature analysis to solve the problem of information perception against the front vehicle. First, the front vehicles are detected and extracted by convolutional neural network and active contour models. Second, it analyzes the characteristics of taillamp pair and then segment the correct taillamp pair by color space conversion and location correlation principle. On this basis, the histograms distribution of taillamp are analyzed in different states and complete the recognition of taillamp. The experimental results show that the method can accurately identify the taillamp signal of the front vehicle in the daytime and effectively reduce the rate of false judgement caused by light interference.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129350196","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00052
Yifeng Cui, Weifeng Lv, Qing Wang, Bowen Du
The rise and wide use of urban shared bicycles brings great convenience to residents' daily trips. However, as the number of shared bicycles continues to increase in the city, more and more shared bicycles are being allocated unreasonably, which has a great impact on the public transportation and city appearance. To solve this problem, in this paper, we focus on the recommendation of the number of shared bicycles near the subway station and take the volume of the station's outbound passenger flow as the potential demand to study. Along this line, we first collect and investigate the subway traffic data which are gathered by automated fare collection(AFC) systems and some related feature data as the basic data source for the demand prediction. Second, upon these data, we develop a novel passenger flow forecast model based on an advanced Xgboost method and the idea of sliding window. Combined with the predicted results, we then design a method to transform the passenger flow results into an usage demand index and recommend a suitable number of shared bicycles for a specific subway station by it. Finally, we conduct a series of experiments with a variety of real-world data, and the results demonstrate our model can achieve a better performance and give us a reasonable quantity recommendation.
{"title":"Usage Demand Forecast and Quantity Recommendation for Urban Shared Bicycles","authors":"Yifeng Cui, Weifeng Lv, Qing Wang, Bowen Du","doi":"10.1109/CYBERC.2018.00052","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00052","url":null,"abstract":"The rise and wide use of urban shared bicycles brings great convenience to residents' daily trips. However, as the number of shared bicycles continues to increase in the city, more and more shared bicycles are being allocated unreasonably, which has a great impact on the public transportation and city appearance. To solve this problem, in this paper, we focus on the recommendation of the number of shared bicycles near the subway station and take the volume of the station's outbound passenger flow as the potential demand to study. Along this line, we first collect and investigate the subway traffic data which are gathered by automated fare collection(AFC) systems and some related feature data as the basic data source for the demand prediction. Second, upon these data, we develop a novel passenger flow forecast model based on an advanced Xgboost method and the idea of sliding window. Combined with the predicted results, we then design a method to transform the passenger flow results into an usage demand index and recommend a suitable number of shared bicycles for a specific subway station by it. Finally, we conduct a series of experiments with a variety of real-world data, and the results demonstrate our model can achieve a better performance and give us a reasonable quantity recommendation.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129358481","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00097
Shijun Chen, Dawei Chen, Yuanyuan Wang, Xinxin Liu
In unfamiliar complicated high-rise building, users not only have to understand the 2D plane location information of floor, but also need to obtain the Z axis information of floor. According to this requirement, we proposed a 3D indoor localization mechanism based on multi-sensor, which integrates the air pressure examination and geomagnetic examination information to assist inertial navigation for calibration, and conduct integrated processing of examined information. The mechanism is based on the dynamic air pressure floor discrimination model, and according to the actual current floor, average time spent by passing each floor and the height of each floor, determine the floor of user in a real-time manner; according to the standard geomagnetic information data of floor, compute the geomagnetic coordinates of current location based on the geomagnetic realtime localization model, and obtain the floor of personnel and the plane location on the floor in a real-time manner. During the localization process of mechanism, it does not require additional hardware equipment or network signal, but only needs the air pressure gauge sensor and geomagnetic sensor to complete the indoor 3D real-time localization of building, which is convenient in practical application with broad adaptability.
{"title":"3D Indoor Localization Mechanism Based on Multiple Sensors","authors":"Shijun Chen, Dawei Chen, Yuanyuan Wang, Xinxin Liu","doi":"10.1109/CYBERC.2018.00097","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00097","url":null,"abstract":"In unfamiliar complicated high-rise building, users not only have to understand the 2D plane location information of floor, but also need to obtain the Z axis information of floor. According to this requirement, we proposed a 3D indoor localization mechanism based on multi-sensor, which integrates the air pressure examination and geomagnetic examination information to assist inertial navigation for calibration, and conduct integrated processing of examined information. The mechanism is based on the dynamic air pressure floor discrimination model, and according to the actual current floor, average time spent by passing each floor and the height of each floor, determine the floor of user in a real-time manner; according to the standard geomagnetic information data of floor, compute the geomagnetic coordinates of current location based on the geomagnetic realtime localization model, and obtain the floor of personnel and the plane location on the floor in a real-time manner. During the localization process of mechanism, it does not require additional hardware equipment or network signal, but only needs the air pressure gauge sensor and geomagnetic sensor to complete the indoor 3D real-time localization of building, which is convenient in practical application with broad adaptability.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127287091","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00040
Deshun Zhao, Kun Zhao, Jun Yang, Wei-yi Ma, Bo Yu
In this paper, we learn the statistical properties of Chinese character networks within the framework of complex network theory. We construct two Chinese character networks (CCN1 and CCN2) based on 7000 Modern Chinese characters in common use from different respects. In CCN1, a link between two nodes (each node correspond to one Chinese character) represents that one node is a component part of the other. In CCN2, a link represents that the two nodes can be combined into a new node. It is found that, both in CCN1 and CCN2, there are some common features such as small-world effect, scale-free behavior, hierarchical organization and disassortative mixing. These results effectively explain the origin and formation of Chinese characters, which is absolutely helpful in understanding and using Chinese characters.
{"title":"Structural Exploration of Chinese Character Combination Networks","authors":"Deshun Zhao, Kun Zhao, Jun Yang, Wei-yi Ma, Bo Yu","doi":"10.1109/CYBERC.2018.00040","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00040","url":null,"abstract":"In this paper, we learn the statistical properties of Chinese character networks within the framework of complex network theory. We construct two Chinese character networks (CCN1 and CCN2) based on 7000 Modern Chinese characters in common use from different respects. In CCN1, a link between two nodes (each node correspond to one Chinese character) represents that one node is a component part of the other. In CCN2, a link represents that the two nodes can be combined into a new node. It is found that, both in CCN1 and CCN2, there are some common features such as small-world effect, scale-free behavior, hierarchical organization and disassortative mixing. These results effectively explain the origin and formation of Chinese characters, which is absolutely helpful in understanding and using Chinese characters.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129084869","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00081
Lanfei Wang, Jun Guo, Qu Wang (王曲）, Jiangming Kan
Robot path planning is the key to robot navigation. We implemented the robot path planning based on ant colony algorithm and genetic algorithm, and proposed simulated annealing genetic algorithm. Under the condition that there is not much difference in running time (within 3 seconds), planning results of different terrains, start and end points based on ant colony algorithm(with 200 iterations)and simulated annealing genetic algorithm show that, the optimal path outputted by simulated annealing genetic algorithm is better than the optimal path outputted by ant colony algorithm in terms of avoiding obstacles; The simulated annealing genetic algorithm has shorter average optimal path length than ant colony algorithm in multiple tests, the average path length is reduced by 6.85%.
{"title":"Ground Robot Path Planning Based on Simulated Annealing Genetic Algorithm","authors":"Lanfei Wang, Jun Guo, Qu Wang (王曲）, Jiangming Kan","doi":"10.1109/CYBERC.2018.00081","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00081","url":null,"abstract":"Robot path planning is the key to robot navigation. We implemented the robot path planning based on ant colony algorithm and genetic algorithm, and proposed simulated annealing genetic algorithm. Under the condition that there is not much difference in running time (within 3 seconds), planning results of different terrains, start and end points based on ant colony algorithm(with 200 iterations)and simulated annealing genetic algorithm show that, the optimal path outputted by simulated annealing genetic algorithm is better than the optimal path outputted by ant colony algorithm in terms of avoiding obstacles; The simulated annealing genetic algorithm has shorter average optimal path length than ant colony algorithm in multiple tests, the average path length is reduced by 6.85%.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130012740","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00025
Suzhen Wang, Lu Zhang, Yanpiao Zhang, Ning Cao
The data collected by the Internet of Things (IOT) technology is becoming larger and larger, and the traditional data processing methods have encountered tremendous challenges. Spark, as a memory-based distributed computing framework, provides support for the data processing of the IOT. Load balancing is an important indicator to measure the performance of Spark computing. The load balancing strategy of Spark cluster only takes into account the locality of data, and neglects the computing capability and resource utilization of each node, which is prone to load unbalance and affecting the IOT data processing efficiency. Aiming at this issue, this paper optimizes and improves the current load balancing strategy of Spark based on the computing performance of each node in the Spark cluster, and proposes a task execution node assignment algorithm based on genetic algorithm and particle swarm optimization (TENAA). Experiments show that, compared with the Spark load balancing strategy, the load balancing strategy proposed in this paper has a significant increase both in load deviation and task completion time.
{"title":"Spark Load Balancing Strategy Optimization Based on Internet of Things","authors":"Suzhen Wang, Lu Zhang, Yanpiao Zhang, Ning Cao","doi":"10.1109/CYBERC.2018.00025","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00025","url":null,"abstract":"The data collected by the Internet of Things (IOT) technology is becoming larger and larger, and the traditional data processing methods have encountered tremendous challenges. Spark, as a memory-based distributed computing framework, provides support for the data processing of the IOT. Load balancing is an important indicator to measure the performance of Spark computing. The load balancing strategy of Spark cluster only takes into account the locality of data, and neglects the computing capability and resource utilization of each node, which is prone to load unbalance and affecting the IOT data processing efficiency. Aiming at this issue, this paper optimizes and improves the current load balancing strategy of Spark based on the computing performance of each node in the Spark cluster, and proposes a task execution node assignment algorithm based on genetic algorithm and particle swarm optimization (TENAA). Experiments show that, compared with the Spark load balancing strategy, the load balancing strategy proposed in this paper has a significant increase both in load deviation and task completion time.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129655850","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：