Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00029
A. Valenzuela, Mauro Schwab, Adolfo A. Silnik, Alfredo F. Debattista, Roberto A. Kiessling
We present the development and evaluation of a basic building block for a future wireless sensor network for agriculture monitoring in Argentina. The module consists of a compact battery-powered wireless sensor node capable of monitoring the ambient air parameters of temperature, humidity, gas and air pressure in the agriculture industry of Argentina's Pampa region. Further in-and outputs allow the system to be extended flexibly by adding more sensors. Throughout the development, a simple, low-cost and open-source-based approach together with a lightweight communication protocol was pursued. The sensor nodes cover ranges of over 400 metres and can be operated on two AAA alkaline batteries for several years. Detailed current consumption values, range limits and battery life estimates are presented.
{"title":"Low Power Wireless Sensor Node Platform for Agriculture Monitoring in Argentina","authors":"A. Valenzuela, Mauro Schwab, Adolfo A. Silnik, Alfredo F. Debattista, Roberto A. Kiessling","doi":"10.1109/CYBERC.2018.00029","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00029","url":null,"abstract":"We present the development and evaluation of a basic building block for a future wireless sensor network for agriculture monitoring in Argentina. The module consists of a compact battery-powered wireless sensor node capable of monitoring the ambient air parameters of temperature, humidity, gas and air pressure in the agriculture industry of Argentina's Pampa region. Further in-and outputs allow the system to be extended flexibly by adding more sensors. Throughout the development, a simple, low-cost and open-source-based approach together with a lightweight communication protocol was pursued. The sensor nodes cover ranges of over 400 metres and can be operated on two AAA alkaline batteries for several years. Detailed current consumption values, range limits and battery life estimates are presented.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"50a 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120858343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00034
Suzhen Wang, Yanpiao Zhang, Lu Zhang, Ning Cao
The development of the Internet of things(IOT) has produced huge diversity of data. In view of the need for massive data processing and application by the Internet of things, the big data service platform arises at the historic moment. Our paper mainly studies the optimization of the big data platform in the background of Internet of things, and combines the Internet of things with the big data platform–Spark. In this paper, we proposed an improved Spark job scheduling scheme based on the genetic and tabu search algorithm. By optimizing the job scheduling algorithm of Spark, it will provide the better technical support for data processing in the Internet of things.
{"title":"The Optimization of Big Data Platform Under the Internet of Things","authors":"Suzhen Wang, Yanpiao Zhang, Lu Zhang, Ning Cao","doi":"10.1109/CYBERC.2018.00034","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00034","url":null,"abstract":"The development of the Internet of things(IOT) has produced huge diversity of data. In view of the need for massive data processing and application by the Internet of things, the big data service platform arises at the historic moment. Our paper mainly studies the optimization of the big data platform in the background of Internet of things, and combines the Internet of things with the big data platform–Spark. In this paper, we proposed an improved Spark job scheduling scheme based on the genetic and tabu search algorithm. By optimizing the job scheduling algorithm of Spark, it will provide the better technical support for data processing in the Internet of things.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121071434","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In multi-sensor systems, data fusion is one of the key technologies for solving information diversification in wireless sensor networks. Data fusion is a process of information processing to automatically analyze and synthesize data collected by multiple sensors under certain rules to complete the required decisions or tasks, including information fusion, feature fusion, relationship fusion and decision fusion. It extends the lifespan of wireless sensor networks and improves data accuracy. It is generally considered that data fusion is an integrated process of information processing. It is generally considered that data fusion is a process of information synthesis and processing, making various information and data detected, correlated, estimated, and synthesized at multiple levels and from many aspects to obtain accurate and complete information. There are many methods for sensor data fusion, such as Bayesian method, D-S method, neural network, fuzzy reasoning, genetic algorithm, deep learning, etc. This article focuses on the application, analysis and comparison of probabilistic statistical methods in multi-sensor data fusion. The data fusion methods of probability statistics are divided into three categories: data fusion method based on estimation theory, data fusion method based on regression theory, and data fusion method based on information theory. This article just has a simple analysis on the three types from the perspective of theory and has a detailed analysis on the core Bayesian fusion in probability statistics.
{"title":"Research on Probability Statistics Method for Multi-sensor Data Fusion","authors":"Maoli Ran, Xiangyu Bai, Fangshuo Xin, Yaping Xiang","doi":"10.1109/CYBERC.2018.00079","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00079","url":null,"abstract":"In multi-sensor systems, data fusion is one of the key technologies for solving information diversification in wireless sensor networks. Data fusion is a process of information processing to automatically analyze and synthesize data collected by multiple sensors under certain rules to complete the required decisions or tasks, including information fusion, feature fusion, relationship fusion and decision fusion. It extends the lifespan of wireless sensor networks and improves data accuracy. It is generally considered that data fusion is an integrated process of information processing. It is generally considered that data fusion is a process of information synthesis and processing, making various information and data detected, correlated, estimated, and synthesized at multiple levels and from many aspects to obtain accurate and complete information. There are many methods for sensor data fusion, such as Bayesian method, D-S method, neural network, fuzzy reasoning, genetic algorithm, deep learning, etc. This article focuses on the application, analysis and comparison of probabilistic statistical methods in multi-sensor data fusion. The data fusion methods of probability statistics are divided into three categories: data fusion method based on estimation theory, data fusion method based on regression theory, and data fusion method based on information theory. This article just has a simple analysis on the three types from the perspective of theory and has a detailed analysis on the core Bayesian fusion in probability statistics.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121287226","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00065
G. Han, Linxin Zhang, X. Mu, Dalong Zhang, Yi Sun
A two-user uplink massive MIMO system is considered in this paper, where each user has a single antenna and the base station (BS) is equipped with a large number of antennas. It is assumed that the small scale channel fading is Rayleigh fading and the channel fading coefficients keep quasi-static in two consecutive slots, and then, change to other values independently in the next two slots. For such a massive MIMO uplink system, a QAM division based space-time modulation scheme is proposed to execute the simultaneous communication of the two users with the same frequency, and four detectors are given to adapt to different conditions. In addition, the channel coefficients can also be figured out after the signals are correctly detected. Computer simulations demonstrate that the proposed scheme performs well and need less than 100 BS antennas to make the average BER below 10^3.
{"title":"QAM Division Based Space-Time Modulation for Two-User Uplink Massive MIMO Systems","authors":"G. Han, Linxin Zhang, X. Mu, Dalong Zhang, Yi Sun","doi":"10.1109/CYBERC.2018.00065","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00065","url":null,"abstract":"A two-user uplink massive MIMO system is considered in this paper, where each user has a single antenna and the base station (BS) is equipped with a large number of antennas. It is assumed that the small scale channel fading is Rayleigh fading and the channel fading coefficients keep quasi-static in two consecutive slots, and then, change to other values independently in the next two slots. For such a massive MIMO uplink system, a QAM division based space-time modulation scheme is proposed to execute the simultaneous communication of the two users with the same frequency, and four detectors are given to adapt to different conditions. In addition, the channel coefficients can also be figured out after the signals are correctly detected. Computer simulations demonstrate that the proposed scheme performs well and need less than 100 BS antennas to make the average BER below 10^3.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126581357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00060
Weibing Long, Kunli Zhang, Hongchao Ma, Donghui Yue, Zhuang Lei
The data-driven medical health information processing has become a new development direction, especially the auxiliary diagnosis based on the electronic medical records (EMRs), which is of great significance to improve population health. In this paper, to obtain excellent obstetric auxiliary diagnostic results, the Chinese obstetric EMRs is analyzed and processed, and finally the auxiliary diagnosis task is transformed into a multi-label classification problem. Moreover, two effective global error functions are proposed by enhancing pairwise labels discrimination to improve the Backpropagation for Multi-label Learning (BP-MLL) that depends on the neural network model. The experiment results of some public multi-label datasets and the Chinese obstetric dataset show that the two error functions have better overall performance compared with BP-MLL original error function and some well-established multi-label learning algorithms.
{"title":"Neural Network Multi-label Learning Based on Enhancing Pairwise Labels Discrimination for Obstetric Auxiliary Diagnosis","authors":"Weibing Long, Kunli Zhang, Hongchao Ma, Donghui Yue, Zhuang Lei","doi":"10.1109/CYBERC.2018.00060","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00060","url":null,"abstract":"The data-driven medical health information processing has become a new development direction, especially the auxiliary diagnosis based on the electronic medical records (EMRs), which is of great significance to improve population health. In this paper, to obtain excellent obstetric auxiliary diagnostic results, the Chinese obstetric EMRs is analyzed and processed, and finally the auxiliary diagnosis task is transformed into a multi-label classification problem. Moreover, two effective global error functions are proposed by enhancing pairwise labels discrimination to improve the Backpropagation for Multi-label Learning (BP-MLL) that depends on the neural network model. The experiment results of some public multi-label datasets and the Chinese obstetric dataset show that the two error functions have better overall performance compared with BP-MLL original error function and some well-established multi-label learning algorithms.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129256739","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00088
Yuewei Jia, D. Qi, Yan Shi, Jianghua Li, Zhuyun Chen, Xiaokai Zhang
In this letter, we investigate the loading analysis of digital channelized satellite communication (SATCOM) system operating in frequency-division multiple access (FDMA) mode, multi-frequency time-division multiple access (MF-TDMA) mode and overlay combined multiple access (OCMA) mode. In an effort to enhance link stability of these systems, a max-min optimization objective for link-margin degree (LMD) is firstly established under the constraint that all loading links are supportable. Basing on fully using the power of transmitting terminals and directly reducing the difference among all LMDs to enhance the minimum LMD as much as possible, an effective maximum value back-off searching (MVBS) algorithm is proposed for the optimization model. Finally, numerical simulations reveal that, under the constraint of link supportability, the proposed algorithm brings about considerable improvement of the minimum LMD for enhancing link stability, which effectively demonstrates the correctness of our scheme.
{"title":"Loading Analysis of Channelized SATCOM System with Link-Margin Degree Optimization","authors":"Yuewei Jia, D. Qi, Yan Shi, Jianghua Li, Zhuyun Chen, Xiaokai Zhang","doi":"10.1109/CYBERC.2018.00088","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00088","url":null,"abstract":"In this letter, we investigate the loading analysis of digital channelized satellite communication (SATCOM) system operating in frequency-division multiple access (FDMA) mode, multi-frequency time-division multiple access (MF-TDMA) mode and overlay combined multiple access (OCMA) mode. In an effort to enhance link stability of these systems, a max-min optimization objective for link-margin degree (LMD) is firstly established under the constraint that all loading links are supportable. Basing on fully using the power of transmitting terminals and directly reducing the difference among all LMDs to enhance the minimum LMD as much as possible, an effective maximum value back-off searching (MVBS) algorithm is proposed for the optimization model. Finally, numerical simulations reveal that, under the constraint of link supportability, the proposed algorithm brings about considerable improvement of the minimum LMD for enhancing link stability, which effectively demonstrates the correctness of our scheme.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"108 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129092905","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00016
Guangxia Xu, G. Gao, Mengxiao Hu
Micro-blog has become an important information dissemination and exchange platform in people's social lives. Massive micro-blog data contains a large number of valuable information, but the micro-blog platform appears to have a lot of spam behavior problems in recent years; behavior consistent with spammers and spam micro-blogs. The spam not only affects the impact of micro-blog's data mining and decision analysis, but also seriously affects the healthy development of micro-blog platform and user experience. In this paper, a new spammer detection method based on fuzzy multi-class support vector machines (FMCSVM) is proposed in micro-blog, it combines the SVM multi-class classifier with the fuzzy mathematics theory in spammer detection. Current researches on micro-blog spammers is to analyze the characteristics of the global spammers, so that the strength of these analyses is not enough, and these researches lack the feature analysis for a certain type spammer. As a result, this will enable the spammer to escape the spam detection system. In this paper, we divide spammers into three categories by analyzing the features of micro-blog spammers, and then construct one-versus-rest SVM multi-class classifier. The fuzzy clustering method is used to deal with the mixed samples generated by the multi class classifier, and the combination classifier is obtained, which improves the detection accuracy.
{"title":"Detecting Spammer on Micro-blogs Base on Fuzzy Multi-class SVM","authors":"Guangxia Xu, G. Gao, Mengxiao Hu","doi":"10.1109/CYBERC.2018.00016","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00016","url":null,"abstract":"Micro-blog has become an important information dissemination and exchange platform in people's social lives. Massive micro-blog data contains a large number of valuable information, but the micro-blog platform appears to have a lot of spam behavior problems in recent years; behavior consistent with spammers and spam micro-blogs. The spam not only affects the impact of micro-blog's data mining and decision analysis, but also seriously affects the healthy development of micro-blog platform and user experience. In this paper, a new spammer detection method based on fuzzy multi-class support vector machines (FMCSVM) is proposed in micro-blog, it combines the SVM multi-class classifier with the fuzzy mathematics theory in spammer detection. Current researches on micro-blog spammers is to analyze the characteristics of the global spammers, so that the strength of these analyses is not enough, and these researches lack the feature analysis for a certain type spammer. As a result, this will enable the spammer to escape the spam detection system. In this paper, we divide spammers into three categories by analyzing the features of micro-blog spammers, and then construct one-versus-rest SVM multi-class classifier. The fuzzy clustering method is used to deal with the mixed samples generated by the multi class classifier, and the combination classifier is obtained, which improves the detection accuracy.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128854441","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00045
Yang Wang, Lan Li, Lei Fan
Data protection and the control of information flow are basic requirements for the security operation of enterprises or organizations. The data provenance of documents is a function that records the transmission of a specific document and provenance afterwards. As an important function of enterprise information security control, it has been confronted with the trouble of high management costs. Therefore, this paper attempts to recover the document content by proactively monitoring the internal traffic data of the enterprise and restore the document and find the parent document accurately through the proposed algorithm, thereby getting rid of the shackle of traditional document tracing. In order to ensure the flexibility and scalability of the streaming data restoration, this paper tries to build algorithm modules based on Flink, a streaming process platform, by migrating key computing services to its platform. In the process, the capture agent is set at the key node to collect traffic data, which is put into the stream processing system through the message queue. The stream processing system restores the file using document restoration algorithm, and finally the file is handed over to the feature extraction module. After the feature extraction module completes the file analysis, it is stored on file systems or structed data storage systems and waits for document tracking requests. The entire system solution achieved above and the daily business of the enterprise are completely seperated, while the load on the internal network flow is also very small. On the other hand, relying on the advantages of Flink's excellent distributed features, the experiments show that the data provenance results are satisfactory.
{"title":"A Binary Feature Extraction Based Data Provenance System Implemented on Flink Platform","authors":"Yang Wang, Lan Li, Lei Fan","doi":"10.1109/CYBERC.2018.00045","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00045","url":null,"abstract":"Data protection and the control of information flow are basic requirements for the security operation of enterprises or organizations. The data provenance of documents is a function that records the transmission of a specific document and provenance afterwards. As an important function of enterprise information security control, it has been confronted with the trouble of high management costs. Therefore, this paper attempts to recover the document content by proactively monitoring the internal traffic data of the enterprise and restore the document and find the parent document accurately through the proposed algorithm, thereby getting rid of the shackle of traditional document tracing. In order to ensure the flexibility and scalability of the streaming data restoration, this paper tries to build algorithm modules based on Flink, a streaming process platform, by migrating key computing services to its platform. In the process, the capture agent is set at the key node to collect traffic data, which is put into the stream processing system through the message queue. The stream processing system restores the file using document restoration algorithm, and finally the file is handed over to the feature extraction module. After the feature extraction module completes the file analysis, it is stored on file systems or structed data storage systems and waits for document tracking requests. The entire system solution achieved above and the daily business of the enterprise are completely seperated, while the load on the internal network flow is also very small. On the other hand, relying on the advantages of Flink's excellent distributed features, the experiments show that the data provenance results are satisfactory.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"103 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127979672","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Real-Time video analytics, whose applications range from safety, public security to smart cities, is a typical use case of Internet of Things (IoT). However, uploading the video stream to the cloud for analytics cannot meet the requirements of low latency and efficient bandwidth usage. Edge video analytics, which uploads the stream at the edge node, is a key to solve the abovementioned problem. This paper proposes an intelligent video analytics system on edge computing platform. Combining the edge computing and video analytics, this system can analyze the video stream by face recognition, indoor positioning, and semantic analytics in real time and archive the videos automatically. Specifically, applied in conference room, the video analytics system analyzes the conference room scenario and files the conference videos, which reduces the cost of manual recording and promotes the data sharing. The implementation results prove that our system can operate smoothly on the edge computing platform to provide real-time and efficient video analytics services.
{"title":"Design and Implementation of Video Analytics System Based on Edge Computing","authors":"Yuejun Chen, Yinghao Xie, Yihong Hu, Yaqiong Liu, Guochu Shou","doi":"10.1109/CYBERC.2018.00035","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00035","url":null,"abstract":"Real-Time video analytics, whose applications range from safety, public security to smart cities, is a typical use case of Internet of Things (IoT). However, uploading the video stream to the cloud for analytics cannot meet the requirements of low latency and efficient bandwidth usage. Edge video analytics, which uploads the stream at the edge node, is a key to solve the abovementioned problem. This paper proposes an intelligent video analytics system on edge computing platform. Combining the edge computing and video analytics, this system can analyze the video stream by face recognition, indoor positioning, and semantic analytics in real time and archive the videos automatically. Specifically, applied in conference room, the video analytics system analyzes the conference room scenario and files the conference videos, which reduces the cost of manual recording and promotes the data sharing. The implementation results prove that our system can operate smoothly on the edge computing platform to provide real-time and efficient video analytics services.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"79 290 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125965047","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2018-10-01DOI: 10.1109/CYBERC.2018.00094
Shumiao Yu, Weifeng Sun, Minghan Jia
With the development of geographic information system, digital earth and digital city play more and more important roles in life. The data generated by sensors or other edge nodes need to be collected by crawlers in the distributed systems in IoT, such as the GIS data in CyberGIS. In some edge networks, network operators have adopted methods to limit crawlers, such as blocking the request IP addresses, requiring logging in verification codes and other measures to avoid disturbance to servers. To collect data from web servers in these types of edge networks, a dynamic IP address based strategy DP-crawler is proposed to solve the anti-crawler strategies in the edge networks. DP-crawler can dynamic get proper IP addresses from a security-aware list and select the best available proxies. The security-aware list is designed to use the block-chain. Security and dynamic storage can be achieved by this method. DP-crawler is used to crawler webs, and the detailed information of Douban movies are obtained in the experiments. The experiment results show that the DP-Crawler can get more information by using the DP-Crawler.
{"title":"A Dynamic Proxy Based Crawler Strategy for Data Collection on CyberGIS","authors":"Shumiao Yu, Weifeng Sun, Minghan Jia","doi":"10.1109/CYBERC.2018.00094","DOIUrl":"https://doi.org/10.1109/CYBERC.2018.00094","url":null,"abstract":"With the development of geographic information system, digital earth and digital city play more and more important roles in life. The data generated by sensors or other edge nodes need to be collected by crawlers in the distributed systems in IoT, such as the GIS data in CyberGIS. In some edge networks, network operators have adopted methods to limit crawlers, such as blocking the request IP addresses, requiring logging in verification codes and other measures to avoid disturbance to servers. To collect data from web servers in these types of edge networks, a dynamic IP address based strategy DP-crawler is proposed to solve the anti-crawler strategies in the edge networks. DP-crawler can dynamic get proper IP addresses from a security-aware list and select the best available proxies. The security-aware list is designed to use the block-chain. Security and dynamic storage can be achieved by this method. DP-crawler is used to crawler webs, and the detailed information of Douban movies are obtained in the experiments. The experiment results show that the DP-Crawler can get more information by using the DP-Crawler.","PeriodicalId":282903,"journal":{"name":"2018 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC)","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128230690","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：