{"title":"Application and research of heuristic search algorithm in crawler field","authors":"Hui Liu, Kun Wang, Zikai Liu","doi":"10.1109/ITCA52113.2020.00016","DOIUrl":null,"url":null,"abstract":"Breadth first search algorithm is widely used in traditional crawler programs. This kind of search algorithm takes the way of blindly searching data, and does not make use of the characteristics of crawling data. When there are many websites, the crawler will be very time-consuming. Based on the above reasons, an adaptive heuristic search algorithm is designed. According to the relevance between the page and the topic and the related information of the link, the more likely link is selected for crawling, and the estimation function is modified according to the crawling result. Experiments show that the algorithm can significantly improve the efficiency of crawler than breadth first search algorithm.","PeriodicalId":103309,"journal":{"name":"2020 2nd International Conference on Information Technology and Computer Application (ITCA)","volume":"31 2","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2020 2nd International Conference on Information Technology and Computer Application (ITCA)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ITCA52113.2020.00016","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1
Abstract
Breadth first search algorithm is widely used in traditional crawler programs. This kind of search algorithm takes the way of blindly searching data, and does not make use of the characteristics of crawling data. When there are many websites, the crawler will be very time-consuming. Based on the above reasons, an adaptive heuristic search algorithm is designed. According to the relevance between the page and the topic and the related information of the link, the more likely link is selected for crawling, and the estimation function is modified according to the crawling result. Experiments show that the algorithm can significantly improve the efficiency of crawler than breadth first search algorithm.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
