Exploring The Role of Web Crawler and Anti-Crawler Technology in Big Data Era

Abstract

In the era of big data, with lower costs and higher efficiency, web crawlers access resources and information from the Internet, bringing a lot of convenience to businesses and individuals. Nevertheless, there are two sides to everything, as malicious crawlers bring incalculable threats and losses to websites. In order to prevent web crawlers from being abused or even developing into malicious crawlers, web sites usually perform anti-crawler based on techniques such as ip access frequency, browsing page speed, account login, input captcha, js encryption, ajax obfuscation, etc. Anti-crawlers cannot completely block crawlers with a particular technique, but only find ways to increase the cost of crawling for attackers, forcing the catching party to make the right choice after weighing the cost-benefit.