A novel approach for avoiding overload in the web crawling

Abstract

In comparison to human searchers, crawlers are capable of retrieving data in greater depth and more quickly. As a result, it can face crippling impact on the performance of the site. Inessential to say, if a particular crawler is carrying out numerous requests per second and/or, Therefore, in a scenario involving multiple crawlers, it would be difficult for a server to handle requests in case each crawler downloads large files and/or performs numerous requests/second. So to avoid the overloading in the retrieving data in the proposed system a sequence flow protocol(Leakey Bucket) is which retrieves the data in the sequence order and also proposed mobbing unrestricted router (MUR)(CFR) which maintains three phases inbound, peak and outbound. If the inbound flow is less than the peak it allows the crawling, and at the same time if the inbound is greater than or equal to height the MUR protocol stops the crawling and request for the server to extend network resource. Is used for retrieving the data in sequential order.