Relevant document crawling with usage pattern and domain profile based page ranking

Abstract

WWW is a distributed heterogeneous information resource. With the exponential growth of WWW, it has become difficult to access desired information that matches with user needs and interest. In spite of strong crawling, indexing and page ranking techniques, the returned result-sets of the search engine lack in accuracy and preciseness. Large number of irrelevant links, topic drift, and load on servers are some of the other issues that need to be addressed towards developing an efficient search engine. In this paper a solution is being proposed for the development of a crawling technique that attempts to reduce server load by taking advantage of migrants for downloading the relevant pages; pertaining to a specific topic only. The downloaded documents are then ranked considering user preferences and past usage patterns of the web page thereby improving the quality of retuned result-sets.