A survey in semantic web technologies-inspired focused crawlers

Abstract

Crawlers are software which can traverse the Internet and retrieve Webpages by hyperlinks. In the face of the inundant spam Websites, traditional Web crawlers cannot function well to solve this problem. Semantic focused crawlers utilize semantic web technologies to analyze the semantics of hyperlinks and Web documents. This paper briefly reviews the recent studies on one category of semantic focused crawlers - ontology-based focused crawlers, which are a series of crawlers that utilize ontologies to link the fetched Web documents with the ontological concepts (topics). The purpose of this is to organize and categorize Web documents, or filtering irrelevant Webpages with regards to the topics. A brief comparison are made among these crawlers,from six perspectives - domain, working environment, special functions, technologies utilized, evaluation metrics and evaluation results. The conclusion with respect to this comparison is made in the final section.