Analysis of Metacrawler approach for URL based DUST removal by knowledge engineering systems

Abstract

Multiple copies of URLs gathered by the web crawlers responsible for pages with similar or near about to similar content. Few pages combined by the web crawlers consist of similar content. Different URLs with Similar Text are generally known as DUST. Result of this is crawl data, to store the data and use such duplicated data results in building of less quality marking, waste of resources and destitute naive user experiences.To study with such problem, multiple studies have been observed. Previous studies focus only on URL based DUST removal .The proposed method removes content depended DUST and URL based DUST. To crawl the documents we are using a new method of metacrawler which fetches results from three Search engine. We are going to compare each website content with the other linked content to remove duplicates using k- gram paraphrased technique.