Design and implementation of competent web crawler and indexer using web services

Abstract

Today the internet has become a part of human beings life. To get the information what the user is requesting is the job of search engine which indeed takes the help of web crawler. Designing and developing a competent web crawler is a challenging task. This paper proposes Web crawler and Indexer. The WebCrawler consist of crawler services and indexer services and realized as web services. The crawler and indexer services communicate using XML, SOAP and WSDL. The web pages are fetched and parsed for retrieving all the hyperlinks by the crawler service, and then the same process is continued recursively using the Breadth-First strategy. The result of crawler service is downloaded and given as an input to the indexer services by passing the message using web services. Then the indexer service parses the HTML pages, removes stop words, stemming of keywords are carried out as pre-processing steps. Finally the result is stored in the form of inverted index.