Architecture specification of rule-based deep web crawler with indexer

Abstract

Suitable architecture specification of a deep web crawler with surface web crawler as well as indexer is proposed for fetching large number of documents from deep web using rules. The functional dependency of core and allied fields in the FORM are identified for generating rules using SVM classifier and classifies them as most preferable, least preferable and mutually exclusive. The FORMs are filled with values from most preferable class for fetching large number of documents. The extracted document is indexed for information retrieval applications. The architecture is extended to distributed crawler using web services. The proposed crawler fetches large number of documents while using the values in most preferable class. This architecture has higher coverage rate and reduces fetching time. The retrieval performance is encouraging and achieves similar precision of retrieval as Google search engine system.