Distributed pipelining processing for index updating method

Abstract

Crawling and indexing have been considered regarding existing search engines, but whereas high speed crawling has been studied widely, improvements in the speed of indexing have seldom been discussed. Building a fresh information search engine should, however, unify these arguments concerning crawling and indexing. In this report, an index updating process based on a pipeline that unifies crawling and indexing was proposed. The techniques of a distributed index updating process are discussed with regards to a distributed cooperative search engine.