Parallel Data Indexing and Storage on a COTS Cluster

In this paper, a data transfer, indexing and storage system on a commodity-off-the-shelf cluster using parallel processing, asynchronous file Input/Output, direct memory access and asynchronous User Datagram Protocol sockets is proposed. Also, a performance evaluation framework for the system is described. There are two main considerations in developing the system. First, as data communication networks support increased data rates due to fiber optical cables and more efficient network devices, better data transfer and storage methods are required to exploit the speed of the networks. Second, applications in particle physics, climate modeling and weapon systems simulation generate petabytes of data from a single experiment. The challenge is to index and store the data as soon as it is produced and preprocessed by several instruments.