An active storage system with dynamic task assignment policy

Abstract

As the amount of data grows exponentially, active storage was proposed as an alternative solution in order to mitigate the I/O performance problem in distributed cluster system. It moves appropriate computation to the location in which data is stored and hence reduces the amount of data transferred. Prior research has investigated and deployed the concept in different forms. However, balancing the computation load between compute and storage nodes in active storage system according to the current system status has been largely unexplored. In this paper, we propose an active storage system that provides programming interfaces for applications to implement the offload computing algorithm and assignment policy. We have implemented a dynamic task assignment policy that assigns computing tasks between compute and storage nodes according the feedback information of system environment. Our experimental results show that there is a optimal allocation proportion with the considerable impact of system status and characteristics of computation, and the proposed dynamic task assignment policy can make an allocation of computing tasks at a proportion close to the optimal assignment.