StorNet: Co-scheduling of end-to-end bandwidth reservation on storage and network systems for high-performance data transfers

Abstract

Modern scientific data-intensive applications brought about the need for novel data transfer technologies and automated tools capable of effectively utilizing available raw network bandwidth and intelligently assisting scientists in replicating large volumes of data to desired locations in a timely manner. In this paper we describe the design of StorNet, an integrated end-to-end resource provisioning and management system for high performance data transfers that can operate with heterogeneous network protocols and storage systems in a federated computing environment. StorNet allocates and co-schedules storage and network resources involved in data transfers. It is based on existing Storage Resource Manager, TeraPaths, and OSCARS capabilities. StorNet provides data intensive applications with the capability of predictable, yet efficient delivery of data at rates of multiple gigabits/second, bridging end-to-end advanced storage and network technologies in a transparent way.