GatorShare: a file system framework for high-throughput data management

Abstract

Voluntary Computing systems or Desktop Grids (DGs) enable sharing of commodity computing resources across the globe and have gained tremendous popularity among scientific research communities. Data management is one of the major challenges of adopting the Voluntary Computing paradigm for large data-intensive applications. To date, middleware for supporting such applications either lacks an efficient cooperative data distribution scheme or cannot easily accommodate existing data-intensive applications due to the requirement for using middleware-specific APIs. To address this challenge, in this paper we introduce Gator-Share, a data management framework that offers a file system interface and an extensible architecture designed to support multiple data transfer protocols, including BitTorrent, based on which we implement a cooperative data distribution service for DGs. It eases the integration with Desktop Grids and enables high-throughput data management for unmodified data-intensive applications. To improve the performance of BitTorrent in Desktop Grids, we have enhanced BitTorrent by making it fully decentralized and capable of supporting partial file downloading in an on-demand fashion. To justify this approach we present a quantitative evaluation of the framework in terms of data distribution efficiency. Experimental results show that the framework significantly improves the data dissemination performance for unmodified data-intensive applications compared to a traditional client/server architecture.