GSC: Greedy shard caching algorithm for improved I/O efficiency in GraphChi

Abstract

Disk-based large scale graph computation on a single machine has been attracting much attention, with GraphChi as one of the most well-accepted solutions. However, we find out that the performance of GraphChi becomes I/O-constrained when memory is moderately abundant, and from some point adding more memory does not help with the performance any more. In this work, a greedy caching algorithm GSC is proposed for GraphChi to make better use of the memory. It alleviates the I/O constraint by caching and delaying the write-backs of GraphChi shards that have already been loaded into the memory. Experimental results show that by minimizing unnecessary I/Os, GSC can be up to 4x faster during computation than standard GraphChi under memory constraint, and achieve about 3x performance gain when sufficient memory is available.