GMaglev: Graph-friendly Consistent Hashing for Distributed Social Graph Partition

Abstract

Consistent hashing plays an important role in distributed systems as well as in distributed graph systems. It assumes the role of load balancing and data relocation for system expansion and contraction. Graph partitioning is used by many distributed graph systems, which can improve the system’s performance. In distributed systems, expansion and contraction cause data relocation. Data relocation consumes CPU and leads to the unreliability of online services.The system’s performance using the graph partition algorithm to optimize data distribution will inevitably decline with data relocation. Data migration due to capacity expansion is accidental, and system efficiency is a long-term indicator. We present GMaglev to find a way to balance the data migration and the fanout optimization. Experiments on multiple datasets show the effectiveness of GMaglev algorithm. In the case of only 5% increase in data relocation radio, fanout is reduced by 20%.