A scalable maximum-clique algorithm using Apache Spark

Abstract

In this paper, we propose a scalable algorithm for finding the exact solution to the maximum-clique problem. At the heart of our approach lies a multi-phase partitioning strategy, which enables iterative, in-memory processing of graphs. The multi-phase partitioning is tuned for the resources of the machine/cluster to get the best performance. To promote parallelization and scalability on both a cluster-level (distributing the problem on a number of machines) and on a machine-level (using all available cores on each machine), we use Apache Spark. We explore the untraditional usage of distributed frameworks, such as Apache Spark, to distribute computational load, as opposed to distributing big data. We focus on dense graphs, typically with thousands of vertices and a few millions edges; this is in contrast to sparse real-world graphs that don't initially fit into the memory of a single driver machine. Our experiments show that, for large dense graphs, we get up to 100% performance speedup compared to the state-of-the-art parallel approaches. Moreover, our algorithm is highly scalable and fault-tolerant.