Parallel Acyclic Joins: Optimal Algorithms and Cyclicity Separation

Abstract

We study equi-join computation in the massively parallel computation (MPC) model. Currently, a main open question under this topic is whether it is possible to design an algorithm that can process any join with load \(O(N {\rm {polylog}} N / p^{1/\rho ^*}) \) — measured in the number of words communicated per machine — where N is the total number of tuples in the input relations, ρ^* is the join’s fractional edge covering number, and p is the number of machines. We settle the question in the negative for the class of tuple-based algorithms (all the known MPC join algorithms fall in this class) by proving the existence of a join query with ρ^* = 2 that requires a load of Ω(N/p^1/3) to evaluate. Our lower bound provides solid evidence that the “AGM bound” alone is not sufficient for characterizing the hardness of join evaluation in MPC (a phenomenon that does not exist in RAM). The hard join instance identified in our argument is cyclic, which leaves the question of whether \(O(N {\rm {polylog}} N / p^{1/\rho ^*}) \) is still possible for acyclic joins. We answer this question in the affirmative by showing that any acyclic join can be evaluated with load \(O(N / p^{1/\rho ^*}) \), which is asymptotically optimal (there are no polylogarithmic factors in our bound). The separation between cyclic and acyclic joins is yet another phenomenon that is absent in RAM. Our algorithm owes to the discovery of a new mathematical structure — we call “canonical edge cover” — of acyclic hypergraphs, which has numerous non-trivial properties and makes an elegant addition to database theory.