An Efficient Collaborative Communication Mechanism for MPI Neighborhood Collectives

Neighborhood collectives are introduced in MPI-3.0 standard to provide users with the opportunity to define their own communication patterns through the process topology interface of MPI. In this paper, we propose a collaborative communication mechanism based on common neighborhoods that might exist among groups of k processes. Such common neighborhoods are used to decrease the number of communication stages through message combining. We show how designing our desired communication pattern can be modeled as a maximum weighted matching problem in distributed hypergraphs, and propose a distributed algorithm to solve it. Moreover, we consider two design alternatives: topology-agnostic and topology-aware. The former ignores the physical topology of the system and the mapping of processes, whereas the latter takes them into account to further optimize the communication pattern. Our experimental results show that we can gain up to 8x and 5.2x improvement for various process topologies and a SpMM kernel, respectively.