Abstracting parallel program specification: a case study on k-means clustering

Abstract

The Forelem framework was first introduced to optimize database queries using compiler techniques. Since its introduction, Forelem has proven to be more versatile and to be applicable beyond database applications. In this paper we show that Forelem can be used to specify parallel programs at an abstract level whilst still guaranteeing efficient parallel execution. This is achieved by a sequence of transformations that can be directly implemented as an optimizing compiler toolchain. To demonstrate this, a case study is described, k-Means clustering, for which four implementations are mechanically generated that improve standard MPI C/C++ and outperform state-of-the-art Hadoop implementations.