Optimizing mapreduce with low memory requirements for shared-memory systems

Abstract

MapReduce is a popular parallel programming model to program both large scale clusters and shared-memory multicore systems. While one of the major bottlenecks for shared-memory MapReduce is memory allocation. In this paper, we present a Memory Controlling Model (MCM) that can reduce the overhead of memory allocation by reducing the memory consumption. Based on MCM, we extend the MapReduce framework with low memory requirements, called LMMR (Low Memory consuming MapReduce). We have implemented LMMR on top of Phoenix++, an already highly optimized shared-memory MapReduce from Stanford. We evaluate our system on an Intel shared-memory multicore machine with 16 processing threads and compare it with both Phoenix++ and Hadoop. Experiments on three different popular applications show that, compared to Phoenix++, LMMR saves up to 94% memory and results in a speedup ranging from 1.8X to 3.7X. LMMR also is up to 120 times faster than Hadoop.