Davram: Distributed Virtual Memory in User Space

Abstract

One of the most challenging problems in modern distributed big data systems lies in their memory management: these systems preallocate a fixed amount of memory before applications start. In the best case where more memory can be acquired, users have to reconfigure the deployment and re-compute many intermediate results. If no more memory is available, users are then forced to manually partition the job into smaller tasks, incurring both development and performance overhead. This paper presents a user-level utility for managing the memory in distributed systems—the Distributed and Autonomous Virtual RAM (Davram). Davram enables to efficiently swap data between memory and disk in a distributed system without users' intervention or applications' awareness.