DXRAM: A Persistent In-Memory Storage for Billions of Small Objects

Abstract

Large-scale interactive applications and real time data-processing are facing problems with traditional disk-based storage solutions. Because of the often irregular access patterns they must keep almost all data in RAM caches, which need to be manually synchronized with secondary storage and need a lot of time to be re-loaded in case of power outages. In this paper we propose a novel key-value storage keeping all data always in RAM by aggregating resources of potentially many nodes in a data center. We aim at supporting management of billions of small data objects (16-64 byte) like for example needed for storing graphs. A scalable low-overhead meta-data management is realized using a novel range-based ID approach combined with a super-overlay network. Furthermore, we provide persistence by a novel SSD-aware logging approach allowing to recover failed nodes very fast.