Improving the performance of HDFS by reducing I/O using adaptable I/O system

Abstract

In this paper, we propose a new framework HDFS-AIO to enhance HDFS with Adaptive I/O System (ADIOS) that supports many different I/O methods and enable the upper application to select optimal I/O routines for a particular platform without source code modification and re-compilation. Specifically, we first customize ADIOS into a chunk-based storage system so that the semantics of its APIs can fit the requirement of HDFS easily; then we utilize Java Native Interface (JNI) to bridge HDFS and the tailored ADIOS together. We use different I/O patterns to compare HDFS-AIO and the original HDFS, and the experimental results show the feasibility and benefits of the design. We also shed light on the performance of HDFS-AIO using different I/O techniques. To the best of our knowledge, this is the first attempt to leverage ADIOS to enrich the functionality of HDFS.