An archive-based method for efficiently handling small file problems in HDFS

Abstract

Hadoop distributed file system (HDFS) performs well when storing and managing large files. However, its performance significantly decreases when dealing with massive small files. In response to this problem, a novel archive-based solution is proposed. The archive refers to merging multiple small files into larger data files, which can effectively reduce the memory usage of the NameNode. The current archive-based solutions have the disadvantages of long access time, long archive construction time, and no support for storage, updating and deleting small files in the archive system. Our method utilizes a dynamic hash function to distribute the metadata of small files across multiple metadata files. We construct a primary index that combines dynamic and static indexes for these metadata files. Regarding data files, include some read-only files and one readable–writable file. A small file's contents are written into a readable and writable file. Upon reaching a predetermined threshold, the readable–writable file transitions into read-only status, with a fresh readable–writable file replacing it. Experimental results show that the scheme improves the efficiency of archive access and archive creation and is more efficient than the original HDFS storage and update efficiency.