SequenceFile Storage Optimization Method Based on Pile Structure

Abstract

Hadoop Distributed File System (HDFS) performs well when storing and managing large data sets, but its performance is significantly reduced when processing massive small files. In response to this problem, a SequenceFile file storage optimization method based on the pile structure (OPSS) is proposed. The method uses the pile as a unit to merge and merge small files in the pile into SequenceFile based on the worst-fit strategy, reducing the data blocks consumed by the same number of small files. In addition, this method stores the index information of small files in a global index file, and accesses the small files through the global index file, which improves the access efficiency and reduces the memory occupied by the index file. Experimental results show that this method effectively optimizes the performance of HDFS for accessing large amounts of small files.