Balanced multifileinput split (BaMS) technique to solve small file problem in hadoop

Abstract

Hadoop Ditributed File system is designed to process large amount of data. However, processing large number of small files seems inefficient, since Hadoop supports only block level operations. But small file processing is inevitable for real time processing, log processing etc. Hence, to rectify this performance bottleneck, we propose a Balanced MultiFileInput Split (BaMS) technique where files are merged together and stored. Data is converted to bytes and collectively stored in ArrayWritable format. To avoid the need for separate indexing, we follow a hierarchical file naming & storing scheme. The method describes how to access the merged files through Map Reduce Programs. Analysis performed on BaMS proves that it is much efficient compared to the existing methods like HAR and sequence files in terms of storage and access efficiency.