Small file access optimization based on GlusterFS

Abstract

This paper describes a strategy to optimize small file's reading and writing performance on traditional distributed file system. Traditional distributed file system like GlusterFS stores data within local file system (XFS, EXT3, EXT4, etc.), which shows a significant bottleneck on file metadata lookup. We try to re-design metadata structure by merging small file into large file, thus to reduce size of metadata, so we can store the whole files' metadata inside main memory. We design and implement the whole strategy on GlusterFS, test results show a great performance optimization on small file operation.