An Improved Small File Storage Strategy in Ceph File System

As the double-write and backup strategies cause low efficiency of storing massive small files in Ceph FS (file system), we design a framework SFPS (Small File Process System) which adopted three technologies, including: k-means, Simhash and Analytic Hierarchy Process (AHP). The designed method can reduce the quantity and redundant data of massive small files by merging similar files after the deduplication with adaptive block skipping. Due to the associated files have high probability to be read in the next time, we also design a pre-fetching mechanism and metadata management module based on the high-efficiency database Redis to guarantee a high efficiency of read rate. The proposed scheme aims to achieve a better trade-off among the utilization of space of hard-disk and bandwidth resources, file access time, hard-disk I/O as well as the cluster performance in Ceph FS by eliminating duplicate copies of repeating data, merging similar small files, and introducing the cache module. Experimental results show that the method presented in this paper can not only effectively improve the utilization of bandwidth resources and space of device storage as well as the small file read rate, but also significantly reduce the amount of disk I/O generated by reading and writing files.