Scalable Distributed Meta-Data Management in Parallel NFS

Abstract

High performance computing (HPC) applications demand high I/O throughput from file systems to match their fast processing requirements. These HPC applications create large amounts of file meta-data that can overwhelm current single meta-data server file systems leading to performance bottlenecks. We address this problem with a multiple meta-data server (M-MDS) design using standard parallel NFS (pNFS). We utilize NFS directory referrals mechanism with hashing to efficiently distribute meta-data across a cluster of metadata servers. We show through a large scale setup in Amazon EC2 that our pNFS M-MDS can scale almost linearly and outperform Lustre CMD (Clustered Metadata) by up to 3 times in some of the file system meta-data operation benchmarks.