A Distributed NameNode Cluster for a Highly-Available Hadoop Distributed File System

Abstract

Recently, Hadoop attracts much attention of engineers and researchers as an emerging and effective framework for Big Data. HDFS (Hadoop Distributed File System) can manage huge amount of data with high performance and reliability using only commodity hardware. However, HDFS requires a single master node, called a NameNode, to manage the entire namespace of the file system. This causes the SPOF (Single Point Of Failure) problem because the file system becomes inaccessible when the NameNode fails. This also causes a bottleneck of efficiency since all the access requests to the file system have to contact the NameNode. Finally the scale up of a namespace is difficult because the NameNode manages all metadata of the namespace on its own memory, which is limited and expensive resource. In this paper, we propose a new HDFS architecture consisting of several NameNodes to resolve all the above problems.