An On-Demand Cloud-Native Containerized Storage Design and its Practice of HDFS-on-Kubernetes

Abstract

Cloud-native big data services become popular in recent years. Two pillars of these services are identified: the separation architecture of compute and storage, and the application-specific controller mechanism. In terms of storage for big data on the cloud, current practices focus on managing a single on-premise storage cluster or building independent PaaS storage services. This paper focuses on the cloud-native containerized storage. An on-demand provisioning design is proposed, which extends the mainstream storage architecture and supports the provisioning of storage clusters for multi-tenancy in a dynamic manner. Its instance of HDFS-on-Kubernetes is implemented. With the mechanisms of global endpoint provisioning and dynamic volume provisioning, this provisioner enables the creation and management of multiple on-demand storage clusters with full-stack resources in an automated way. It guarantees the native performance of host network and local storage, which has been validated through experiments and production applications. It is also easy to use because of its high-level abstraction and single-point configuration mechanism. The design as well as the provisioner has served real business in industrial scenarios.