A novel data redundancy scheme for de-duplication storage system

Abstract

As the amount of digital data growing explosively, data deduplication becomes an attractive technique for conserving the storage and network requirement. However, the tradeoff between redundancy for data availability and data duplicate elimination for storage utilization becomes an issue in the deduplication storage system. To address the issue, we proposed a novel data redundancy scheme to improve the data availability for the deduplication storage system. The key idea is to optimize the system-level availability by assigning an optimal replication degree for the individual data chunk based on its commonality in the data set. We present the design and implementation of the prototype of the high availability storage system and evaluate the feasibility and effectiveness of the optimization approach. The evaluation experiments demonstrated the approach is practical and effective.