An Empirical Study on Efficient Storage of Human Genome Data

Abstract

Next-generation sequencing (NGS) has become affordable and fast, facilitating large scale population-level Whole Genome Sequencing (WGS) studies. NGS and its processing pipeline generate 100's of gigabytes of data per human subject, which can grow to petabytes for large studies, such as the upcoming GenomeIndia program. At these scales, affordable and reliable storage of data becomes a challenge. Here, we propose a preliminary data management architecture for storage and querying of data from the GenomeIndia project. In this initial empirical study, we focus on existing generic and domain-specific compression techniques for reducing the storage space of genome sequence data and compare erasure coding and replication in providing reliability on commodity hardware. We report the time and space complexity of these approaches, and this will reform the future design of our architecture.