ArchDB: Towards Parallelized Recovery in Massive Archived Databases

Abstract

Monitoring online transactions or tracking users' behavior will generate large-scale archived streaming data in scientific experiments, inner-network audit logs and so on. These archived systems may scale up to petabytes (1015 Bytes). Storing and analyzing the structural data in such scale calls forth at least three challenging issues: data reliability, data storing and analyzing performance, and tradeoff between high reliability and high performance. Based on analyzing the characteristics of the archived streaming data, we propose a novel high reliable log-free database architecture, ArchDB. In order to meet the three challenges, this paper designs optimized data placement policy, data block size and data archiving occasion, pipelining and parallelizing archiving procedure. The experimental results show ArchDB can double the insertion performance and speed up the recovery process by a factor of the parallel recovery degree.