NPA: Increased Partitioning Approach for Massive Data in Real-Time Data Warehouse

Abstract

In many business and scientific data warehouses, not only the data amount is growing in geometric series, but also the requirement of real-time capability is increasing. Database partitioning technique which adopts ???divide and conquer??? method can efficiently simplify the complexity of managing massive data and improve the performance of the system, especially the range partitioning. The traditional range partitioning approach brings heavy burden to the system without a increased partitioning algorithm, so it does not adapt to the real-time data warehouse partitioning. To speed up the partitioning algorithm, the current partitioning technology is well studied and three effective range partitioning algorithms for the massive data are proposed, which based on allowing the fluctuation of data amount in each range of partitions. At last, some experiments and applications show that the proposed algorithms are more effective and efficient to partitioning and repartitioning tables in the real-time data warehouse.