A high performance multi-band data fusion approach for CSST based on column-oriented database

Abstract

High-performance multi-catalog fusion or cross-matching has always been an essential issue in astronomical data processing. In this study, we focus on the fusion of multi-band catalog data in a wide-area survey for the China Space Station Telescope (CSST). We implemented a simple and efficient data fusion method based on column-oriented database technology to produce a more consistent and accurate catalog, and this method can carry out the fusion of millions of source records in a few dozen seconds. We analyze and discuss several significant issues related to data fusion, such as the spatial partitioning and indexing of the target sky regions, the efficient implementation of fusion based on joining in the database, and the segmented processing method to address the issue of missing sources at different declinations. The performance profiling results show that by employing the MergeTree table engine within ClickHouse, establishing high-speed indexes based on the spatial partition index number, adopting an appropriate partitioning strategy, and maintaining orderly storage of records in the database according to the spatial partition index number, the efficient fusion of astronomical catalogs can be accomplished through SQL statements. Performance tests show that the proposed method can fulfill CSST data processing requirements, and it is also of reference value for future work related to massive astronomical data fusion. Compared with data fusion systems such as Large Survey DataBase (LSDB), our method can achieve similar performance results with consistent results.