Learning complex predicates for cardinality estimation using recursive neural networks

Abstract

Cardinality estimation is one of the most vital components in the query optimizer, which has been extensively studied recently. On one hand, traditional cardinality estimators, such as histograms and sampling methods, struggle to capture the correlations between multiple tables. On the other hand, current learning-based methods still suffer from the feature extraction of complex predicates and join relations, which will lead to inaccurate cost estimation, eventually a sub-optimal execution plan. To address these challenges, we present a novel end-to-end architecture leveraging deep learning to provide high-quality cardinality estimation. We exploit an effective feature extraction technique, which can fully make use of the structure of tables, join conditions and predicates. Besides, we use sampling-based technique to construct sample bitmaps for the tables and join conditions respectively. We also utilize the characteristics of predicate tree combined with recursive neural network to extract deep-level features of complex predicates. Finally, we embed these feature vectors into the model, which consists of three components: a recursive neural network, a graph convolutional neural network (GCN) and a multi-set convolutional neural network, to obtain the estimated cardinality. Extensive results conducted on real-world workloads demonstrate that our approach can achieve significant improvement in accuracy and be extended to queries with complex semantics.