Cross-modal knowledge transfer for 3D point clouds via graph offset prediction

Abstract

A Point cloud is an important representation of three-dimensional (3D) objects, playing an important role in computer vision. However, the inherent sparseness and disorder of point clouds do not provide a stable representation comparable to 2D image pixels. Graph convolutional neural network (GCNN) can generate local neighborhood descriptions of 3D modalities by constructing a graph but it is difficult to capture relationships between distant points. This study proposes a hierarchical encoder based on graph offset convolution, which aggregates the long short distance relationships within local neighborhoods to extend the graph semantic information contained in the adjacency matrix. Furthermore, to address the difficulty of aligning point clouds with image features while avoiding the limitations of text annotations, we introduce a joint point-view pre-training strategy. This strategy learns a unified representation of the two modalities, improving the network’s comprehension of the limited 3D data. Finally, a cross-modal alignment is used to map point and view information to the same feature space, thereby constraining the training states of the two modalities. The proposed method is validated on both standard and zero-shot classification tasks, showing excellent performance. The proposed 3D backbone network achieves 93.6% overall accuracy on the ModelNet40 dataset, with our pre-training strategy that improves the performance of the model by 1.3%.