ICV-Net: An identity cost volume network for multi-view stereo depth inference

Abstract

The construction of 3D cost volumes is essential for deep learning-based Multi-view Stereo (MVS) methods. Although cascade cost volumes alleviate the GPU memory overhead and improve depth inference performance in a coarse-to-fine manner, the cascade cost volumes are still not the optimal solution for the learned MVS methods. In this work, we first propose novel identity cost volumes with the identical cost volume size at each stage, which dramatically decreases memory footprint while clearly improving depth prediction accuracy and inference speed. The depth inference is then formulated as a dense-to-sparse search problem that is solved by performing a classification to locate predicted depth values. Specifically, in the first stage, a dense linear search is adopted to calculate an initial depth map. The depth map is then refined by sampling less depth hypotheses in the following two stages. In the final stage, we exploit a binary search with only two depth hypotheses to obtain the final depth map. Combining identity cost volumes with the dense-to-sparse search strategy, we propose an identity cost volume network for MVS, denoted as ICV-Net. The proposed ICV-Net is validated on competitive benchmarks. Experiments show our method can dramatically reduce the memory consumption and extend the learned MVS to higher-resolution scenes. Moreover, our method achieves state-of-the-art accuracy with less runtime. Particularly, among all the learning-based MVS methods, our method achieves the best accuracy (an accuracy score of 0.286) on DTU benchmark with the least GPU memory (with a testing memory overhead of 1221 MB).