A 39pJ/label 1920x1080 165.7 FPS Block PatchMatch Based Stereo Matching Processor on FPGA

Abstract

Depth is a fundamental information for lots of computer vision applications. Stereo matching is a commonly used depth estimation method that mimics the human binocular vision system. Most stereo matching systems suffer when attempt to strike a balance between accuracy and computational complexity because of two reasons: 1) It is usually assumed that all the surfaces are fronto-parallel, meaning that neighbors share the same disparity. But a lot of real-world situations are slant surfaces like roads and walls. 2) Conventional methods usually utilize winner takes all (WTA) strategy [1]–[4], where aggregated costs in all disparity levels (usually 128 or 256) must be calculated. But there is only one true guess for each pixel position, such that most of the calculated costs are meaningless. To solve the above issues, in this work, a block PatchMatch-based FPGA accelerator for stereo matching is proposed. PatchMatch introduces random search strategy and slant label, where rectangle superpixels called blocks are used as the basic computing element. Main improvements of this work are: 1) Utilized random search strategy and block level computation can save massive computation. 2) Closer-to-reality slant label improves accuracy. Moreover, plane slant is also helpful for following tasks like 3D reconstruction [5], but none of the existing hardware accelerators can provide this information. 3) Algorithm-hardware co-optimized 6-points Census feature and multi-scale propagation (MSP) are proposed. 4) Based on the testing results on industrial-level KITTI dataset, the real-time performance and energy efficiency of the proposed design outperform state-of-the-art FPGA and ASIC designs, with comparable accuracy.