Synthetic Aperture Imaging with Events and Frames

Abstract

The Event-based Synthetic Aperture Imaging (E-SAI) has recently been proposed to see through extremely dense occlusions. However, the performance of E-SAI is not consistent under sparse occlusions due to the dramatic de-crease of signal events. This paper addresses this problem by leveraging the merits of both events and frames, leading to a fusion-based SAl (EF-SAI) that performs consistently under the different densities of occlusions. In particular, we first extract the feature from events and frames via multi-modal feature encoders and then apply a multi-stage fusion network for cross-modal enhancement and density-aware feature selection. Finally, a CNN decoder is employed to generate occlusion-free visual images from selected features. Extensive experiments show that our method effectively tackles varying densities of occlusions and achieves superior performance to the state-of-the-art SAl methods. Codes and datasets are available at https://github.com/smjsc/EF-SAI