Matching Images from Different Viewpoints with Deep Learning Based on LoFTR and MAGSAC++

Abstract

Matching 2D images from different viewpoints plays a crucial role in the fields of Structure-from-Motion and 3D reconstruction. However, image matching for assorted and unstructured images with a wide variety of viewpoints leads to difficulty for traditional matching methods. In this paper, we propose a Transformer-based feature matching approach to capture the same physical points of a scene from two images with different viewpoints. The local features of images are extracted by the LoFTR, which is a detector-free deep-learning matching model on the basis of Transformer. The subsequent matching process is realized by the MAGSAC++ estimator, where the matching results are summarized in the fundamental matrix as the model output. By removing image feature points with low confidence scores and applying the test time augmentation, our approach can reach a mean Average Accuracy 0.81340 in the Kaggle competition Image Matching Challenge 2022. This score ranks 45/642 in the competition leaderboard, and can get a silver medal in this competition. Our work could help accelerate the research of generalized methods for Structure-from-Motion and 3D reconstruction, and would potentially deepen the understanding of image feature matching and related fields.