TMSF: Taylor expansion approximation network with multi-stage feature representation for optical flow estimation

Abstract

Optical flow estimation is a fundamental task in computer vision. Existing CNN-based and transformer-based methods have proven their powerful ability in generating preferable performance, but they still suffer from the loss of fine details and objects' shape. To cope with these problems, this paper develops a Taylor expansion approximation network with multi-stage feature representation, namely TMSF, including a basic network and a refine network. In the basic network, multi-stage modules, including feature enhancement module (FEM) for enriching image features, feature/context network for feature extraction, and iterative update module (IUM) for coarse optical flow estimation, are employed to represent fine features. In the refine network, a refinement architecture is constructed based on the third-order Taylor approximation expansion to further refine features from the basic network for optical flow, in which a feature attention module (FAM) is used to estimate each derivative layer. Meanwhile, a novel loss function is formed by end-point-error (EPE) and structural similarity (SSIM) to ensure the convergence of our TMSF to a satisfactory solution. Quantitative associated with qualitative experimental results validate that our TMSF performs better than state-of-the-art optical flow estimation methods on performance improvement and shape preservation. The code will be available at https://github.com/MysterYxby/TMSF.