HAMSA: Hybrid attention transformer and multi-scale alignment aggregation network for video super-resolution

Abstract

Video Super-Resolution (VSR) aims to enhance the resolution of video frames by utilizing multiple adjacent low-resolution frames. For across-frame information extraction, most existing methods usually employ the optical flow or learned offsets through deformable convolution to perform alignment. However, due to the complexity of real-world motions, the estimating of flow or motion offsets can be inaccurate while challenging. To address this problem, we propose a novel hybrid attention transformer and multi-scale alignment aggregation network for video super-resolution, named HAMSA. The proposed HAMSA adopts a U-shaped architecture to achieve progressive alignment using a multi-scale manner. Specifically, we develop a hybrid attention transformer (HAT) feature extraction module, which uses the proposed channel motion attention (CMA) to extract features that facilitate inter-frame alignment. Second, we first design a U-shaped multi-scale feature alignment (MSFA) module that ensures precise motion estimation between different frames by starting from large-scale features, gradually aligning them to smaller scales, and then restoring them using skip connections and upsampling. In addition, to further refine the alignment process, we introduce a non-local feature aggregation (NLFA) module, which serves to apply non-local operations to minimize alignment errors and enhance the detail fidelity, thereby improving the overall quality of the super-resolved video frames. Extensive experiments on the Vid4, Vimeo90k-T, and REDS4 datasets demonstrate that our HAMSA achieves superior VSR performance compared to other state-of-the-art (SOTA) methods while maintaining a good balance between model size and performance.