TFS-Net: Temporal first simulation network for video saliency prediction

Abstract

Video saliency prediction (VSP) plays a critical role in modern video processing systems by optimizing computational resource allocation and enhancing overall system performance. However, existing VSP methods either lack effective temporal modeling or incur high computational costs, particularly struggling with the initialization of video sequences. This paper presents TFS-Net, a novel temporal-first simulation network for VSP that integrates both static and dynamic modeling via parallel-optimized self-attention mechanisms. Specifically, TFS-Net addresses the challenge of initial frame processing with the innovative F31 algorithm and improves multi-scale spatiotemporal feature integration through a Hierarchical Decoder with Multi-dimensional Attention (HDMA). Drawing inspiration from primate saccadic behavior, the F31 algorithm optimizes processing efficiency during both training and inference phases, demonstrating particular effectiveness in unmanned aerial vehicle (UAV) real-time applications. Extensive evaluations on public datasets demonstrate that TFS-Net achieves significant improvements over state-of-the-art methods, with gains of 14.6%, 12.0%, and 11.2% in AUC-J, CC, and SIM metrics, respectively. Further experiments on UAV video analysis validate the model’s robustness and practicality in real-world scenarios.