MMFINet: A multimodal fusion network for accurate fish feeding intensity assessment in recirculating aquaculture systems

Abstract

In recirculating aquaculture systems, a precise and efficient assessment of fish feeding intensity is essential for minimizing feed costs and ensuring optimal feeding practices. Traditional single-modal methods for analyzing fish feeding intensity are highly susceptible to environmental interference, making it difficult to comprehensively capture the entire feeding process. These limitations hinder their practical application in real-world aquaculture environments. To address these challenges, a novel multimodal fusion interaction network (MMFINet) is proposed to enhance the accuracy and robustness of feeding intensity analysis by integrating audio, video, and dissolved oxygen data. In MMFINet, features are first extracted from the input data, including video, audio, and dissolved oxygen. The extracted audio and video features are then fused in the audio–video aggregation module, which employs self-attention and cross-attention mechanisms. The cross-modal fusion module further refines key features by applying a cross-attention mechanism to incorporate the dissolved oxygen data. To reduce model complexity, a lightweight separable convolutional feedforward block is used in place of the traditional transformer’s feedforward network, reducing both the parameter count and computational cost. Experimental results demonstrate that MMFINet achieves a classification accuracy of 97.6% on the multimodal fish feeding intensity dataset, outperforming unimodal methods and exceeding the performance of other multimodal fusion approaches by more than 3%. These findings highlight the potential of MMFINet as a robust solution for enhancing feeding precision and optimizing feed utilization in aquaculture.