Variable multi-scale attention fusion network and adaptive correcting gradient optimization for multi-task learning

Network architecture and optimization are two indispensable parts in multi-task learning, which together improve the performance of multi-task learning. Previous work has rarely focused on both aspects simultaneously. In this paper, we analyze the multi-task learning from network architecture and optimization. In network architecture aspect, we propose a variable multi-scale attention fusion network, which overcomes the issue of feature loss when processing small-scale feature maps during upsampling and resolves the problem of inadequate learning in conventional multi-scale models due to significant spatial size disparities. In optimization aspect, a adaptive correcting gradient scheme is put forward to treat the defects of conflicts and dominance among multiple tasks during the process of training, and it effectively alleviates the imbalance of multi-task training. Various ablation experiments and comparative experiments demonstrate that simultaneously considering the network framework and optimization can make great improvement for the performance of multi-task learning. Our code is available at https://github.com/SyqxhSt/Net-Opt-MTL