PAdapter: Adapter combined with prompt for image and video classification

Abstract

In computer vision, parameter-efficient transfer learning has become an extensively used technology. Adapter is one of the commonly used basic modules, and its simplicity and efficiency have been widely proven. In the case of freezing the network backbone, only fine-tuning additional adapters can often achieve similar or even better results with lower computational costs compared to fully fine-tuning. However, the bottleneck structure of Adapter leads to a non-negligible information loss, thereby limiting the performance of Adapter. To alleviate this problem, this work proposes a plug-and-play lightweight module called PAdapter, which is a Prompt-combined Adapter that can achieve parameter-efficient transfer learning on image classification and video action recognition tasks. PAdapter is improved based on Adapter, and Prompt is introduced at the bottleneck to supplement the information that may be lost. Specifically, in the bottleneck structure of Adapter, we concatenate a learnable Prompt with bottleneck features at dimension D to supplement information and even enhance the visual expression ability of bottleneck features. Many experiments on image classification and video action recognition show that PAdapter achieves or exceeds the accuracy of full fine-tuning models with less than 2% extra parameters updated. For example, on the SSv2 and HMDB-51 datasets, PAdapter improves the accuracy by 5.49% and 16.68% respectively compared to full fine-tuning. And in almost all experiments, our PAdapter achieved higher accuracy than Adapter with similar number of tunable parameters. Code is available at https://github.com/owlholy/PAdapter.