Efficient fine-tuning of vision transformer via path-augmented parameter adaptation

Abstract

Fine-tuning pre-trained Vision Transformer (ViT) models have been adopted as the de facto paradigm for achieving promising performance on visual tasks. However, the exponential growth in parameter size presents significant challenges to computational and storage efficiency when transferring ViT models to downstream tasks. Leveraging the assumption that trained models are over-parameterized and intrinsically reside a lower-dimensional space, learning a small number of parameters while freezing the backbone has emerged as a promising strategy for efficiently fine-tuning ViT models. In this paper, a path-augmented parameter adaptation method, termed as PPA, is proposed for fine-tuning ViT models. Specifically, a multi-path strategy is designed to learn the parameter updates in pre-trained ViT models, which aims to promote information flow and subspace representation learning via augmented paths. Based on this design, heterogeneous modules with a few learnable parameters are adopted which enable augmented paths to capture diverse information in low-dimensional subspaces. Since the parameters in the augmented paths can be reparametrized to the pre-trained model after fine-tuning, the proposed method incurs no additional inference cost. Extensive experiments and comparisons conducted on various visual benchmark tasks demonstrate the effectiveness of the proposed PPA method.