An empirical analysis of feature fusion task heads of ViT pre-trained models on OOD classification tasks

Abstract

ViT pre-training model has been widely used in various downstream tasks, and the structure of task head has a significant impact on downstream tasks. While it is a common practice to empirically concatenate the last few layers’ cls token of the ViT model for classification, there exists limited research on whether the feature fusion structure holds significance for the model. This paper primarily discusses the impact of attention-mechanism-based fusion structure on the backbone network and classification performance. Initially, we examine the relationship between dataset and feature fusion task head, followed by an exploration of how different locations of fusion middle layer affect model performance as well as how feature fusion task head influences the backbone network itself. Finally, we characterize the task head through the loss of models based on feature fusion structure. Based on empirical findings, we identify 5 important insights and provide recommendations for the model structures during downstream task fine-tuning.