Instance-Aware Hierarchical Structured Policy for Prompt Learning in Vision-Language Models

Abstract

In recent years, learnable prompts have emerged as a major prompt learning paradigm, enhancing the performance of large-scale vision-language pre-trained models in few-shot image classification. However, enhancing methods are often time-consuming and inflexible because 1) class-specific prompts are inefficient in certain situations; 2) instance-specific prompts are put in a fixed position. To address these issues, inspired by the coarse-to-fine decision-making paradigm of human, we propose an Instance-Aware Hierarchical-Structured Policy (IAHSP) that integrates instance-specific prompt selection and appropriate position selection using a reinforcement learning fashion. Specifically, IAHSP consists of two sub-policies: 1) the root policy selects the most suitable prompt from the prompts pool, and 2) the leaf policy identifies the optimal position for inserting the selected prompt. We train these two policies iteratively with rewards constraining the prompts while maintaining their diversity. Extensive experiments on 11 public benchmarks demonstrate that our IAHSP significantly boosts the few-shot image classification performance of vision-language pre-trained models, while also exhibiting superior generalization performance.