LLM Knowledge-Driven Target Prototype Learning for Few-Shot Segmentation

Abstract

Few-Shot Segmentation (FSS) aims to segment new class objects in a query image with few support images. The prototype-based FSS methods first model a target prototype and then match it with the query feature for segmentation. Recent research has focused on mining visual features to model the prototype. However, modeling the target prototype using visual features alone is not sufficient to represent target objects due to appearance differences between targets in support and query images. To address this limitation, based on the generalizable knowledge implied in the Large Language Model (LLM), we propose an LLM Knowledge-Driven Target Prototype Learning method (KD-TPL) to learn a robust prototype for the target object in the query image. Specifically, a knowledge-driven semantic prior generator is constructed to mine semantic priors in the query image applied to LLM knowledge. Based on the modeled semantic priors, a knowledge-driven hybrid prototype learner is designed to learn a representative target prototype. A knowledge-driven query feature enhancer is developed to enhance the semantics of the query feature. Finally, competitive comparison and ablation experimental results on COCO-$20^i$ and PASCAL-$5^i$ demonstrate the effectiveness of our method.