Knowledge guided relation enhancement for human-object interaction detection

The Human-Object Interaction (HOI) detection task aims to locate humans and objects, find their matching relationships, and infer their interactions. While existing HOI methods have leveraged the CLIP model, a pre-trained visual-language model capable of understanding both images and text, to improve performance, they still fall short in fully capturing the complexity and fine-grained details of human-object interactions. As a result, their ability to reason about interactions accurately and in-depth remains limited. Therefore, we propose a knowledge-guided interaction perception module that combines multiple relationship information with CLIP’s visual feature information. Then, we utilize prior interaction knowledge from intersection regions to guide the process, resulting in more accurate human-object interaction detection. Moreover, we find that the potential interaction of images relies on subtle visual cues but is masked by other irrelevant information, making it difficult for algorithms to capture the basic features of interaction accurately. To address this, we have designed a human-object salient region enhancement module to enhance the feature information of humans and objects and enable better interaction pairing. Experimental results demonstrate that our method with knowledge guided (KGRE) achieves state-of-the-art performance on both the HICO-DET and V-COCO benchmark datasets.