Cross-modal Prompt-Driven Network for low-resource vision-to-language generation

Abstract

Image captioning is a classic vision-to-language generation task, which aims to generate a descriptive sentence to describe the input image, involving the understanding of the image and the generation of natural language. Conventional methods require a large-scale labeled dataset for training, which includes a large volume of image-caption pairs. However, for several application scenarios, e.g., medicine and non-English, such plenty of image-caption pairs are usually not available. In this work, we propose the Cross-modal Prompt-Driven Network (XProDNet) to perform low-resource image captioning, which can generate accurate and comprehensive image captioning, with extremely limited data for training. We conduct experiments on (1) six benchmark datasets; (2) three application scenarios, i.e., conventional image captioning, medical image captioning, and non-English image captioning; (3) four target languages, i.e., English, Chinese, German, and French; (4) two experimental settings, i.e., fully-supervised learning and few-shot learning. The extensive experiments prove the effectiveness of our approach, which can not only generate high-quality and comprehensive image captions but also significantly surpass previous state-of-the-art methods under both the few-shot learning and fully-supervised learning settings. The improved results suggest that our method has great potential for improving image captioning in real-world applications.