GRPIC: an end-to-end image captioning model using three visual features

lmage captioning is a multimodal task involving both computer vision and natural language processing. Recently, there has been a substantial improvement in the performance of image captioning with the introduction of multi-feature extraction methods. However, existing single-feature and multi-feature methods still face challenges such as a low refinement degree, weak feature complementarity, and lack of an end-to-end model. To tackle these issues, we propose an end-to-end image captioning model called GRPIC (Grid-Region-Pixel Image Captioning), which integrates three types of image features: region features, grid features, and pixel features. Our model utilizes the Swin Transformer for extracting grid features, DETR for extracting region features, and Deeplab for extracting pixel features. We merge pixel-level features with region and grid features to extract more refined contextual and detailed information. Additionally, we incorporate absolute position information and pairwise align the three features to fully leverage their complementarity. Qualitative and quantitative experiments conducted on the MSCOCO dataset demonstrate that our model achieved a 2.3% improvement in CIDEr, reaching 136.1 CIDEr compared to traditional dual-feature methods on the Karpathy test split. Furthermore, observation of the actual generated descriptions shows that the model also produced more refined captions.