Residual Self-Attention for Visual Question Answering

Abstract

Over these years, many attention mechanism-based neural network models have been put forward in the visual question answering (VQA) by some researchers. However, the semantic gap between the multimodality cannot be bridged simply, and the combination of the questions and the images may be arbitrary, which both hinder the jointly representing learning and make the VQA model easy for overfitting. Therefore, in this paper, a multi-stage attention model has been put forward, that is, for the image, the bottom-up attention and residual self-attention for the image itself and the question-guided double-headed soft (top-down) attention method were used to extract the image features; for the question, the pre-trained GloVe word embeddings were used in this paper to represent the semantics and the GRU was used to encode the questions into fixed-length sentence embedding; finally, the image features were fused with the question embedding through the Hadamard product and then input into the sigmoid multi-classifier. The experiment showed that with the model the overall accuracy of 67.26% in the VQA 2.0 dataset was finally achieved, higher than that of other advanced models.