End-To-End Training Of Object Segmentation Task And Video Question-Answering Task

Abstract

For complicated Visual Question Answering(VQA) tasks that incorporates multiple objects, to train the VQA model using segmented objects data as inputs is proved to be effective for various downstream tasks. In this work, we tried to train the VQA task model and object segmentation model in end-to-end fashion instead of training independently. We employed CLEVRER as a target VQA task. We first trained MONet(Multiple Object Network), an object segmentation network, with the dataset, and trained Aloe, a VQA model, using the output of the trained MONet. Finally we connect MONet and Aloe to fine-tune them in end-to-end setting and confirmed that the performance of VOA task has been greatly improved.