EduVQA: A multimodal Visual Question Answering framework for smart education

Visual Question Answering (VQA) shows great potential in educational fields like textbook analysis, classroom interaction, and gamified learning. However, existing VQA systems face significant challenges in addressing the unique complexities of educational scenarios. On one hand, most models lack the ability to dynamically comprehend multimodal contexts, making it difficult to meet the diverse semantic demands of educational tasks. On the other hand, many methods fail to fully leverage the reasoning capabilities of large language models (LLMs), resulting in limited performance on knowledge-driven tasks. To overcome these challenges, we propose a novel VQA framework, EduVQA, specifically designed for educational scenarios. EduVQA incorporates a dynamic context selection mechanism and a pre-answer generation module to effectively manage the complexity of multimodal data in educational contexts. Furthermore, by integrating a fine-tuned large language model, EduVQA significantly enhances the understanding and reasoning needed for complex educational questions. Specifically, EduVQA dynamically filters context information relevant to the questions to reduce noise and employs a multi-level pre-answer generation module, combined with external knowledge bases, to provide precise guidance for answer generation. Experimental results show that EduVQA significantly outperforms state-of-the-art models on the OK-VQA and A-OKVQA datasets, excelling in tasks requiring knowledge reasoning and logical analysis.