Dynamic multi teacher knowledge distillation for semantic parsing in KBQA

Abstract

Knowledge base question answering (KBQA) is an important task of extracting answers from a knowledge base by analyzing natural language questions. Semantic parsing methods convert natural language questions into executable logical forms to obtain answers on the knowledge base. Conventional approaches often prioritize singular logical forms, overlooking the distinct strengths inherent in various logical frameworks for problem solving. Recognizing that different logical forms may excel in addressing specific types of questions, our aim is to harness these strengths. By integrating the strengths of different logical forms, we expect to achieve more comprehensive and effective semantic parsing solutions. In our paper, we propose a Dynamic Multi Teacher Knowledge Distillation for Semantic Parsing (DMTKD-SP). DMTKD-SP leverages a collection of teacher models, each mastering a unique logical form, to collaboratively guide a student model so that knowledge from different logical forms can be transferred into the student model. To achieve this, we employ a confidence-based weight assignment module to dynamically assign weights for each teacher model. Furthermore, we introduce a self-distillation mechanism to mitigate the confusion caused by simultaneous learning from multiple teachers. We evaluate DMTKD-SP across variants of the KQA Pro dataset, demonstrating an accuracy improvement of 0.35% on five types of questions, with a notable 0.75% improvement for Count questions.