Dynamic Routing in Machine Reading Comprehension

Abstract

Dynamic routing mechanism was introduced by Geoffrey E Hinton et al. in 2017, which is a new information flow mechanism applied in the deep neural networks (Capsule Networks) for the first time, and achieves state-of-the-art performance on MNIST. The typical mark of CapsuleNet “Vector in Vector out” is a natural property in NLP models (word embedding). So, be inspired by this, we introduce the dynamic routing mechanism in NLP tasks. We focus on machine reading comprehension (MRC) task. On MRC tasks, by introducing the dynamic routing mechanism into BiDAF and BERT, our two models DR-BiDAF and DR-BERT were formed. Experiments on SQuAD2.0 dataset and Facebook bAbI project (The (20) QA bAbI tasks) show that the accuracy and robustness of DR-BiDAF and DR-BERT have a significant improvement compared with their counterpart original models. Besides, the process of model training shows that the convergence speed of new models with dynamic routing is faster than original models. It indicates the optimization problem has also been improved.