A Comparative Analysis of Math Word Problem Solving on Characterized Datasets

Benefit from the neural network research, a couple of neural solvers have been developed for automatically solving math word problems (MWPs). These neural solvers are evaluated on several benchmark datasets with diverse characteristics, which leads to a poor comparability of the performance of each solver. To address the problem, a comparative analysis is conducted in this paper to explore the performance variations of neural solvers in solving different characteristic MWPs. The architectures of the typical neural solvers are studied and a four-dimensional index model is proposed to characterize the benchmark dataset into different subsets. The experimental results show that the Seq2Seq-based model solvers perform well on most of the subsets, while Graph2Tree based solvers seem to have more potential in solving problems with complex expression structures.