Binary-perspective Asymmetrical Twin Gain: a Novel Evaluation Method for Question Generation

Abstract

We propose a novel evaluation method for Question Generation (QG) task. It is designed to verify the quality of the generated questions in terms of different references, including not only the manually-written questions (i.e., ground truth) but also their variants. Back translation is utilized to obtain the variants, and accordingly, they generally appear as paraphrases of the ground-truth examples. In particular, an Asymmetrical Twin Gain (ATG) is proposed for binary-perspective evaluation using the existing metrics, such as BLEU and ROUGE-L, respectively. It enables both the metrics to be observed from two perspectives, including the consistency between QG results and ground-truth examples, as well as that of variants. The experiments on the publicly-available benchmark SQuAD demonstrate the reliability of ATG. More importantly, ATG is proven effective for indicating the stable QG performance. It is noteworthy that the proposed binary-perspective evaluation is explored for assisting the conventional evaluation methods, instead of replacing them. The contribute can be identified as the additional insight into the robustness of QG when some slightly-different references (e.g., paraphrases) are offered for evaluation. All the models and source codes in the experiments will be made publicly available to support reproducible research.