An Evaluation Dataset Construction Approach for Task-Oriented Dialogue

Abstract

Aiming to construct an evaluation dataset for task-oriented dialogues under slot filling task, this paper proposes a dataset construction approach based on two optimized data augmentation techniques named back-translation annotation synchronization and slot substitution. These optimized techniques perform well in reducing error annotations introduced by data augmentation and help maintain the style and difficulty of the original dataset. Besides, these techniques can be easily implemented by leveraging commercial interfaces and executing automated scripts, making the approach especially suitable for evaluation dataset construction. In experiments, MultiWOZ 2.0 was utilized as the benchmark dataset to generate new samples. The newly generated dialogues have lower error rate in annotations, and show the same evaluation capability as the original data, which verifies the feasibility of the construction approach and the effectiveness of two optimization methods.