The errors analysis of natural language generation — A case study of Topic-to-Essay generation

Although natural language generation (NLG) has achieved great success, there are still many problems with the generated text, if humans carefully examine it. To analyze the problems of NLG, we use manual evaluation methods to annotate and analyze the text generated by NLG. According to the analysis results, we can understand the defects of NLG in-depth, comprehensively, and accurately. Further, these provide cues for future improvement. In this paper, we first use a state-of-the-art Topic-to-Essay generation model to generate texts conditional on some topic words. Then, by analyzing the generated text, we propose an annotation framework, and then quantify the main drawbacks of current NLG, including poor semantic coherence, content duplication, logic errors, and repetition. It shows that the text generated by the current sequence-to-sequence model is still far from human expectation.