Transgenerators

Abstract

Pre-trained Transformers(GPT) are showed great performance in natural language generation task. This model was trained in a self-supervised manner on a large amount of text data crawled from the WEB. Such a dataset has not the highest quality, many sentences are prone to errors such as typos or grammar mistakes. As a result, text generated by GPTs consists of a lot of grammar incorrect sentences. While Transformers is also showed great performance in translation tasks, we propose the conception when a model can handle a generation and a translation task at the same time. But we propose a specific type of translation, in our method Transformer is training to translate a sentence with grammar errors to the same sentences without errors. In the full case, an incorrectly generated sentence can be corrected by the extended version of the same model, we call this type of model Transgenerator. We applied several experiments to estimate a generative power of Transgenerator based on GPT-2 architecture and the proposed method outperformed original GPT-2 model on the range of tasks