An End-to-End Uyghur Speech Recognition Based on Multi-task Learning

Abstract

With the emergence of deep learning, speech recognition research for languages with a wide speaker base, such as English and Chinese, has become reasonably mature. However, Uyghur speech recognition research has only developed slowly, because the modeling is subpar since Uighur is a low-resource language. To solve the aforementioned problem, we propose an end-to-end Uyghur speech recognition model based on multi-task learning with Branchformer. Branchformer can capture both global and local contexts, allowing it to learn richer features from low volumes of data to improve model performance in resource-constrained situations. The multi-task learning model can fully utilize the data through multiple related tasks, enhancing the model’s generalizability under low resources. In this paper, the multi-task learning model is trained and decoded using modeling units including phoneme, sub-word, and word. The performance of the model is then evaluated using various datasets. The results show that the proposed model outperforms the mainstream models in terms of recognition, in the self-built database and open-source corpus Thuyg-20, respectively, the word accuracy of the proposed model reaches 94.48% and 91.16%, which is a significant improvement compared with each benchmark model.