Research on Pre-training of Tibetan Natural Language Processing

In the field of natural language processing, pre-training can effectively improve the performance of downstream tasks. In recent years, pre-training has been continuously developed in Tibetan NLP. We built three pre-trained models of Tibetan Word2Vec, Tibetan ELMo, and Tibetan ALBERT, and applied them to the two downstream tasks of Tibetan text classification and Tibetan part-of-speech tagging. Comparing them with the baseline models of these two downstream tasks, it is found that the performance of the downstream tasks using the pre-training is significantly better than the baseline model. The three pre-trained models have also brought a gradual improvement in performance for Tibetan downstream tasks.