Language Resource Construction of Multi-Domain Philippine English Text for Pre-training Objective

Abstract

Pre-trained language models (PLMs) have gained significant attention in NLP because of its effectiveness in improving the performance of several downstream tasks. Pre-training these PLMs requires benchmark datasets to create universal language representation and to generate robust models. This paper established the first linguistic resource for Philippine English language to help future researchers in language modeling and other NLP tasks. We used NLP approach to prepare and build our data and transformers paradigm to generate small PLMs. The PHEnText corpus is composed of multi-domain Philippine English text data in formal language scraped from different sources. Tokenization process was performed using BPE and WordPiece tokenizer algorithms. Using a subset of the PHEnText, we generated four small versions of transformer-based language models. Cross-validation during the pre-training reported that a RoBERTa-base model outperformed all other variants in terms of training loss, evaluation loss and accuracy. This work introduced the PHEnText benchmark corpus composed of 2.6B tokens primarily intended for pre-training objective. The corpus provides starting point and opportunities for current and future NLP researches and once trained, can be used more efficiently via fine-tuning. Additionally, the dataset was prepared to be pre-training compatible with different transformer models. Furthermore, the generated PLMs using a subset of PHEnText rendered notable results in terms of minimal loss and nearly acceptable accuracy. Next step for this undertaking is to train PLMs using the entire PHEnText dataset and to test the models' effectiveness by fine-tuning them to NLP downstream tasks.