IndoKEPLER, IndoWiki, and IndoLAMA: A Knowledge-enhanced Language Model, Dataset, and Benchmark for the Indonesian Language

Abstract

Pretrained language models posses an ability to learn the structural representation of a natural language by processing unstructured textual data. However, the current language model design lacks the ability to learn factual knowledge from knowledge graphs. Several attempts have been made to address this issue, such as the development of KEPLER. KEPLER combines the BERT language model and TransE knowledge embedding method to achieve a language model that can incorporate knowledge graphs as training data. Unfortunately, such knowledge enhanced language model is not yet available for the Indonesian language. In this experiment, we propose IndoKEPLER: a language model trained usingWikipedia Bahasa Indonesia andWikidata. We also create a new knowledge probing benchmark named IndoLAMA to test the ability of a language model to recall factual knowledge. The benchmark is based on LAMA, which is designed to test the suitability of our language model to be used as a knowledge base. IndoLAMA tests a language model by giving cloze style question and compare the prediction of the model to the factually correct answer. This experiment shows that IndoKEPLER increases the ability of a normal DistilBERT model to recall factual knowledge by 0.8%. Moreover, the most significant increase happens when dealing with many-to-one relationships, where IndoKEPLER outperforms it’s original text encoder model by 3%.