Building Morphological Analyzer for Informal Text in Indonesian

Abstract

Informal text is heavily used by Indonesian in social media. However, NLP tool that can process such text is still very limited. In this work, we built a morphological analyzer for informal text in Indonesian by adding new rules for informal words to an existing Indonesian morphological analyzer named Aksara. Moreover, we also enrich the Aksara lexicon with informal words. The tool can perform tokenization, lemmatization, and part-of-speech (POS) tagging. Aksara uses a rule-based method using a finite-state transducer with a compiler named Foma. To evaluate the tool, we created a gold standard of 102 sentences with 1434 tokens which around 30 % are informal. The test results show that our tool has a tokenization accuracy of 97.21 %, while lemmatization accuracy for case insensitive is 90.37 %, and POS tagging evaluation reached an F1-Score value of 80%.