Tokenization and N-Gram for Indexing Indonesian Translation of the Quran

Abstract

Tokenization is an important process used to break the text into parts of a word. N-gram model now is widely used in computational linguistics for predicting the next item in such a contiguous sequence of $\mathbf{n}$ items from a particular sample of text. This paper focuses on the implementation of tokenization and n-gram model using RapidMiner to produce unigram and bigram word for indexing Indonesian Translation of the Quran (ITQ). This study uses ITQ data sets consisting of 114 documents. The methods are data extracting and preprocessing text including tokenization, stemming, stopword removal, transformation cases, and n-grams. The results of this study showed the model produces the 6794 and 60323 tokens combination unigram and bigram use for index ITQ. Significant the contribution of this study is to enhance the digital index of ITQ.