Text Prediction Zero Probability Problem Handling with N-gram Model and Laplace Smoothing

Abstract

In Natural Language Processing, text prediction represents the process of predicting the word with the highest probability through a predictive language model from a series of text corpus. The N-gram model is familiar and considered the handiest and most computationally cost-effective model for text processing. Additionally, higher N-gram models, especially the 5-gram ones, give the best text prediction. Interestingly, these better prediction results were obtained only on the training dataset. In contrast, the highest N-gram model imploded badly on the evaluation dataset. This paper proposes an approach where the N-gram model, especially the bi-gram model, and the fine tuning with Laplace Smoothing, provide the best prediction results at the evaluation stage.