HMM based Named Entity Recognition for inflectional language

Abstract

Named Entity Recognition (NER) is the problem of identifying named entities in natural language text, classifying them into various classes and assigning the proper class tag to each word in its context. This paper describes a Named Entity Recognition system for Marathi using Hidden Markov Model (HMM). It addresses the problem of assigning the correct named entity class tag to each word using probabilistic model trained on a manually tagged corpus for the Marathi language. The most probable named entity tag is assigned to each word using the Viterbi algorithm. Proposed system reports an overall F1-score of 62.70% when no preprocessing was applied whereas it reports an overall F1-score of 77.79% when preprocessing was applied on the same data. Thus, the performance of the system is improved by 15% when linguistic knowledge is used to preprocess the test and training dataset.