Urdu noun phrase chunking: HMM based approach

Abstract

Extraction of noun phrase (NP) from text is useful for many natural language processing applications, such as name entity recognition, indexing, searching, parsing etc. We present a noun phrase chunker for Urdu which is based on a statistical approach. A 100,000 words Urdu corpus is manually tagged with NP chunk tags. The corpus is used to develop a statistical approach. Initially, a statistical approach based on standard HMM model is developed for automatics NP chunking. In Urdu phrases, the case marker (CM) indicates the end of a noun phrase and is appended at its end. Thus, if one scans the sentence in reverse order, one may be able to better predict phrase endings. So, the technique is enhanced by changing scanning direction. The technique is further enhanced by merging chunk and POS tags to achieve maximum accuracy. The results of all experiments are reported with maximum overall accuracy of 97.61% achieved using HMM based approach with extended tagset and right to left (RTL) scanning.