Chinese term extraction from web pages based on expected point-wise mutual information

Abstract

Point-wise Mutual Information(PMI) has been widely used in many areas of lexicon construction, term extraction and text mining. However, PMI has a well-known tendency, which is overvaluing the relatedness of word pairs that involve low-frequency words. To overcome this limitation, Expected Point-wise Mutual Information (PMIK) has been proposed empirically. In this paper, we propose an automatic term recognition system for Chinese and theoretically prove that with variant k ≥ 3, PMIK method can overcome the bias of low-frequency words. The experiment results on Chinese SINA blog and Baidu Tieba corpus show that with a proper k value of 5, the system can achieve a precision greater than 81% for top 1000 extracted terms without decreasing the recall.