Joint Chinese word segmentation and punctuation prediction using deep recurrent neural network for social media data

Abstract

In this work, we propose to jointly perform Chinese word segmentation (CWS) and punctuation prediction (PU) in a unified framework using deep recurrent neural network (DRNN). We further perform a comparative study among the joint frameworks, the isolated prediction and the pipeline methods that link the two tasks sequentially, on a social media corpus. Our experimental results show that joint models improve performance of CWS and affect PU marginally. We also study the effects of CWS and PU on Chinese-to-English machine translation (MT) quality by evaluating on a parallel social media corpus. It is shown that joint models are superior to the isolated prediction and the pipeline approaches.