A comparative study on code-mixed data of Indian social media vs formal text

Abstract

This paper presents comparative experiment results of code mixed data with the normal text. We first identify the Languages present in social media text, in the case of code mixed data existing language detector fails to detect language at the word level because of the use of roman script to write their own language. So we bootstrap language identification step and we caluculate the Code Mixe Index to show the amount of code mix in the corpora. We use the RNNLM to create a language model of code mixed data as well as pen tree bank data. We use the model to evaluate the similarity of code mixed data and open tree bank data. Using Perplexity measure we show that the code mixed data of Indian social media very less similarity to the normal data.