From Translation to Generative LLMs: Classification of Code-Mixed Affective Tasks

Abstract

Code-mixed (CM) discourse combines multiple languages in a single text. It is commonly used in informal discourse in countries with several official languages, but also in many other countries in combination with English or neighboring languages. With the recent rise of large transformer language models dominating NLP tasks, we explored their effectiveness in CM contexts. We developed four new bilingual pre-trained masked language models for Hinglish and English-Slovene languages, tailored to handle informal language. We then evaluated monolingual, bilingual, few-lingual, massively multilingual, and larger generative models across multiple languages using two affective tasks involving CM texts: sentiment analysis and offensive speech prediction in social media posts. We compared these models with two translation baselines, one obtained with a neural machine translation tool and the other produced by large generative models. The experiments conducted in five languages: French, Hindi, Russian, Slovene, and Tamil, reveal that fine-tuned bilingual models and multilingual models designed for social media texts outperform others, with massively multilingual and monolingual models following, while larger generative models lag. For the affective tasks studied, models generally performed better on CM data than on non-CM data. The monolingual models with translated datasets rarely compete with multilingual models trained on CM datasets.