Investigating fairness in machine learning-based audio sentiment analysis

Abstract

Audio sentiment analysis is a growing area of research, however little attention has been paid to the fairness of machine learning models in this field. Whilst the current literature covers research on machine learning models’ reliability and fairness in various demographic groups, fairness in audio sentiment analysis with respect to gender is still an uninvestigated field. To fill this knowledge gap, we conducted experiments aimed at assessing the fairness of machine learning algorithms concerning gender within the context of audio sentiment analysis. In this research, we used 442 audio files of happiness and sadness—representing equal samples of male and female subjects—and generated spectrograms for each file. Then we performed feature extraction using bag-of-visual-words method followed by building classifiers using Random Forest, Support Vector Machines, and K-nearest Neighbors algorithms. We investigated whether the machine learning models for audio sentiment analysis are fair across female and male genders. We found the need for gender-specific models for audio sentiment analysis instead of a gender-agnostic-model. Our results provided three pieces of evidence to back up our claim that gender-specific models demonstrate bias in terms of overall accuracy equality when tested using audio samples representing the other gender, as well as combination of both genders. Furthermore, gender-agnostic-model performs poorly in comparison to gender-specific models in classifying sentiments of both male and female audio samples. These findings emphasize the importance of employing an appropriate gender-specific model for an audio sentiment analysis task to ensure fairness and accuracy. The best performance is achieved when using a female-model (78% accuracy) and a male-model (74% accuracy), significantly outperforming the 66% accuracy of the gender-agnostic model.