Multi-label Prediction for Visual Sentiment Analysis using Eight Different Emotions based on Psychology

Abstract

In visual sentiment analysis, sentiment estimation from images is a challenging research problem. Previous studies focused on a few specific sentiments and their intensities and have not captured abundant psychological human feelings. In addition, multi-label sentiment estimation from images has not been sufficiently investigated. The purpose of this research is to build a visual sentiment dataset, accurately estimate the sentiments as a multi-label multi-class problem from images that simultaneously evoke multiple emotions. We built a visual sentiment dataset based on Plutchik's wheel of emotions. We describe this ‘Senti8PW’ dataset, then perform multi-label sentiment analysis using the dataset, where we propose a combined deep neural network model that enables inputs from both hand-crafted features and CNN features. We also introduce a threshold-based multi-label prediction algorithm, in which we assume that each emotion has a probability distribution. In other words, after training our deep neural network, we predict evoked emotions for an image if the intensity of the emotion is larger than the threshold of the corresponding emotion. Extensive experiments were conducted on our dataset. Our model achieves superior results compared to the state-of-the-art algorithms in terms of subsets.