OWA Operators for the Fusion of Social Networks’ Comments with Audio-Visual Content

Social networks’ comments are rich sources of information that may be fused with audio-visual contents to improve emotional video retrieval systems. The rationale behind this fusion is that different sources can complement each other since they have different natures, formats, and origins. Although emotional information expressed in users’ comments on the Web is in accordance with the emotional audio-visual content of videos, they are not synchronized. In order to address this problem, decision-level fusion is needed when such asynchronous modalities should be fused. In this article, a new decision-level fusion approach based on Ordered Weighted Averaging (OWA) operators is proposed. In this approach, emotion is first detected based on the audio, video, and users’ comments and then, individual decisions are fused using the OWA method. The proposed method is evaluated on the music videos of the standard DEAP data set. The results of comparing the proposed method with average, product, sum, and Dempster-Shafer fusion methods show that the proposed OWA-based method outperforms other methods in different fusion settings.