Meme Detection For Sentiment Analysis and Human Robot Interactions Using Multiple Modes

Memes are a way of communicating concepts across social media. However, while most memes are intended to be funny, some can turn into offensive as well when text and images are combined together. Recently many successful studies related to sentiment analysis of both image and text have been performed. Such technology, when developed successfully, can be useful for effective Human-Robot-Interactions, specially with humanoid and collaborative robots. In this research, we intend to first develop such technology with available data set using given classes only, since getting labelled data in the robotics domain, specially in robot grasping domain is difficult. In subsequent research, we may extend the same technology for intelligent robot grasping. However, the majority of the research uses either text or images for the sentiment analysis. Since the content and image in memes are sometimes unrelated, detecting hateful memes is a more challenging problem, so the present work considers both as features and uses a multimodal approach for sentiment analysis which could also be useful for Human-Robot-Interactions. Being constrained however with the available data sets, in the present investigation, our focus is on developing multimodal and sequential approaches for classifying these memes into different required classes, more specifically, here two classes: offensive and non-offensive. The fusion approach has been used within multiple modes to take features of both image and text through different models and then it has been used for the classification. While in the sequential approach, the image captioning model which is trained on the MS COCO dataset, with Optical Character Recognition (OCR), is used and classified with the help of the FastText classifier. Both approaches are used on two datasets, one is the MultiOFF dataset, and the other is the Facebook Hateful Meme dataset. Results on both datasets are found to be promising for both approaches.