Labeling abnormalities in video based complex Human-Object Interactions by robust affordance modelling

Abstract

Identifying abnormalities in complex Human Object Interaction (HOI) based videos and labeling their possible categories is a novel and ambitious research problem, which requires an optimal blend of the state of the art computer vision and machine learning algorithms. For classifying a HOI event normal or abnormal and subsequently classifying the potential abnormal categories requires the knowledge of the mutual relations between the Human, object and the ambient environment. Researchers have been using various contexts like spatial, temporal, sequential etc. to classify the abnormal actions. In this paper, we have introduced a novel context of object's affordance (which is a semantic map of the human, object and the ambient environment) to identify abnormalities in Human Object Interactions. Furthermore, the sub-classification of the abnormalities is also realized. In order to achieve our goal, we have introduced a set of novel attributes associated with the Human and the Objects and mapped them in a Bayesian network framework. The inference capabilities of the system depict the successful identification of abnormal events. We have also initiated a novel dataset of abnormal Human-Object Interactions in domestic settings. This research work also made a valiant effort to capitalize the abundant statistical data sources currently available, related to the domestic accidents and use them to nourish a practical classifier.