Estimating the Operation of Unknown Appliances for Service Robots Using CNN and Ontology

We can expect robots to efficiently perform tasks using appliances in a similar way that humans do. A common approach is to build appliances' models so that robot can operate them but this process is time-consuming. In this paper, we propose a method to estimate the proper operation of appliances using ontology and convolutional neural networks (CNN). We propose to use CNNs to detect the appliances and the operating parts, and then perform an ontology analysis of the operating parts (e.g., buttons) and the appliances to infer the proper operation. This method can be used for appliances which it was not trained for because the dataset has a high generalization due to the inclusion of multiple appliances and the separated training for appliances and operating parts. We experimentally verify the effectiveness of the proposed method with a service robot operating in multi-object environments.