Recognition of Assembly Instructions Based on Geometric Feature and Text Recognition

Recent advances in machine learning methods have increased the performances of object detection and recognition systems. Accordingly, automatic understanding of assembly instructions in manuals in the form of electronic or paper materials has also become an issue in the research community. This task is quite challenging because it requires the automatic optical character recognition (OCR) and also the understanding of various mechanical parts and diverse assembly illustrations that are sometimes difficult to understand even for humans. Although deep networks are showing high performance in many computer vision tasks, it is still difficult to perform this task by an end-to-end deep neural network due to the lack of training data, and also because of diversity and ambiguity of illustrative instructions. Hence, in this paper, we propose to tackle this problem by using both conventional non-learning approaches and deep neural networks, considering the current state-of-the-arts. Precisely, we first extract components having strict geometric structures, such as characters and illustrations, by conventional non-learning algorithms, and then apply deep neural networks to recognize the extracted components. The main targets considered in this paper are the types and the numbers of connectors, and behavioral indicators such as circles, rectangles, and arrows for each cut in do-it-yourself (DIY) furniture assembly manuals. For these limited targets, we train a deep neural network to recognize them with high precision. Experiments show that our method works robustly in various types of furniture assembly instructions.