Transforming Hand-drawn Sketches of Linkage Mechanisms into their Digital Representation

Abstract This paper presents an approach based on deep neural networks for interactive digital transformation and simulation of n-bar planar linkages composed of revolute and prismatic joints from hand-drawn sketches. Rather than relying solely on computer vision, our approach leverages the topological knowledge of linkage mechanisms in combination with the output of a convolutional deep neural network. This creates a framework for recognition of hand-drawn sketches. Our methodology involves first generating a dataset of synthetic images of linkage mechanism sketches that resemble hand-drawn sketches. We then fine-tune a state-of-the-art deep neural network capable of detecting discrete objects using a set of building blocks of linkage mechanisms, specifically joints and links in various positions, scales, and orientations. We perform a topological analysis on the set of detected objects to create a kinematic model of the sketched mechanisms. Results indicate that our algorithm performs well on hand-drawn sketches, and it can aid in the conversion of such sketches into their digital representations. This has implications for effective communication, analysis, cataloging, and classification of planar mechanisms. Furthermore, our approach could lay the groundwork for image-based synthesis of planar mechanisms, which would be insensitive to their complexity or properties, such as the algebraic degree of the coupler curves.