Automata Classification with Convolutional Neural Networks for Use in Assistive Technologies for the Visually Impaired

Abstract

Our goal is to evaluate the use of Convolutional Neural Networks (CNN) in the recognition of automata images and to create a model that can be used in the construction of assistive tools. Visually impaired individuals that are studying Computer Science have difficulty in accessing and learning diagrams. Despite the solutions available in the literature to make diagrams accessible to blind students and allow the creation and manipulation of such material, we seek to give access to images of didactic materials and books. The method used consists of two steps: classification of the data using three types of CNN and the combination of the results to make a final decision. Two approaches were chosen to be tested: recognition of the type of automaton and recognition of the number of states of the automaton. Our best result was using late fusion of the three CNNs by the product rule, which resulted in an accuracy of 97% for the automaton type recognition and 91% for the recognition of the number of states of the automaton.