Handwritten Text Extraction from Application Forms using Image Processing and Deep Learning

Abstract

Around 400 million people worldwide use English as their first language, resulting it as the most extensively spoken language across the globe. Many government offices and other businesses use offline forms, the majority of which must be filled out in English. Manually digitalizing those forms is an impossible, time-consuming, and error-prone task, so extracting text from them may solve the problem. In many organizations, they have forms that have text boxes to fill out the information. So extracting text from these forms is a crucial solution. The proposed solution comprises image recognition, so the neural network is a far more convincing approach. A deep learning model, i.e., convolutional neural networks, is used to classify the characters. The handwritten alphabets and numbers are collected from Kaggle and Mnist datasets. Using those datasets, two CNN models were trained. Image processing techniques were used, which helped in the preprocessing of the image. Finding the image's coordinates and performing a perspective transform results in the removal of undesirable areas of the input image. Horizontal and vertical lines were detected, which resulted in the finding of the rectangular boxes in the form where the data is contained. According to the type of the detected box, each character in the box is sent to the respective model, which results in identifying the character and helping to find the content in that particular field. All detected content is automatically saved in an Excel sheet. The proposed system achieves an accuracy of 85%.