Medical Text Extraction and Classification from Prescription Images

The right to health is one of the fundamental human rights. Every state is obliged to provide healthcare facilities to its population. In Bangladesh, the government is working hard to provide a better healthcare system, though the country needs to go a long way to have a unified healthcare system. There is a lack of a proper referral system in the country, and proper diagnosis is hindered due to a patient’s lack of medical history. In this paper, we propose a system that helps the patient to create a medical history from images of the prescriptions. Our system extracts and classifies data from an unstructured Bangladeshi medical prescription that can be used to create a repository of medical history. The proposed method works in four phases: phase I text localization and extraction from the images of prescriptions, phase II - classification of the extracted images, phase III - image to text conversion using OCR, and phase IV - classification of the text in four categories symptoms, medicines, diagnostic tests, and others. For image classification, we use a very deep convolutional network, VGG-16 and for text classification, we use the Bidirectional Encoder Representations from Transformers (BERT) model. Performance evaluation of the proposed system is very promising and the system can be used in any country like Bangladesh to facilitate better treatment.